Jump to content

Scientific workflow system

Add links
From Wikipedia, the free encyclopedia

A scientific workflow system is a specialized form of a workflow management system designed specifically to compose and execute a series of computational or data manipulation steps, or workflow, in a scientific application.[1] Scientific workflow systems are generally developed for use by scientists from different disciplines like astronomy, earth science, and bioinformatics. All such systems are based on an abstract representation of how a computation proceeds in the form of a directed graph, where each node represents a task to be executed and edges represent either data flow or execution dependencies between different tasks. Each system typically provides a visual front-end, allowing the user to build and modify complex applications with little or no programming expertise.[2][3][4]

Applications

[edit]

Distributed scientists can collaborate on conducting large scale scientific experiments and knowledge discovery applications using distributed systems of computing resources, data sets, and devices. Scientific workflow systems play an important role in enabling this vision.

More specialized scientific workflow systems provide a visual programming front end enabling users to easily construct their applications as a visual graph by connecting nodes together, and tools have also been developed to build such applications in a platform-independent manner.[5] Each directed edge in the graph of a workflow typically represents a connection from the output of one application to the input of the next. A sequence of such edges may be called a pipeline.

Scientific workflows

[edit]

The simplest computerized scientific workflows are scripts that call in data, programs, and other inputs and produce outputs that might include visualizations and analytical results. These may be implemented in programs such as R or MATLAB, using a scripting language such as Python with a command-line interface, or more recently using open-source web applications such as Jupyter Notebook.

There are many motives for differentiating scientific workflows from traditional business process workflows. These include:

  • providing an easy-to-use environment for individual application scientists themselves to create their own workflows.
  • providing interactive tools for the scientists enabling them to execute their workflows and view their results in real-time.
  • simplifying the process of sharing and reusing workflows between the scientists.
  • enabling scientists to track the provenance of the workflow execution results and the workflow creation steps.

By focusing on the scientists, the focus of designing scientific workflow system shifts away from the workflow scheduling activities, typically considered by grid computing environments for optimizing the execution of complex computations on predefined resources, to a domain-specific view of what data types, tools and distributed resources should be made available to the scientists and how can one make them easily accessible and with specific Quality of Service requirements [6]

Scientific workflows are now recognized[by whom?] as a crucial element of the cyberinfrastructure, facilitating e-Science. Typically sitting on top of a middleware layer, scientific workflows are a means by which scientists can model, design, execute, debug, re-configure, and re-run their analysis and visualization pipelines. Part of the established scientific method is to create a record of the origins of a result, how it was obtained, experimental methods used, machine calibrations and parameters, etc. It is the same in e-Science, except provenance data are a record of the workflow activities invoked, services and databases accessed, data sets used, and so forth. Such information is useful for a scientist to interpret their workflow results and for other scientists to establish trust in the experimental result.[7]

Sharing workflows

[edit]

Social networking communities such as myExperiment have been developed to facilitate sharing and collaborative development of scientific workflows. Galaxy provide collaborative mechanisms for editing and publication of workflow definitions and workflow results directly on the Galaxy installation.

Analysis

[edit]

A key assumption underlying all scientific workflow systems is that the scientists themselves will be able to use a workflow system to develop their applications based on visual flowcharting, logic diagramming, or, as a last resort, writing code to describe the workflow logic. Powerful workflow systems make it easy for non-programmers to first sketch out workflow steps using simple flowcharting tools, and then hook in various data acquisition, analysis, and reporting tools. For maximum productivity, details of the underlying programming code should normally be hidden.

Workflow analysis techniques can be used to analyze the properties of such workflows to verify certain properties before executing them. An example of a theoretical formal analysis framework for the verification and profiling of the control-flow aspects of scientific workflows and their data flow aspects for the Discovery Net system is described in the paper, "The design and implementation of a workflow analysis tool" by Curcin et al.[8]

The authors note that introducing program analysis and verification into the workflow world requires detailed understanding of execution semantics of workflow language, including execution properties of nodes and arcs in the workflow graph, understanding functional equivalencies between workflow patterns, and many other issues. Doing such analysis is difficult, and addressing these issues requires building on formal methods used in computer science research (e.g. Petri nets) and building on these formal methods to develop user-level tools to reason about the properties of both workflows and workflow systems. The lack of such tools in the past stopped automated workflow management solutions from maturing from nice-to-have academic toys to production-level tools used outside the narrow circle of early adopters and workflow enthusiasts.

Notable systems

[edit]

Notable scientific workflow systems include:[9]

  • Anduril, bioinformatics and image analysis
  • Apache Airavata, a general purpose workflow management system[10]
  • Apache Airflow, a general purpose workflow management system
  • Apache Taverna, widely used in bioinformatics, astronomy, biodiversity
  • BioBIKE, a cloud-based bioinformatics platform
  • Bioclipse, a graphical workbench, with a scripting environment that lets you perform complex actions as a kind of workflow.
  • Collective Knowledge, a Python-based general workflow and experiment crowdsourcing framework with JSON API and cross-platform package manager
  • Common Workflow Language, a community-developed YAML-based workflow language, supported by multiple engine implementations.
  • Cuneiform, a functional workflow language.
  • Discovery Net, one of the earliest examples of a scientific workflow system
  • Galaxy, initially targeted at genomics
  • GenePattern, a powerful scientific workflow system that provides access to hundreds of genomic analysis tools.[11]
  • Kepler, a scientific workflow management system
  • KNIME, an open-source data analytics platform
  • Nextflow, a bioinformatic data analysis workflow system
  • OnlineHPC, online scientific workflow designer and high performance computing toolkit
  • Orange, open source data visualization and analysis
  • Pegasus, an open-source scientific workflow management system[12]
  • Pipeline Pilot, graphical programming with many tools to address Cheminformatics workflows [13]
  • Swift parallel scripting language, a scripting language with many of the capabilities of scientific workflow systems built-in.
  • VisTrails, a scientific workflow system developed in Python

More than 280 computational data analysis workflow systems have been identified,[14] although the distinction between data analysis workflows and scientific workflows is fluid, as not all analysis workflow systems are used for scientific purposes.

Bioinformatics workflow management systems

[edit]

In alphabetical order, some examples of bioinformatics workflow management systems include:

Comparisons between bioinformatics workflow systems

[edit]

With a large number of bioinformatics workflow systems to choose from,[24] it becomes difficult to understand and compare the features of the different workflow systems. There has been little work conducted in evaluating and comparing the systems from a bioinformatician's perspective, especially when it comes to comparing the data types they can deal with, the in-built functionalities that are provided to the user or even their performance or usability. Examples of existing comparisons include:

  • The paper "Scientific workflow systems-can one size fit all?",[4] which provides a high-level framework for comparing workflow systems based on their control flow and data flow properties. The systems compared include Discovery Net, Taverna, Triana, Kepler as well as Yawl and BPEL.
  • The paper "Meta-workflows: pattern-based interoperability between Galaxy and Taverna"[25] which provides a more user-oriented comparison between Taverna and Galaxy in the context of enabling interoperability between both systems.
  • The infrastructure paper "Delivering ICT Infrastructure for Biomedical Research"[26] compares two workflow systems, Anduril and Chipster,[27] in terms of infrastructure requirements in a cloud-delivery model.
  • The paper "A review of bioinformatic pipeline frameworks"[28] attempts to classify workflow management systems based on three dimensions: "using an implicit or explicit syntax, using a configuration, convention or class-based design paradigm and offering a command line or workbench interface".

See also

[edit]

References

[edit]
  1. ^ Sun, LiewChee; P, AtkinsonMalcolm; GaleaMichelle; Fong, AngTan; MartinPaul; Van, HemertJano I. (2016-12-12). "Scientific Workflows". ACM Computing Surveys. 49 (4): 1–39. doi:10.1145/3012429. hdl:20.500.11820/774ef69e-a499-4bd2-a609-09f050e682ae. S2CID 9408644.
  2. ^ Oinn, T.; Greenwood, M.; Addis, M.; Alpdemir, M. N.; Ferris, J.; Glover, K.; Goble, C.; Goderis, A.; Hull, D.; Marvin, D.; Li, P.; Lord, P.; Pocock, M. R.; Senger, M.; Stevens, R.; Wipat, A.; Wroe, C. (2006). "Taverna: Lessons in creating a workflow environment for the life sciences" (PDF). Concurrency and Computation: Practice and Experience. 18 (10): 1067–1100. doi:10.1002/cpe.993. S2CID 10219281.
  3. ^ Yu, J.; Buyya, R. (2005). "A taxonomy of scientific workflow systems for grid computing". ACM SIGMOD Record. 34 (3): 44. CiteSeerX 10.1.1.63.3176. doi:10.1145/1084805.1084814. S2CID 538714.
  4. ^ a b Curcin, V.; Ghanem, M. (2008). "Scientific workflow systems - can one size fit all?". 2008 Cairo International Biomedical Engineering Conference. pp. 1–9. doi:10.1109/CIBEC.2008.4786077. ISBN 978-1-4244-2694-2. S2CID 1885579.
  5. ^ D. Johnson; et al. (December 2009). "A middleware independent Grid workflow builder for scientific applications" (PDF). 2009 5th IEEE International Conference on E-Science Workshops. pp. 86–91. doi:10.1109/ESCIW.2009.5407993. ISBN 978-1-4244-5946-9. S2CID 3339794.
  6. ^ Kyriazis, Dimosthenis; Tserpes, Konstantinos; Menychtas, Andreas; Litke, Antonis; Varvarigou, Theodora (2008). "An innovative workflow mapping mechanism for Grids in the frame of Quality of Service". Future Generation Computer Systems. 24 (6): 498–511. doi:10.1016/j.future.2007.07.009.
  7. ^ Automatic capture and efficient storage of e-Science experiment provenance. Concurrency Computat.: Pract. Exper. 2008; 20:419–429
  8. ^ Curcin, V.; Ghanem, M.; Guo, Y. (2010). "The design and implementation of a workflow analysis tool". Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 368 (1926): 4193–4208. Bibcode:2010RSPTA.368.4193C. doi:10.1098/rsta.2010.0157. PMID 20679131.
  9. ^ Barker, Adam; Van Hemert, Jano (2008), "Scientific Workflow: A Survey and Research Directions", Parallel Processing and Applied Mathematics, 7th International Conference, PPAM 2007, Revised Selected Papers, Lecture Notes in Computer Science, vol. 4967, Gdansk, Poland: Springer Berlin / Heidelberg, pp. 746–753, CiteSeerX 10.1.1.105.4605, doi:10.1007/978-3-540-68111-3_78, ISBN 978-3-540-68105-2
  10. ^ Marru, Suresh; Gardler, Ross; Slominski, Aleksander; Douma, Ate; Perera, Srinath; Weerawarana, Sanjiva; Gunathilake, Lahiru; Herath, Chathura; Tangchaisin, Patanachai; Pierce, Marlon; Mattmann, Chris; Singh, Raminder; Gunarathne, Thilina; Chinthaka, Eran (2011-11-18). Proceedings of the 2011 ACM workshop on Gateway computing environments - GCE '11. p. 21. doi:10.1145/2110486.2110490. ISBN 9781450311236. S2CID 18341808.
  11. ^ Reich, Michael; Liefeld, Ted; Gould, Joshua; Lerner, Jim; Tamayo, Pablo; Mesirov, Jill P (2006). "GenePattern 2.0". Nature Genetics. 38 (5): 500–501. doi:10.1038/ng0506-500. PMID 16642009. S2CID 5503897.
  12. ^ Deelman, Ewa; Vahi, Karan; Juve, Gideon; Rynge, Mats; Callaghan, Scott; Maechling, Philip J.; Mayani, Rajiv; Chen, Weiwei; Ferreira da Silva, Rafael; Livny, Miron; Wenger, Kent (May 2015). "Pegasus, a workflow management system for science automation". Future Generation Computer Systems. 46: 17–35. doi:10.1016/j.future.2014.10.008.
  13. ^ "BIOVIA Pipeline Pilot | Scientific Workflow Authoring Application for Data Analysis". Accelrys.com. Retrieved 2016-12-04.
  14. ^ "Existing Workflow systems". Common Workflow Language wiki. Archived from the original on 2019-10-17.
  15. ^ "Anduril workflow website".
  16. ^ Ovaska, Kristian; Laakso, Marko; Haapa-Paananen, Saija; Louhimo, Riku; Chen, Ping; Aittomäki, Viljami; Valo, Erkka; Núñez-Fontarnau, Javier; Rantanen, Ville (2010-09-07). "Large-scale data integration framework provides a comprehensive view on glioblastoma multiforme". Genome Medicine. 2 (9): 65. doi:10.1186/gm186. ISSN 1756-994X. PMC 3092116. PMID 20822536.
  17. ^ Elhai, J.; Taton, A.; Massar, J.; Myers, J. K.; Travers, M.; Casey, J.; Slupesky, M.; Shrager, J. (2009). "BioBIKE: A Web-based, programmable, integrated biological knowledge base". Nucleic Acids Research. 37 (Web Server issue): W28 – W32. doi:10.1093/nar/gkp354. PMC 2703918. PMID 19433511.
  18. ^ Brandt, Jörgen; Bux, Marc N.; Leser, Ulf (2015). "Cuneiform: A functional language for large scale scientific data analysis" (PDF). Proceedings of the Workshops of the EDBT/ICDT. 1330: 17–26.
  19. ^ Goecks, J.; Nekrutenko, A.; Taylor, J.; Galaxy Team, T. (2010). "Galaxy: A comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences". Genome Biology. 11 (8): R86. doi:10.1186/gb-2010-11-8-r86. PMC 2945788. PMID 20738864.
  20. ^ Reich, Michael; et al. (2006). "GenePattern 2.0". Nature Genetics. 38 (1): 500–5001. doi:10.1038/ng0506-500. PMID 16642009. S2CID 5503897.
  21. ^ Tiwari, Abhishek; Sekhar, Arvind K.T. (2007). "Workflow based framework for life science informatics". Computational Biology and Chemistry. 31 (5–6): 305–319. doi:10.1016/j.compbiolchem.2007.08.009. PMID 17931570.
  22. ^ Okonechnikov, K; Golosova, O; Fursov, M; Ugene, Team (2012). "Unipro UGENE: A unified bioinformatics toolkit". Bioinformatics. 28 (8): 1166–7. doi:10.1093/bioinformatics/bts091. PMID 22368248.
  23. ^ Bavoil, L.; Callahan, S.P.; Crossno, P.J.; Freire, J.; Scheidegger, C.E.; Silva, C.T.; Vo, H.T. (2005). "VisTrails: Enabling Interactive Multiple-View Visualizations". VIS 05. IEEE Visualization, 2005. pp. 135–142. doi:10.1109/VISUAL.2005.1532788. ISBN 978-0-7803-9462-9.
  24. ^ "Existing Workflow systems". Common Workflow Language wiki. Archived from the original on 2019-10-17. Retrieved 2019-10-17.
  25. ^ Abouelhoda, M.; Alaa, S.; Ghanem, M. (2010). "Meta-workflows". Proceedings of the 1st International Workshop on Workflow Approaches to New Data-centric Science - Wands '10. p. 1. doi:10.1145/1833398.1833400. ISBN 9781450301886. S2CID 17343728.
  26. ^ Nyrönen, TH; Laitinen, J; et al. (2012), Delivering ICT infrastructure for biomedical research, Proceedings of the WICSA/ECSA 2012 Companion Volume (WICSA/ECSA '12), ACM, pp. 37–44, doi:10.1145/2361999.2362006, ISBN 9781450315685, S2CID 18199745
  27. ^ Kallio, M. A.; Tuimala, J. T.; Hupponen, T; Klemelä, P; Gentile, M; Scheinin, I; Koski, M; Käki, J; Korpelainen, E. I. (2011). "Chipster: User-friendly analysis software for microarray and other high-throughput data". BMC Genomics. 12: 507. doi:10.1186/1471-2164-12-507. PMC 3215701. PMID 21999641.
  28. ^ Leipzig J (2016). "A review of bioinformatic pipeline frameworks". Briefings in Bioinformatics. 18 (3): 530–536. doi:10.1093/bib/bbw020. PMC 5429012. PMID 27013646.
[edit]
Retrieved from "https://en.wikipedia.org/w/index.php?title=Scientific_workflow_system&oldid=1286819495"