| |
 |
| Paper: |
Workflows using Pegasus: Enabling Dark Energy Survey Pipelines |
| Volume: |
523, Astronomical Data Analysis Software and Systems XXVIII |
| Page: |
689 |
| Authors: |
Vahi, K.; Wang, M. H.; Chang, C.; Dodelson, S.; Rynge, M.; Deelman, E. |
| Abstract: |
Workflows are a key technology for enabling complex scientific applications. They capture the inter-dependencies between processing steps in data analysis and simulation pipelines, as well as the mechanisms to execute those steps reliably and efficiently in a distributed computing environment. They also enable scientists to capture complex processes to promote method sharing and reuse and provide provenance information necessary for the verification of scientific results and scientific reproducibility. We describe a weak-lensing pipeline that is modeled as a Pegasus workflow with pipeline codes available as a Singularity container. This has enabled us to make this analysis widely available and easily replicable to the astronomy community. Using Pegasus, we have executed various steps of pipelines on different compute sites with varying infrastructures, with Pegasus seamless managing the data across the various compute clusters in a transparent manner. |
|
|
|
|
|
