| |
 |
| Paper: |
Automated and Scalable Data Reduction in the \textsc{Sofia} Data Processing System |
| Volume: |
495, Astronomical Data Analysis Software and Systems XXIV (ADASS XXIV) |
| Page: |
363 |
| Authors: |
Krzaczek, R.; Shuping, R.; Charcos-Llorens, M.; Alles, R.; Vacca, W. |
| Abstract: |
In order to provide suitable data products to general investigators
and other end users in a timely manner, the Stratospheric
Observatory for Infrared Astronomy SOFIA) has developed a
framework supporting the automated execution of data processing
pipelines for the various instruments,
called the Data Processing System (DPS),
see Shuping et al. (2014) for overview).
The primary requirement is to process all data collected from a
flight within eight hours, allowing data quality assessments and
inspections to be made the following day.
The raw data collected during a flight requires processing by a
number of different software packages and tools unique to each
combination of instrument and mode of operation, much of it
developed in-house, in order to create data products for use by
investigators and other end-users.
The requirement to deliver these data products
in a consistent, predictable, and
performant manner presents a significant challenge for the observatory.
Herein we present aspects of the DPS that
help to achieve these goals.
We discuss how it supports data reduction software written in a
variety of languages and environments, its support for
new versions and live upgrades to that software and other necessary
resources (e.g., calibrations),
its accommodation of sudden processing
loads through the addition (and eventual removal) of
computing resources, and close with an observation of the performance
achieved in the first two observing cycles of SOFIA. |
|
|
|
|
|
