| |
 |
| Paper: |
The Euclid Archive Processing and Data Distribution Systems: a Distributed Infrastructure for Euclid and Associated Data |
| Volume: |
527, Astronomical Data Analysis Software and Systems XXIX |
| Page: |
291 |
| Authors: |
Williams, O. R.; Begeman, K.; Belikov, A. N.; Boxhoorn, D.; Droge, B.; Nutma, T. A.; Tsyganov, A.; Valentijn, E. A.; Vriend, W.–J.; Dabin, C. |
| Abstract: |
The Euclid Archive System is an ambitious information system, which sits at the heart of the Euclid Science Ground Segment. It is a joint development between the Euclid Consortium and the ESAC Science Data Centre. It encompases both Euclid data and the large volume of associated ground based data (e.g. KiDS, DES and LSST).
The Euclid Science Ground Segment consists of the Euclid Science Operations Centre and ten national Science Data Centres. The large data volumes demand that data transfer is minimized and that the processing is taken to the data. This is supported by the Euclid Archive Data Processing System and the Euclid Archive Distributed Data System. The Data Processing System consists of a central metadata repository, which contains the information necessary to process any data item and full data lineage of any data product created. The Distributed Data System provides a cloud solution with a node at each of the national Science Data Centres, which controls data storage and transfer. It supports a large number of storage types, including POSIX, iRODS, gridftp and Xrootd. No limitations are placed on the storage implemented at an individual SDC. Further more, the user of the system needs no knowledge of where data is located. Jobs will be started at the most appropriate locations, or data transferred as necessary. |
|
|
|
|
|
