| |
 |
| Paper: |
Pioneering the Exascale Era with Astronomy |
| Volume: |
527, Astronomical Data Analysis Software and Systems XXIX |
| Page: |
295 |
| Authors: |
Oonk, J. B. R.; Schrijvers, C.; van den Berg, Y. |
| Abstract: |
SURF is the collaborative ICT organisation for Dutch education & research and coordinates the national e-infrastructure (surf.nl). To accelerate scientific discovery, SURF invests in, operates and explores high-end IT solutions for and with researchers in the Netherlands. In this ADASS 2019 contribution we present our latest developments in high performance and high throughput cloud computing. These developments are particularly relevant for Astronomy, as this science domain: (i) generates large (Petabyte sized) data collections, (ii) uses rapid release and deployment schemes for their software, (iii) requires flexible and interactive test and staging environments and (iv) needs to execute complex workflows on diverse data structures.
We highlight our new OpenStack-based, cloud infrastructure layer for data processing. This allows us to efficiently build, deploy and scale infrastructure as a service (IaaS) & managed platforms as a service (PaaS) that may be tailored to the specific needs of individual research projects or scientific domains. It also enables dedicated and customizable use of IT resources by different user groups. Currently SURF supports several PaaS solutions for data processing. Examples are, our Grid processing facility, our Kubernetes cluster deployments, and Spider, our new addition to SURF's high throughput data processing (HTDP) platforms.
Here we focus on Spider, offering HTDP users a rich ecosystem and enabling both batch and interactive data processing at scale. Spider is seamlessly integrated with a large variety of storage systems and tightly coupled with related services via a powerful EVPN network. Modern IT ecosystems, such as Spider, that are created from re-usable building blocks are particularly relevant for the current and next generation of data-intensive astronomical experiments and the diverse need of their users. These experiments include, but are not limited to, the Square Kilometre Array (SKA), the Low Frequency Array (LOFAR) and the Atacame Large Millimetre Array (ALMA). |
|
|
|
|
|
