| |
 |
| Paper: |
Empowering SKA Data Challenges: A Homogeneous Platform for Enhanced Collaboration and Scalability Fully Aligned with Open Science |
| Volume: |
541, ADASS XXXIII |
| Page: |
361 |
| Authors: |
Manuel Parra-Royón; James Collinson; Jesús Sánchez-Castañeda; Pablo Llopis-Sanmillán; Carolina Lindqvist; MA Mendoza; Susana Sánchez-Expósito; Laura Darriba; Javier Moldón; Julián Garrido; Jesús Salgado; Lourdes Verdes-Montenegro |
| DOI: |
10.26624/PONQ1847 |
| Abstract: |
The Square Kilometre Array Observatory (SKAO) is leading a collaborative effort to build and operate an advanced radio telescope. The SKAO Science Data
Challenges (SDCs) are competitions aimed at fostering innovative data analysis techniques for the substantial data expected from the SKAO. Traditionally, these challenges
have relied on computing resources from various scientific institutions, resulting in an
uneven user experience due to differing configurations of Virtual Machines (VMs) or
high-performance computing (HPC) resources. To address this, our proposal suggests
implementing a uniform platform for computing and storage resources to ensure fairness, scalability, and consistent collaboration among teams. This homogeneous setup
would simplify resource management, support, and evaluation, ultimately enhancing
efficiency and producing dependable results. In this context, JupyterHub facilitates the
provisioning of compute resources via Kubernetes, offering user demand scaling and
centralized authentication. We present the configuration process of a Kubernetes cluster and the setup for BinderHub/JupyterHub, along with a practical use case demonstrating data analysis and workflow in radio astronomy, leveraging Dask for parallel
and distributed computing on large cloud-based Kubernetes clusters. To test this environment we have deploy it in several nodes within the SRCNet cloud platforms, ESP-SRC (Spain), CHSRC (Switzerland) and SKAO (UK), along with the resources of the
CESGA supercomputing center (Spain). |
|
|
|
|
|
