| Abstract: |
The size and complexity of data from new observatories continue to bring science platforms into the spotlight. The Canadian Advanced Network for Astronomical Research (CANFAR) began in 2008 and has gone through many changes, none more significant than the transition from virtual machines to containers in 2018, which resulted in the CANFAR Science Platform. With this transition came several challenges and shifts in our thinking about how an astronomy platform can be built.
CANFAR is a general-purpose astronomy platform used by projects from a variety of facilities such as the James Webb Space Telescope (JWST), the Atacama Large Millimeter Array (ALMA), the Rubin C. Vera Observatory, and many others. One of the difficulties users face in the original virtual machine-based cloud is the steep technical learning curve to get a simple science function operating. The model is now quite different: users are offered a choice of a variety of ready-to-use containers. We provide and maintain a small core set, but the user community has contributed the majority of containers. Shifting this responsibility has allowed us to focus on platform-wide improvements that benefit all projects. The community brings value to the platform by providing shared astronomy software containers.
A science container is in one of several categories: Jupyter Notebooks, CARTA (Cube Analysis and Rendering Tool for Astronomy), X11 Desktops (ARCADE), and another general web interface category. These allow for the accounting of a small set of differing requirements. When launched, a science container becomes an active session that users interact with through their browser. When using an ARCADE Desktop session, a variety of other non-web-based containers can be run, whose displays are attached and displayed on the desktop. A user-managed image registry hosts the set of containers that are available to run on the platform.
This self-serve model is a common theme in the platform. Project leads create groups representing their teams. Using those groups gives teams access to the software images and the shared, distributed storage available in containers. Our participation in the International Virtual Observatory Alliance (IVOA) has driven many of these concepts.
Though we feel we have a sound and sustainable model and architecture, our efforts have not been without challenges. Adopting Kubernetes was difficult but ultimately well worth it. Also, some usability problems are still difficult, such as efficient resource management and storage latency. We have a long list of features and improvements to make, including support for different types of batch processing.
|
