| |
 |
| Paper: |
Running the SkyMapper Science Data Pipeline: To be a Big Fish in a Small Pond, or a Small Fish in a Big Ocean? |
| Volume: |
512, Astronomical Data Analysis Software and Systems XXV |
| Page: |
393 |
| Authors: |
Luvaul, L. C.; Onken, C. A.; Wolf, C.; Smillie, J. G.; Sebo, K. M. |
| Abstract: |
We review structure and frameworks behind the SkyMapper Science Data Pipeline (SDP), and consider the challenges of deploying on two disparate platforms: a publicly shared, massively parallel, queue-scheduled compute fabric, and a dedicated NUMA-based, multi-core, mini-supercomputer. Concepts reviewed include a) how to impose a layer of central operator control over hundreds of jobs of varying type and CPU/IO profile, all running concurrently and at different stages in their logic, b) how to maintain configuration control in an ever-changing algorithmic environment while not giving up ease of build and deployment, and c) how to configure a NUMA-architected machine for optimal cache buffer usage, process-to-memory locality, and user/system CPU cycle ratio. |
|
|
|
|
|
