| |
 |
| Paper: |
Reliable, Automatic Transfer and Processing of Large Scale Astronomy Datasets |
| Volume: |
347, Astronomical Data Analysis Software and Systems XIV |
| Page: |
277 |
| Authors: |
Kosar, T.; Kola, G.; Livny, M.; Brunner, R.J.; Remijan, M. |
| Abstract: |
Astronomers are increasingly obtaining larger datasets, particularly in the optical and near-infrared. Unfortunately, the technologies to process large amounts of image data and share the data and the results with collaborators spread around the globe, have not kept pace with the data flow. In the past, this type of software has required significant human involvement to deal with failures. We have designed and implemented a fault-tolerant system that can process large amounts of astronomy images using idle CPUs on desktops, commodity clusters and grid resources. It reliably replicates data and results to collaborating sites and performs on-the-fly optimization to improve throughput. It is highly resilient to failures and can recover automatically from network, storage server, software and hardware failures. To demonstrate the capabilities of this framework, we have successfully processed three terabytes of DPOSS images using idle grid resources spread across three organizations. |
|
|
|
|
|
