| |
 |
| Paper: |
The NOAO Data Cache Initiative - Building a Distributed Online Datastore |
| Volume: |
347, Astronomical Data Analysis Software and Systems XIV |
| Page: |
679 |
| Authors: |
Seaman, R.; Barg, I.; Zarate, N.; Smith, C.; Saavedra, N. |
| Abstract: |
The Data Cache Initiative (DCI) of the NOAO Data Products Program is a prototype Data Transport System for NOAO and affiliate facilities. DCI provides pre-tested solutions for conveying data from our large suite of instrumentation to a central mountain data cache. The heart of DCI is an extension of the Save-the-Bits safestore, running for more than a decade (more than 4 million images saved, comprising more than 40 Tbytes). The iSTB server has been simplified by the removal of STB's media handling functionality, and iSTB has been enhanced to remediate each incoming header with information from a database of NOAO instrumentation and an interface to the NOAO proposal database. Each mountain data cache has been implemented on commodity hardware running Redhat 9.0. Software RAID 1 runs over hardware RAID 5 to provide maximum storage reliability for each copy of the data. Each image is transferred from Kitt Peak or Cerro Tololo to the corresponding datastore at the Tucson or La Serena data centers using an rsync-based queue adopted from NCSA. From each data center, the files are transported to the other NOAO data center and also to NCSA for off-site storage using the Storage Resource Broker (SRB) of the San Diego Supercomputer Center. Thus we have three copies of each file on spinning disks or near-online. Major institutional users will be given access to the datastores. |
|
|
|
|
|
