| |
 |
| Paper: |
FITS Tile Compression in the NOAO DMS |
| Volume: |
411, Astronomical Data Analysis Software and Systems XVIII |
| Page: |
497 |
| Authors: |
Stobie, E.; Seaman, R.; Barg, I. |
| Abstract: |
The NOAO Data Management system (DMS) captures data from eleven NOAO and partner telescopes and transports these data from three mountaintops to replicate them between three data centers both North and South of the equator. Image files are annotated, remediated, ingested, and persisted through interfaces of the NOAO Science Archive. Wide-field optical and infrared images flow out of the archive, through the NOAO High Performance Pipeline creating several new data products that flow back into the archive. Raw, pipeline-reduced, and survey data products, both proprietary and postproprietary, are made available through the NOAO Portal using VO standards and services.
Each of these several steps requires access to both image data and metadata in the form of image header keywords. Measures of storage efficiency and throughput characterize performance, cost, schedule, and risk in a matrix across all telescopes and all subsystems. Anything that impedes access to data or metadata diminishes throughput, thus slowing schedules, increasing costs, revealing risks, and adversely affecting performance.
The familiar gzip compression algorithm is often used to increase data storage efficiency. However, gzip actually reduces throughput due to initial and recurring overhead of compression and later uncompression. For example, if metadata for an image require remediation, the whole image must be compressed, uncompressed, and compressed again. By contrast, the FITS tile convention using the Rice algorithm achieves about 40% better compression than gzip in just one-third the time. Image headers remain readable such that images often need never be uncompressed at all; metadata can be simply edited in place. Further, a library such as CFITSIO can support tile compression as a native image format. The pixel tiling feature means that for applications such as a cutout service, only the tiles overlapping the desired image section need be uncompressed. |
|
|
|
|
|
