| |
 |
| Paper: |
Column-Oriented Table Access Using STIL: Fast Analysis of Very Large Tables |
| Volume: |
394, Astronomical Data Analysis Software and Systems (ADASS) XVII |
| Page: |
422 |
| Authors: |
Taylor, M.B.; Page, C.G. |
| Abstract: |
By use of column-oriented storage and file mapping, great improvements in efficiency over more conventional methods can be made for some important kinds of access to large and very large tabular datasets. These techniques have been implemented in the STIL library, enabling their use in the established table analysis applications TOPCAT and STILTS. Benchmarks are presented which show certain common analysis tasks running 10–40 times faster than their MySQL equivalents. Applied to datasets in the range hundreds of Mbyte to hundreds of Gbyte this speedup can be put to good use both on the desktop and at the data center to bring new regimes of data exploration within practical reach. |
|
|
|
|
|
