| |
 |
| Paper: |
ExXmatch: a Tool for Extremely Fast Cross-matches |
| Volume: |
527, Astronomical Data Analysis Software and Systems XXIX |
| Page: |
685 |
| Authors: |
Pineau, F.–X. |
| Abstract: |
Forthcoming projects like Euclid or LSST will produce catalogs containing billions to tens of billions sources possibly distributed on clusters of machines. Satisfying large multiple users queries on such volumes with the lower possible number of machines is a challenge. Possible solutions will probably rely on efficient and scalable algorithms implemented to use as few resources as possible.
ExXmatch has been designed with those constraints in mind. It benefits from both the experience gained by implementing the engine of the CDS Cross-match service and the tests of Big Data tools like Apache Spark. The result is a standalone software able to compute and count the number of matches between Gaia and SDSS in less than 5 (7) seconds with warm (cold) disk cache, on a single regular server (64 GB RAM, 2x10 physical cores, general public SSDs). Implemented in Rust, the process requires a small amount of dedicated memory and does not suffer from any overhead before starting. With a warm (cold) cache, it is able to compute the 1.6 billions associations (at 5 arcsec) between Gaia DR1 and IGSL3 in around 30 (34) seconds. |
|
|
|
|
|
