| |
 |
| Paper: |
DTS: The NOAO Data Transport System |
| Volume: |
485, Astronomical Data Analysis Software and Systems XXIII |
| Page: |
109 |
| Authors: |
Fitzpatrick, M.; Semple, T. |
| Abstract: |
The NOAO Data Transport System (DTS) provides high-throughput, reliable,
data transfer between telescopes, pipelines and archive centers located in
the Northern and Southern hemispheres. It is a distributed application
using XML-RPC for command and control, and either parallel-TCP or UDT
protocols for bulk data transport. The system is data-agnostic, allowing
arbitrary files or directories to be moved using the same infrastructure.
Data paths are configurable in the system by connecting nodes as the
source or destination of data in a queue. Each leg of a data path may be
configured independently based on the network environment between the
sites. A queueing model is currently implemented to manage the automatic
movement of data, a streaming model is planned to support arbitrarily
large transfers (e.g. as in a disk recovery scenario) or to provide a
'pass-thru' interface to minize overheads. A web-based monitor allows
anyone to get a graphical overview of the DTS system as it runs, operators
will be able to control individual nodes in the system. Through careful
tuning of the network paths DTS is able to achieve in excess of 80-percent
of the nominal wire speed using only commodity networks, making it ideal for
long-haul transport of large volumes of data. |
|
|
|
|
|
