The MWA archive infrastructure: archiving terabytes of data over dedicated WAN connections

Andreas Wicenec; Dave Pallot; Alessio Checcucci; Slava Kitaeff; Kevin Vinsen; Chen Wu

doi:10.1117/12.925901

24 September 2012 The MWA archive infrastructure: archiving terabytes of data over dedicated WAN connections

Andreas Wicenec, Dave Pallot, Alessio Checcucci, Slava Kitaeff, Kevin Vinsen, Chen Wu

Proceedings Volume 8451, Software and Cyberinfrastructure for Astronomy II; 845118 (2012) https://doi.org/10.1117/12.925901
Event: SPIE Astronomical Telescopes + Instrumentation, 2012, Amsterdam, Netherlands

Abstract

The Murchison Wide Field Array (MWA) is being upgraded from 32 tiles to 128 tiles of 16 dual-polarization dipole antennas. In the course of this project the software and the data infrastructure are also undergoing a major overhaul in order to cope with a more continuous and remote operational model; and the substantial increase in data rate (400 MB/s from the correlator plus 160 MB/s from the real time imaging pipeline). During the course of 2012/13 the data collected by the MWA will be transported via a dedicated 40 Gbit WAN network link between the Murchison Radio Observatory (MRO) and Perth (700 km). However, this network will not be available for some time; and until then, the data will be transported using disk arrays instead. Once in Perth, the data will be ingested into a tape library. The archiving process itself consists of various steps executed either at the MRO site or in Perth, and makes use of a modied version of the archiving system from the Atacama Large Millimeter Array (ALMA). This includes the extraction of meta-data from the original raw data and the ingestion and generation of the appropriate data links for the MWA archive. The MWA correlator generates a collection of small les all belonging to the same observation. In order to optimise the network transfer and the storage of those les they are transparently packed into larger containers at the MRO site and subsequently handled as one big le. This paper describes the setup of the MWA archiving infrastructure.

Citation Download Citation

Andreas Wicenec, Dave Pallot, Alessio Checcucci, Slava Kitaeff, Kevin Vinsen, and Chen Wu "The MWA archive infrastructure: archiving terabytes of data over dedicated WAN connections", Proc. SPIE 8451, Software and Cyberinfrastructure for Astronomy II, 845118 (24 September 2012); https://doi.org/10.1117/12.925901

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available