KEYWORDS: Image processing, Data processing, Data storage, Data modeling, Charge-coupled devices, Photometry, Calibration, Stars, Distributed computing, Data compression
The scientific community is in the midst of a data analysis crisis. The increasing capacity of scientific CCD
instrumentation and their falling costs is contributing to an explosive generation of raw photometric data. This data must
go through a process of cleaning and reduction before it can be used for high precision photometric analysis. Many
existing data processing pipelines either assume a relatively small dataset or are batch processed by a High Performance
Computing centre. A radical overhaul of these processing pipelines is required to allow reduction and cleaning rates to
process terabyte sized datasets at near capture rates using an elastic processing architecture. The ability to access
computing resources and to allow them to grow and shrink as demand fluctuates is essential, as is exploiting the parallel
nature of the datasets. A distributed data processing pipeline is required. It should incorporate lossless data compression,
allow for data segmentation and support processing of data segments in parallel. Academic institutes can collaborate and
provide an elastic computing model without the requirement for large centralized high performance computing data
centers. This paper demonstrates how a base 10 order of magnitude improvement in overall processing time has been
achieved using the "ACN pipeline", a distributed pipeline spanning multiple academic institutes.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.