Petabyte-scale data mining: dream or reality?

Alexander S. Szalay; Jim Gray; Jan VandenBerg

doi:10.1117/12.461427

24 December 2002 Petabyte Scale Data Mining: Dream or Reality?

Alexander S. Szalay, Jim Gray, Jan VandenBerg

Proceedings Volume 4836, Survey and Other Telescope Technologies and Discoveries; (2002) https://doi.org/10.1117/12.461427
Event: Astronomical Telescopes and Instrumentation, 2002, Waikoloa, Hawai'i, United States

Abstract

Science is becoming very data intensive. Today's astronomy datasets with tens of millions of galaxies already present substantial challenges for data mining. In less than 10 years the catalogs are expected to grow to billions of objects, and image archives will reach Petabytes. Imagine having a 100GB database in 1996, when disk scanning speeds were 30MB/s, and database tools were immature. Such a task today is trivial, almost manageable with a laptop. We think that the issue of a PB database will be very similar in six years. In this paper we scale our current experiments in data archiving and analysis on the Sloan Digital Sky Survey data six years into the future. We analyze these projections and look at the requirements of performing data mining on such data sets. We conclude that the task scales rather well: we could do the job today, although it would be expensive. There do not seem to be any show-stoppers that would prevent us from storing and using a Petabyte dataset six years from today.

Citation Download Citation

Alexander S. Szalay, Jim Gray, and Jan VandenBerg "Petabyte Scale Data Mining: Dream or Reality?", Proc. SPIE 4836, Survey and Other Telescope Technologies and Discoveries, (24 December 2002); https://doi.org/10.1117/12.461427

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available