hn.today

Big Data Is Dead

motherduck.com16 points1 comments
Screenshot of Big Data Is Dead

The claim that “big data” is the central obstacle to making better decisions has outlived its usefulness. For years, technology vendors warned that exploding data volumes would overwhelm traditional systems and urged organizations to adopt platforms built for massive scale. But data has not grown as dramatically as predicted, while hardware and conventional databases have improved. Drawing on more than a decade of experience building BigQuery and analyzing customer usage, Jordan Tigani argues that many organizations still struggle to use their data even after buying large-scale tools, because data volume was never the main problem.

The evidence suggests most organizations hold modest amounts of analytical data: the vast majority of BigQuery customers had less than a terabyte, and heavily active customers had a median well below 100 GB. Industry feedback put a typical warehouse around 100 GB; even large technology companies often had only a few terabytes, while many had far less. Data sizes follow a steep power-law distribution, with a few extreme cases masking the norm. Ordinary business activity also generates data slowly: a medium-sized company’s orders might take years to reach a gigabyte. The argument distinguishes storage growth from computing needs. Data accumulates over time, but most analysis concerns recent records, so compute requirements often stay relatively stable. Cloud separation of storage and compute makes retaining large archives practical without requiring equally large processing capacity. The practical priority, then, is using data effectively - not scaling for hypothetical volume.

Read on motherduck.com1 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Programming

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.