Information has gone from scarce to superabundant. That brings huge new benefits, says Kenneth Cukier (interviewed here)—but also big headaches

Data, data everywhere

Data, data everywhere | Special report | The Economist

data

computer

big data

Microsoft

SAP SE

business enterprise

Google

website

data stream

programmatic_custom_mhi

future_of_work_test

ts_tech

gs_tech_computing

ibm_cloud

gs_tech

ge_tech_enthusiasts

gs_interest_male

gs_interest_frequent_travelers

ibm_blacklist

neg_omd_exclusion

gv_safe

The American-led financial order is giving way to a more divided one

Worlds apart

A conflict could deal a death blow to the global financial order

How the financial system would respond to a superpower war

How crises reshaped the world financial system

Attempts to challenge the dollar have only strengthened its dominance

The fight to dethrone the dollar

That spurs competition, innovation—and cross-border frictions

National payment systems are proliferating

Special report

science and technology

IT/computer sciences

computing and information technology

economy, business and finance

All too much

A different game

Clicking for gold

The open society

Show me

Needle in a haystack

New rules for big data

Handling the cornucopia

gb_safe_from_high

gb_safe_from_high_med

pos_goldman_inclusion_2024

pos_israel-hamas_war

gs_tech_compute

gs_tech_compute_net

gs_busfin

artificial_intelligence

america_department_commerce

neg_facebook

cigna_healthyhybridworkplace

gt_mixed

gs_science_misc

chanel_neg

custom_punkt

gs_business

gs_busfin_indus

it_decisionmakers

business_it_decisionmakers

gs_health

gs_health_misc

neg_goldmansachs_brandsafety

neg_google_youtube_2020

cigna_puttingfamilyfirst

gs_tech_compute_net_support

healthcare_related_topics

gs_economy

gs_economy_misc

it_professionals

neg_3166_vca_brand-safety

fow_barclays

test_mba

gs_science

america_federal_reserve

ssga_sp400

gs_busfin_business

fidelity_investment

investment_banking_personal_finance

gs_business_misc

neg_jpm_2024q4

neg_morgan_stanley_2019_neg_keywords

gt_positive_curiosity

legal_law

neg_dit4

google_negative_keywords

gs_tech_compute_net_search

gb_crime_high_med_low

universal_negative_keywords

neg_fidelity_gmo

test

gs_tech_compute_apps

neg_ey_brandsafety

neg_ey_kwbl

neg_ibm_brandsafety

bayer_investment

thought_leader

gs_auto

neg_dit

gs_politics

neg_3166_vca_brand-safety3

business_education

america_department_health

covid_bhp

WHEN the Sloan Digital Sky Survey started work in 2000, its telescope in New Mexico collected more data in its first few weeks than had been amassed in the entire history of astronomy. Now, a decade later, its archive contains a whopping 140 terabytes of information. A successor, the Large Synoptic Survey Telescope, due to come on stream in Chile in 2016, will acquire that quantity of data every five days. 

Such astronomical amounts of information can be found closer to Earth too. Wal-Mart, a retail giant, handles more than 1m customer transactions every hour, feeding databases estimated at more than 2.5 petabytes—the equivalent of 167 times the books in America's Library of Congress (see article for an explanation of how data are quantified). Facebook, a social-networking website, is home to 40 billion photos. And decoding the human genome involves analysing 3 billion base pairs—which took ten years the first time it was done, in 2003, but can now be achieved in one week. 

Such astronomical amounts of information can be found closer to Earth too. Wal-Mart, a retail giant, handles more than 1m customer transactions every hour, feeding databases estimated at more than 2.5 petabytes—the equivalent of 167 times the books in America's Library of Congress (see <a href="https://www.economist.com/special-report/2010/02/27/all-too-much">article</a> for an explanation of how data are quantified). Facebook, a social-networking website, is home to 40 billion photos. And decoding the human genome involves analysing 3 billion base pairs—which took ten years the first time it was done, in 2003, but can now be achieved in one week.

All these examples tell the same story: that the world contains an unimaginably vast amount of digital information which is getting ever vaster ever more rapidly. This makes it possible to do many things that previously could not be done: spot business trends, prevent diseases, combat crime and so on. Managed well, the data can be used to unlock new sources of economic value, provide fresh insights into science and hold governments to account. 

But they are also creating a host of new problems. Despite the abundance of tools to capture, process and share all this information—sensors, computers, mobile phones and the like—it already exceeds the available storage space (see chart 1). Moreover, ensuring data security and protecting privacy is becoming harder as the information multiplies and is shared ever more widely around the world. 

Alex Szalay, an astrophysicist at Johns Hopkins University, notes that the proliferation of data is making them increasingly inaccessible. “How to make sense of all these data? People should be worried about how we train the next generation, not just of scientists, but people in government and industry,” he says. 

“We are at a different period because of so much information,” says James Cortada of IBM, who has written a couple of dozen books on the history of information in society. Joe Hellerstein, a computer scientist at the University of California in Berkeley, calls it “the industrial revolution of data”. The effect is being felt everywhere, from business to science, from government to the arts. Scientists and computer engineers have coined a new term for the phenomenon: “big data”.

Epistemologically speaking, information is made up of a collection of data and knowledge is made up of different strands of information. But this special report uses “data” and “information” interchangeably because, as it will argue, the two are increasingly difficult to tell apart. Given enough raw data, today's algorithms and powerful computers can reveal new insights that would previously have remained hidden.

The business of information management—helping organisations to make sense of their proliferating data—is growing by leaps and bounds. In recent years Oracle, IBM, Microsoft and SAP between them have spent more than $15 billion on buying software firms specialising in data management and analytics. This industry is estimated to be worth more than $100 billion and growing at almost 10% a year, roughly twice as fast as the software business as a whole. 

Chief information officers (CIOs) have become somewhat more prominent in the executive suite, and a new kind of professional has emerged, the data scientist, who combines the skills of software programmer, statistician and storyteller/artist to extract the nuggets of gold hidden under mountains of data. Hal Varian, Google's chief economist, predicts that the job of statistician will become the “sexiest” around. Data, he explains, are widely available; what is scarce is the ability to extract wisdom from them.

There are many reasons for the information explosion. The most obvious one is technology. As the capabilities of digital devices soar and prices plummet, sensors and gadgets are digitising lots of information that was previously unavailable. And many more people have access to far more powerful tools. For example, there are 4.6 billion mobile-phone subscriptions worldwide (though many people have more than one, so the world's 6.8 billion people are not quite as well supplied as these figures suggest), and 1 billion-2 billion people use the internet.

Moreover, there are now many more people who interact with information. Between 1990 and 2005 more than 1 billion people worldwide entered the middle class. As they get richer they become more literate, which fuels information growth, notes Mr Cortada. The results are showing up in politics, economics and the law as well. “Revolutions in science have often been preceded by revolutions in measurement,” says Sinan Aral, a business professor at New York University. Just as the microscope transformed biology by exposing germs, and the electron microscope changed physics, all these data are turning the social sciences upside down, he explains. Researchers are now able to understand human behaviour at the population level rather than the individual level.

The amount of digital information increases tenfold every five years. Moore's law, which the computer industry now takes for granted, says that the processing power and storage capacity of computer chips double or their prices halve roughly every 18 months. The software programs are getting better too. Edward Felten, a computer scientist at Princeton University, reckons that the improvements in the algorithms driving computer applications have played as important a part as Moore's law for decades.

A vast amount of that information is shared. By 2013 the amount of traffic flowing over the internet annually will reach 667 exabytes, according to Cisco, a maker of communications gear. And the quantity of data continues to grow faster than the ability of the network to carry it all. 

People have long groused that they were swamped by information. Back in 1917 the manager of a Connecticut manufacturing firm complained about the effects of the telephone: “Time is lost, confusion results and money is spent.” Yet what is happening now goes way beyond incremental growth. The quantitative change has begun to make a qualitative difference. 

This shift from information scarcity to surfeit has broad effects. “What we are seeing is the ability to have economies form around the data—and that to me is the big change at a societal and even macroeconomic level,” says Craig Mundie, head of research and strategy at Microsoft. Data are becoming the new raw material of business: an economic input almost on a par with capital and labour. “Every day I wake up and ask, ‘how can I flow data better, manage data better, analyse data better?” says Rollin Ford, the CIO of Wal-Mart.

Sophisticated quantitative analysis is being applied to many aspects of life, not just missile trajectories or financial hedging strategies, as in the past. For example, Farecast, a part of Microsoft's search engine Bing, can advise customers whether to buy an airline ticket now or wait for the price to come down by examining 225 billion flight and price records. The same idea is being extended to hotel rooms, cars and similar items. Personal-finance websites and banks are aggregating their customer data to show up macroeconomic trends, which may develop into ancillary businesses in their own right. Number-crunchers have even uncovered match-fixing in Japanese sumo wrestling. 

“Data exhaust”—the trail of clicks that internet users leave behind from which value can be extracted—is becoming a mainstay of the internet economy. One example is Google's search engine, which is partly guided by the number of clicks on an item to help determine its relevance to a search query. If the eighth listing for a search term is the one most people go to, the algorithm puts it higher up.

As the world is becoming increasingly digital, aggregating and analysing data is likely to bring huge benefits in other fields as well. For example, Mr Mundie of Microsoft and Eric Schmidt, the boss of Google, sit on a presidential task force to reform American health care. “Early on in this process Eric and I both said: ‘Look, if you really want to transform health care, you basically build a sort of health-care economy around the data that relate to people',” Mr Mundie explains. “You would not just think of data as the ‘exhaust' of providing health services, but rather they become a central asset in trying to figure out how you would improve every aspect of health care. It's a bit of an inversion.”

To be sure, digital records should make life easier for doctors, bring down costs for providers and patients and improve the quality of care. But in aggregate the data can also be mined to spot unwanted drug interactions, identify the most effective treatments and predict the onset of disease before symptoms emerge. Computers already attempt to do these things, but need to be explicitly programmed for them. In a world of big data the correlations surface almost by themselves. 

Sometimes those data reveal more than was intended. For example, the city of Oakland, California, releases information on where and when arrests were made, which is put out on a private website, Oakland Crimespotting. At one point a few clicks revealed that police swept the whole of a busy street for prostitution every evening except on Wednesdays, a tactic they probably meant to keep to themselves.

But big data can have far more serious consequences than that. During the recent financial crisis it became clear that banks and rating agencies had been relying on models which, although they required a vast amount of information to be fed in, failed to reflect financial risk in the real world. This was the first crisis to be sparked by big data—and there will be more. 

 The way that information is managed touches all areas of life. At the turn of the 20th century new flows of information through channels such as the telegraph and telephone supported mass production. Today the availability of abundant data enables companies to cater to small niche markets anywhere in the world. Economic production used to be based in the factory, where managers pored over every machine and process to make it more efficient. Now statisticians mine the information output of the business for new ideas. 

“The data-centred economy is just nascent,” admits Mr Mundie of Microsoft. “You can see the outlines of it, but the technical, infrastructural and even business-model implications are not well understood right now.” This special report will point to where it is beginning to surface. 

Get to grips with a region where capital, climate and geopolitics collide

Go deeper with our Middle East newsletter. Every week our correspondents delve into the political, social and economic stories of a region that is rarely out of the news.

Sign up to our Middle East newsletter

Middle East Dispatch

Follow the action in Latin America, from the depths of the Brazilian Amazon to the Mexican-American border

Every week our correspondents share insights on politics, economics and business from an increasingly complex region

Stay informed with our weekly Latin America newsletter

El Boletín

A sharp round-up of the most important US political news

Sign up for updates from our correspondents and data journalists, poll tracker and election-forecast model as the 2024 race unfolds

The US in brief

Inside Europe, from the front lines in Ukraine to the corridors of power in Brussels

Stay informed with our weekly Europe newsletter

Café Europa

Understand a vast and complicated country that will increasingly shape the future

Stay informed with our weekly India newsletter

Essential India

The best of The Economist’s defence coverage, in one place

The War Room

A weekly conversation about culture. Because culture is a serious business

Exploring trends and connections across entertainment, the arts and politics

Broaden your perspective with our weekly culture newsletter

Plot Twist

Analysing the challenges facing Britain and what needs to be done to overcome them

Stay informed with our weekly Britain newsletter

Blighty

Understand what the world makes of China—and what China makes of the world

Stay informed with our weekly China newsletter

Drum Tower

Your essential guide to global business and technology

From supply chains to semiconductors, The Bottom Line newsletter has you covered

Need to track the megatrends shaping business and technology?

The Bottom Line

Catch up quickly on the global stories that matter

Sign up to enjoy a mind-expanding mix of stories, delivered six days a week

Catch up quickly on the stories that matter

The World in Brief

A behind-the-scenes look at how we chose and designed this week’s cover

How we chose this week’s cover image

Cover Story

Exclusive insight from our correspondents in America

Exclusive insight and reading recommendations from our correspondents in America

Checks and Balance

A weekly fix of our mind-expanding science coverage

Curious about the world? Enjoy a weekly fix of our mind-expanding science coverage

Simply Science

Climate-change analysis that you won’t read elsewhere in The Economist

The Climate Issue

Expert analysis of the biggest stories in economics and markets

Money Talks

Taking you behind the scenes of our data journalism

Off the Charts

Long reads and life from 1843 magazine

The Extraordinary Story

The very best of our journalism, handpicked for you each weekday

A daily newsletter with the best of our journalism

Handpicked stories, in your inbox

The Economist today

Highlights from the latest weekly issue, introduced by our editor

Data, data everywhere

Information has gone from scarce to superabundant. That brings huge new benefits, says Kenneth Cukier (interviewed here)—but also big headaches

From the February 27th 2010 edition