The Next Wave of Information Overload is Here

According to a first-ever IDC/EMC Corp. survey released in early March, the information explosion we’ve all become so accustom to is about to take off so fast new mathematical terms (think exabyte, which is equal to a billion gigabytes) will have to enter the popular lexicon to account for it all.

During 2006, the “digital universe,” i.e., all the bits and bytes created by everyone, everywhere, was 161 billion gigabytes or 161 exabytes. Over the next four years, IDC predicts that number to grow six-fold every year for a 57% CAGR.

To get an idea of just how much information is flowing around globe right now, 161 exabytes is “approximately three million times the information in all the books ever written,” according to IDC.

“I don’t think the number in of itself is really a surprise, said Steve Minton, VP of Worldwide IT Markets and Strategies within IDC’s Global Research Organization and co-author of the report.

“In a sense, this is really just putting a number on what people would have told you before. The bigger discussion is really around all the implications around this and what it means and what they need to do about dealing with all this data.”

And that’s the rub. While most information is being created by individuals, some 70%, 85% of that information is being touched by organizations, corporations and government bodies in some form or another. This then raises a whole host of questions over responsibility for the information and, therefore, how to deal with it.

“If it touches the organizations network anywhere then they can be held liable from a compliance point of view, from a legal standpoint,” said Minton. “If that information is crossing their network … then their responsible for it.”

The Growth of ILM & BI

In an attempt to get a handle on all this data, corporations have turned to information lifecycle management (ILM) concepts and tools, and business intelligence (BI) suites. Which accounts for those two market segment’s rapid growth over the past couple of years, said Minton.

But these two technologies are still, relatively speaking, in their infancies, said James Short, research director at the Information Storage Industry Center (ISIC) of the University of California, San Diego.

“ILM is the first step” towards a solution, said Short. “It’s essentially a collection of a series of hardware and software innovations in storage that allow for a more economic utilization of both the hardware and storage medium as well as a first phase attempt at defining policies that will allow you a more policy-based storage approach in your IT organization.”

Also, since the biggest problem with so much data isn’t necessarily storing it or the cost of storing it — Moore’s Law has seen to that — it’s deciding what information is valuable and what it not, said Roger Bohn, ISIC’s director. The volume only exacerbates this problem.

“Somewhere along the way … classification of the value of that information has to be reconciled before the IT organization can make an economic system for storing that information,” said Bohn.

Tech Solutions

While the vendor community works feverishly to develop the needed automation to tackle such a vast problem, their attempts so far have only begun to scratch the surface, said Short.