Data Mining To Transform Bytes Into Bucks

Data mining is looking to make sense of massive quantities of information and present results in human friendly format

Every day the an estimated 2.5 quintillion (10 to the 18) bytes of data pass through the internet. Streaming videos, emails, web postings, social media updates, GPS signals, VOIP phone calls, digital photos and more are moving from each of us to everyone else. Each on-line purchase, music download or search is part of the “big data” that makes up the total of information on the world-wide-web.

Many companies are working on solutions to turn this data from bytes into bucks. The issue is there is so much data, and so little useful information contained within. Only by taking each of the individual pieces of the big data and correlating it can relevant patterns be determined. To do this, we need very complex and very specific algorithms crunching data on high performance server farms.

Imagine for example the relatively simple process that a video streaming company must employ to suggest titles based on only your previous viewing history. More complex, and some would argue vital, is analyzing cell phone calls and emails to monitor for illegal activities. And somewhere in between are sports teams which can record, replay and analyze the details of every aspect of each players performance by game, venue or time-line.

What’s really needed is a form of artificial intelligence which can import all of the random pieces of information and using a simple query interface the export the quantified results. This is the process of taking big, dumb data sets and transforming them into small, smart sub-sets.

Quite a few companies have been implementing data-mining internally to uncover new revenue streams. Other ventures are in the business of selling data-mining tools and software. IBM acquired i2 last year to bolster its capabilities in this area.

Core to i2 was technology for crime prevention and data intelligence gathering. IBM will use the i2 tools for its analytics in both public and corporate security and safety. Another company which recently went public is Splunk which can troubleshoot, monitor and analyze cloud applications in real time.

Palantir (named for the “seeing stones” in “The Lord Of The Rings”) software crunches tremendous amounts of data to produce visual representations which are easily usable in understanding the underlying information. They literally let clients see into their data, and make sense of seemingly unconnected pieces of intelligence.

The amount of data out there is growing (see related article Data Storage Technologies ), but the tools available to understanding it are getting better.