The Era of Big Data

Big data has became an hot IT buzzword shaking things up in the high-performance computing segment of the IT world. It definitely have deep implications on how the information business continues to evolve, thus creating the potential for further disruption. So what is big data, anyway and why should we even care?

According to Wikipedia, the free encyclopedia, big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. Big data exceeds the processing capacity of conventional database systems. According to Gartner, it can also be defined as high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. Data can be text and numbers but can also include maps, images and videos.

Due to the recent technological advances, the types of big data that can be harnessed and stored have expanded. Also the rules of data accessibility are changing as more people now have more access to these data via public domain resources like data.gov (US gov) which permits anyone with an Internet connection to view and download large datasets on subjects. These subjects range from local unemployment statistics in the US and the rates of depression by the US census tract to the recent natural disasters activities. The government is making data public at both the national, state, and city level for users to develop new applications that can generate public good.

According to IBM, 2.5 quintillion bytes of data are generated daily and 90 percent of the total data in existence was created within the last two years. Wal-Mart, a retail giant, handles more than 1m customer transactions every hour, feeding databases estimated at more than 2.5 petabytes-the equivalent of 167 times the books in America’s Library of Congress. Twitter handles over 230 million tweets each day. Facebook has over 30 petabytes of user-generated data stored houses up to more than 250 billion photos uploaded to their site which currently averages 350 millions uploads per day, according to a white paper shared by Facebook. Google roughly has 7.2 billion daily page views with 87.8 billion monthly worldwide searches conducted on Google sites. Currently over 50% of US population owns a smartphone and it is projected that there will be 10 billion mobile devices in use worldwide by 2020. All these have been made possible with recent technological breakthroughs and is the reason why it has become important and relevant to business, schools, governments and other establishments.

While many large businesses have decided to jump on the bandwagon of big data, most small businesses have slowly approached and adopted this modern phenomenon. In a recent study conducted by Lee Bonger, a Marketing Technologist and a Professor of eCommerce, only 23% of the organizations they assessed in their study have an enterprise-wise Big Data strategy. It is important to note that big data can be used in organizations of all sizes-and in all industries and not just for big companies. It is not a fad, a luxury or just noise. Businesses like Apple, Facebook, Google, Twitter, and others have shown that it is here for the long term.It is not yet complete, neat, orderly or perfect because of the new paradigm making old tools like relational databases and SQL statements not able to work well with petabytes of unstructured data.

There is no doubt we are in the era of big data. Is your organization using big data? How do you think it is impacting your business or industry?