Big Data is all the rage, a term which quickly became synonymous with diving into the data from social media to discover new insights. As with most things the reality could never live up to the hype because, by design, hype is generated primarily those who are getting educated. Once educated, the hype dies as reality sets it. So what I don't understand is why is everyone still talking about and focusing on Big Data when it's not the goal. Isn't the goal really Real Time Analytics of Big Data?
Sure Big Data is sexy. Gaining insight into what people are thinking and doing en-masse by reading their Twitter updates, Facebook posts, FourSquare check-ins and the like lets companies know what people are doing and saying (not what they are thinking as is often reported - believe it or not many people say things that conflict with their thinking). However as interesting as that information is, the depth of knowledge it takes to glean valuable information supersedes, by definition, the depth of knowledge required to glean the information. It sounds obvious, but gaining the ability to do something is not the same as having the capability of benefiting from doing it. Two quarterbacks throw the ball, but only one wins the game.
It's the benefits of Big Data that should be the focus, not the technology, and so far its benefits have been pretty limited. Am I throwing Big Data under the bus as another disillusioned soul who went on the hype train? No. I'm throwing our overall lack of understanding across businesses of what data is, the conditions under which it can deliver value, how to find the value, and finally realizing the value. We are putting the tools into the hands of people who are not qualified to mine the data. It's not their fault. Nobody has taken the time and effort to train people on data: taxonomy, structure, access, normalization, and on and on. I fancy myself reasonably good at data architecture and yet I know people who can make me look like a fool because it's their focus in life. Who is going to do a better job at surgery? The surgeon or the groundskeeper. We are talking about a seismic shift in the skills mix required in business today. It started back, oh, in the 1970's. As computing spread as a productivity tool we increasingly protected people from having to understand data. We treated data as something too difficult, arcane, or unimportant to understand. Now we are in the Matrix where in order to see the truth you have to be able to look past the "visualizer" (aka application) and at the raw data.
Big Data is forcing a new dialog to occur. In itself Big Data represents a new set of data storage, management, and analysis tools for large data sets. As a result we can now do things we couldn't do before that deliver tremendous value: real time analytics (RTA). RTA is marriage of our exploits in data warehousing and data mining with the ability to traverse nearly unlimited data sets. Real time analysis helps companies keep web sites up and running, car engines running, and planes in the sky. We are in a new evolution of reducing the time it takes to find, access, normalize and filter data. The faster we slice and dice the more rocks we can look under, the more we can learn, and the better we can understand the dynamics of our world.
So in the end it's not about the name. Big Data. Real Time Analytics. Call it whatever you want. However remember that the keys to the kingdom are in understanding the domain, knowing what data is important, and using the right tools to derive the most value in the shortest time.
Dollars follow hype and innovations follow dollars. And that's why I think the Big Data hype is great.