This story was written by Keith Dawson for UBM DeusM’s community Web site Business Agility, sponsored by IBM. It is archived here for informational purposes only because the Business Agility site is no more. This material is Copyright 2012 by UBM DeusM.

Limits of Big-Data Analytics

Why big data analytics must heed context change, complexity, and overconfidence.

Those intent on mining unstructured "big data" for business intelligence will do well to familiarize themselves with the inherent and human limitations of that approach.

Strategist Daniel W. Rasmus has a long and thought-provoking piece in Fast Company titled Why Big Data Won't Make You Smart, Rich, Or Pretty. It spells out some of the factors data modelers need to be aware of when trying to make sense of vast swaths of data that may come from multiple sources around the company or around the Internet.

We have extensively covered the promise, and the nuts and bolts, of the coming wave of big-data analytics (see the related posts below). Cassimir Medford wrote about two obstacles in the road to the big-data future -- a shortage of mathematically trained, analytically inclined workers, and the difficult of locating the right hardware and software tools. These are problems of a different order than the fundamental (some might say existential) ones with which Rasmus concerns himself. Let's look at a few of these.

Rasmus outlines a handful of other existential difficulties in the business of big-data analytics, including the modelers' motives, feedback loops, complexity, and the unpredictable effects on the model itself of acting on computed intelligence. Go read his article. You'll be glad you did.

Few doubt that big-data analytics will be a large and growing trend from here forward. It's important to recognize what kinds of questions for which the technology will provide reliable answers, and where its limitations lie. "In the end," Rasmus writes, "we may not be equipped to apply the technology in a meaningful and safe way at scales that outstrip our ability to represent, understand, and validate the models and their data."

Related posts: