This story was written by Keith Dawson for UBM DeusM’s community Web site Business Agility, sponsored by IBM. It is archived here for informational purposes only because the Business Agility site is no more. This material is Copyright 2012 by UBM DeusM.

Staffing for Big-Data Analytics

Data scientists are in short supply; but maybe you don't need that many of them.

How many data scientists do you need to add as BI meets big-data? Fewer than you think; an 80/20 rule may apply to big-data analytics staffing.

A few months back we took a look at some of the considerations involved in building out an analytics team as the enterprise begins to grapple with the big-data explosion. That blog post was informed by an article written by one of the fathers of data science -- in fact DJ Patil is one of the two people credited with coining the term.

Data scientists are something like the mythical unicorn: rare and beautiful. I wrote last February:

...universities are not turning out people ready-made for the role. The ideal data scientist, according to Patil, possesses these characteristics: deep technical expertise in some (usually hard) scientific discipline, curiosity, cleverness, and the ability to tell a story (with data).

Demand for data scientists is running high, if this Simply Hired listing of (currently) 202 open positions is any indication. A McKinsey study (PDF) last fall estimated that the next decade will see a shortfall of 140,000 to 190,000 workers with the deep analytical skills required of a data scientist.

But perhaps all those companies posting job openings with "data scientist" in the title don't actually need that many people so highly qualified. Here's another perspective on staffing an analytics function from Rahul Deshmukh, who is Director, Web Intelligence Product Marketing at Splunk, a supplier of big-data solutions. you need a "data scientist" for every "big data" problem? Not really. The algorithm, data mining, or advanced statistical modeling pieces represent 10 to 15 percent of all analytics needs within the organization. scientists work on futuristic products; data or web [analysts] work on current [products]...

Deshmukh is in agreement with Patil about the skills needed for big-data analytics, but drills deeper to distinguish a continuum of expertise. He believes that an 80/20 rule works well in this situation: staff the analytics function with 80 percent data / business / Web analysts and 20 percent data scientists.

A well-built data science team can generate insights about the business, along with products that can move the business in new directions. Such a function is coming to be a necessary part of any company that is reaching for agility in this era of big data.