This story was written by Keith Dawson for UBM DeusM’s community Web site Business Agility, sponsored by IBM. It is archived here for informational purposes only because the Business Agility site is no more. This material is Copyright 2012 by UBM DeusM.

US Government Research Push on Big Data

$200 million in research funding will accelerate big data knowledge, tools, and education.

The federal government announced a $200 million program of foundational research in big-data management and analytics. It will result in breakthrough tools that agile enterprises can put to work.

On Thursday, the Office of Science and Technology Policy brought together six government departments and agencies to kick off a program of basic research into big data. The video of the announcement event should be available here within a few days.

In IT we have known for some time of the growing importance of big data in giving agile corporations a competitive edge; the government initiative means that soon everyone will know about it.

The centerpiece of the announcement was a joint program by the National Science Foundation and the National Institutes of Health that will look for new technologies and methods for data management, data analysis, and machine learning. Other organizations offering research money to universities and corporations are the Departments of Defense and Energy, DARPA, and the US Geological Survey.

Corporations will benefit directly in the medium term from this research push because it is certain to spur universities to turn out more graduates who are at least conversant with big data -- in fact one of the grants NSF announced yesterday will go directly to creating more such degree programs. Last year McKinsey estimated that the US will suffer in the next five years a shortfall of 190,000 data scientists, 300,000 data technicians, and 1.5 million data-literate managers. At this point graduate programs featuring big-data analytics are thin on the ground. North Carolina State University has been offering one for a few years, and Northwestern's MS program in analytics will start in the fall.

While the exact parameters of the set of skills and knowledge required in a data scientist are not completely nailed down, there is some agreement that they include hacking skills, math and statistics knowledge, and subject-matter expertise. University programs need to be crafted to attract the right candidates and to provide them with exposure at least to the first two of those. (Subject-matter expertise is much more likely to come with experience in the industry in question.)

In the longer term, $200 million will make a difference in this industry. From a quick look at the 87 companies tagged with "big-data" in CrunchBase, the $200 million sum seems to be on the rough order of the amount venture capitalists have put into companies focused on big data. Given that the federal cash will fund basic research, it's not unreasonable to expect large (and unpredictable) breakthroughs from it over the coming decade. Tom Kalil, deputy director of the White House Office of Science and Technology Policy, compared the big data research initiatives with such earlier efforts as government support of high-speed networking and supercomputing centers. "This is that level of importance," Kalil is quoted in the New York Times.

Executives committed to making their companies more agile over the long term should keep an eye on this government initiative and be prepared to reap the benefits it will provide to the field of big data -- beginning with a deeper pool of data science practitioners.