Skip to Content

Don’t Forget About Small Data – and Matching It to Goals

In the era of big data, more and more companies are gathering all the data they can. Business leadership has invested in data capture and hired people to scrub and manipulate data sets. But it’s also important for business strategy to remember that smaller sets of data and the objectives they are designed to achieve are sometimes more crucial – and more useful – in solving problems.

Risk Assessments Are Important…

A case in point comes from the healthcare industry. The Stanford Graduate School of Business publication Insights recently highlighted the use of smaller data sets in solving a problem involving risk assessments.

Companies for some time have been trying to identify people at risk for certain diseases, such as heart disease or diabetes so that they can be provided with options for preventive care through corporate wellness programs. Statistics show that these types of chronic diseases are responsible for a huge burden of healthcare expenditures, but can be prevented if early intervention strategies are followed.

How do companies identify populations at risk, though? Usually, employees are asked to fill out risk assessments, and lab work is done to identify biomarkers that may indicate risk factors.

Alas, these biomarkers are often inaccurate. Considerable sums are expended in pursuit of an excellent idea – improving healthcare outcomes and reining in healthcare costs across the country – but the idea is thwarted because risk assessments both identify people who turn out not to develop the disease and don’t identify people who do.

But researchers presented with this problem actually found a better way to find accurate biomarkers not by widening the pool of data, but by narrowing it. Essentially, they reduced the amount of information by focusing on objectives and a manageable amount of data. It is possible to have a large set of biomarkers for every chronic disease, for example, but it might yield too much information to be manageable. Instead, they decided on a maximum of 30 and worked with machine learning principles to optimize the 30 chosen.

So it wasn’t that the tools of big data were slighted, but that big data wasn’t allowed to run amuck.

But goals and strategy have to have first importance.

…but Objectives and Data Management Matter Too

The good news? When their markers were tested on 75,000 people, they proved a considerable improvement of accuracy over past markers, predicting the risk of developing nine chronic diseases with great accuracy.

The researchers attribute the success of the program partly to the markers, but also partly to working to objectives that guided them to have as accurate risk assessments as possible, not to have the largest group of data possible. Objectives should come first.

Clearly, this can optimize employee wellness programs in the fight against chronic disease, improving both health and containing costs.

While the example is in one sector, the implications pertain to every sector. Big data is exciting partly because its potential seems to be limitless. But the pitfalls can be, well, big as well. The solution is to have objectives come first, with manageable data tasked with helping, not leading.