Skip to Content

Data Scientists Need to be Able to Answer “Why?”

Data scientist positions are one of the fastest-growing jobs in the U.S., and companies are making large investments to ensure a competitive edge and develop new products and services. However, as a recent Harvard Business Review points out regarding technology news, both Big Data and artificial intelligence projects that use data scientists have a dismal track record. The failure rate is estimated at roughly 85 percent.

Data science is a burgeoning part of U.S. companies, but many projects fail.

Why? There are likely several reasons. A successful venture needs excellent Big Data or AI work, of course, but it also needs a sophisticated understanding of the business problems data scientists have been hired to fix or impact. They need to know the “why” of what they are doing. The political climate and personal relationships at companies also matter deeply; business leadership needs buy-in and a comfort level, as do other stakeholders.

Data scientists may tend to adhere to their training, which is to find interesting patterns in data. There is a risk that they view that as the goal, rather than the business goal itself. Second, understanding the overall business climate in which data scientists operate can be complicated and difficult to navigate.

What are the best methods of ensuring success in data science projects, then? The HBR suggests the following.

1. Make sure business objectives are developed and communicated.

The first step is ensuring that the business objectives the company wants to reach are developed and communicated clearly. Everything else proceeds from goal alignment.

2. Measure your progress toward meeting the objectives.

Develop the metrics that will measure progress toward the company’s goals most effectively. While it may take some time for a data program to get up to speed, companies have a right to expect measurable progress toward the goal in a few years. Should the data program increase revenues? Result in cost reductions? Improve customer retention or satisfaction? Both metrics and goalposts need to be in alignment with the objectives desired.

3. Place data scientists directly in daily work with the rest of the company.

The HBR cautions against siloing data scientists. They need to be embedded in the company and aware of its culture and goals.

4. Conduct pre- and post-analysis.

Just like other research projects, data projects need a fully developed problem statement. Pre-analysis development needs to be done to get there. The issues you want to explicate and the goals you are intended to hit need to be clear from the beginning, so the project can sync.

Post-analysis study also is crucial. The results of the project need to be tested in the real world to see if they work and bring about results.

5. Encourage data scientists to teach the organization.

Data science and its potential applications are new and exciting. It’s a good idea to encourage data scientists to teach both intradepartmentally and throughout the company. They can explain their project’s potential and results and make data science familiar, which will help with corporate acceptance and trust.