Organizations can manage the shortage of data scientists by automating routine data work, enabling self-service analytics, building cross-functional teams, defining data science roles more clearly, and training current employees. Together these approaches let a small number of specialists focus on high-value work.
Companies are building data-centric cultures in planning, management and decision making, which raises demand for data scientists. Not every business can hire them, so many are adapting how they use the few they have.
Key takeaways
- Automation takes data management and operational tasks off data scientists.
- Self-service analytics lets non-specialists answer their own questions.
- Cross-functional teams help break down data silos and tie projects to business value.
- Clearer role definitions and internal training widen the talent pool.
What are the top ways to manage the data scientist shortage?
| Approach | Main idea |
|---|---|
| Automation | Automate data engineering, model development and operations |
| Self-service analytics | Let citizen data scientists and analysts work with data directly |
| Cross-functional teams | Combine business units, or outside consultants, around clear goals |
| Better role definitions | Split broad data science jobs into focused roles |
| Internal training | Grow data skills among existing employees |
How does automation help?
A large share of a data scientist’s work is data management and operations, such as locating data sources, merging data sets and verifying data quality. These are necessary but are not the high-value tasks data scientists are hired for.
Ryohei Fujimaki, CEO and founder of dotData, an automated machine learning software company, has said that automation makes model development and operationalization easier. Modern data science automation lets organizations run and maintain operational and production processes with less effort, which helps them speed up investment in AI and machine learning.
Matthew Baird, founder and CTO of AtScale, a data engineering software company, points to autonomous data engineering as a prominent development. It automates data management tasks and arrives in the form of just-in-time data engineering. In effect, the automation acts as a model data engineering team that understands underlying data structures across databases, network characteristics, data location, policies and security processes.
What is self-service analytics?
Automating modeling and data management does more than free senior data scientists. It also standardizes data resources for citizen data scientists, meaning non-specialists who work with data.
Another popular option is self-service analytics. According to Baird, autonomous data engineering combined with more active citizen analysts frees data science and engineering resources to work on higher-value tasks, such as building the next machine learning models.
Why build cross-functional teams?
Automation and self-service tools have limits. Chris Nicholson, founder and CEO of Pathmind, a deep learning software company, notes that every tool that eases data science work also limits user flexibility. It can make customization of complex tasks harder or even impossible.
As a result, he says, many enterprises are exploring cross-functional data science teams. These companies work with several business units or bring in external consultants. His point is that the value of data science in a company is limited less by a lack of data scientists than by the data the company collects and how people use it.
Tips for cross-functional teams
Cross-functional teams help organizations deal with fragmented data silos, which often arise from technical and internal political barriers. Those barriers ease when the right stakeholders work together in the same team.
They also address a problem that looks like a talent shortage but is something else: projects with unclear business value. Sten Vesterali, principal consultant at More Than Code, an IT consulting firm in Denmark, says too many projects are wild goose chases in which data is handed to data scientists with the instruction to see what they can find. He adds that nearly 80% of data science projects fail for lack of execution because they never leave the lab for production. His conclusion is that companies should assign data scientists to the most valuable business goals.
Why define data science roles better?
Amy Hodler, director of graph analytics and AI programs at graph database company Neo4j, argues that many hiring problems come from treating data scientist as a single title with responsibilities far beyond what one person can handle. This makes it hard for new hires to understand the company’s business objectives.
She expects organizations to revise their titles, create more specialized sub-roles and pay closer attention to job requirements.
How does internal training help?
Hodler also expects companies to address the shortage through extensive internal training of existing employees who want to move into data science. The strategy can be hit or miss, so it needs a structured plan with specific skills to build.
It also requires a clear way to assess and describe the skill sets needed. Pairing junior and senior data scientists is expected to have a strong effect on how these employees develop.
Frequently asked questions
Why is there a shortage of data scientists?
Demand has grown as organizations adopt data-driven decision making, while the supply of qualified candidates is limited. Overly broad job definitions make the gap look larger.
Can automation replace data scientists?
No. It handles routine data engineering and operational work so that data scientists can spend time on higher-value tasks. Automation tools also limit flexibility for complex, custom work.
What are citizen data scientists?
They are non-specialists who analyze data using self-service tools, which reduces the load on professional data scientists.
Why do data science projects fail?
The original cites an estimate from Sten Vesterali that nearly 80% fail for lack of execution, often because they never reach production or lack a clear business goal. Treat this figure as one consultant’s claim.
Conclusion
No single fix solves the data scientist shortage. Automation, self-service tools, cross-functional teams, clearer roles and internal training work best in combination. The common thread is to point scarce expertise at clearly defined, high-value business problems.
