In today’s on-demand economy, the ability to derive business value from data is the secret sauce that will separate the winners from the losers. Data-driven decision making is now more critical than ever. Analytics could mean the difference between finding the right mix of strategic moves or falling behind. In fact, Forrester Research predicted that insight-driven companies would grow seven to 10 times faster than the global GDP through 2021.
Most enterprise companies recognize the need to be data-driven, yet 60% of data projects fail to move past preliminary stages, and 87% of science projects never make it to production. More surprisingly the number of data-driven companies has actually fallen from 37% to 31% since 2017, despite increased investment, according to Gartner.
What gives?
Becoming data-driven is hard and data teams are suffering. They are caught between the competing demands of data consumers, data providers, and supporting teams. Typically, data consumers live in an Amazon world and expect trusted, original insight on-demand. Yet data providers often send inaccurate, late, or error-prone data sets. The flawless collaboration and production required from teams in other parts of the organization often just isn’t there.
Taken together, the need to manage complex toolchains and data, as well as collaborate with other organizations, roles, locations, and data centers, saps the data team’s time. In fact, most data teams spend more time fixing errors and addressing operational issues than innovating and providing business value. According to Gartner, only 22% of a data team’s time is spent on new initiatives and innovation. As a result, many data teams are not meeting expectations, or worse, are beaten down and disempowered.
Figure 1: According to Gartner, only 22% of a data team’s time is spent on new initiatives and innovation.
Focus on Operations, Not the Next Feature
Data teams can learn important lessons from other industries. According to management guru W. Edwards Deming, 94% of problems are “common cause variation,” and to decrease this variation you must focus on the system or process, not look for a person to blame. A relentless process focus has led to dramatic improvements in the auto industry, where lean manufacturing principles have led to dramatically higher levels of productivity and quality. Or more recently in software development, the principles of DevOps have enabled companies to perform millions of software releases each year. __
“We realized that the true problem, the true difficulty, and where the greatest potential is – is building the machine that makes the machine . In other words, it’s building the factory. I’m really thinking of the factory like a product.” – Elon Musk
This mind shift was more recently highlighted by Elon Musk who said “we realized the true problem, the true difficulty, and where the greatest potential is – is building the machine that builds the machine. In other words, it’s building the factory. I’m really thinking of the factory like a product.” Successful data organizations are also wise to think of their data pipelines like a factory where quality and efficiency must be managed. But how can a data team shift its focus from the next big tool, technology or data feature to the people and process?
A Solution to the Suffering
In data analytics, DataOps provides the path forward. DataOps aligns the people, processes, and technologies of the data analytics organization. Supported by automation, it puts the focus on the underlying systems and managing the ‘data factory.’ Companies that follow DataOps principles spend less time worrying about the next model, algorithm, tool, visualization or even the data itself, but instead focus on how to develop, deploy, test, monitor, collaborate and measure their analytic operations.
By doing so, these companies realize multiple, simultaneous benefits.
- They experience orders of magnitude improvements in cycle time. They are able to deploy new features quickly and confidently, often improving from months/weeks to days/hours.
- They lower or even eliminate costly and embarrassing errors, enabling them to build a strong culture of trust with their data customers.
- They dramatically increase productivity. Better intra- and inter-team collaboration means less time spent on meetings and bureaucracy.
All of this creates the time and space for the data team to focus on what they signed up for in the first place – creating innovative analytics and delivering business value.
Figure 2: DataOps reduces time spent on errors and operational tasks and increases innovation.
Data organizations that neglect to modernize their processes, risk being left behind in an increasingly on-demand economy. DataOps enables teams to reclaim control of their data pipelines, reduce time- and soul-sucking errors and minimize the time from new ideas to the deployment of working analytics.
In any economy, but especially in challenging times, the most innovative companies will be those that can quickly adapt to rapidly evolving market conditions. The data teams that adopt DataOps and produce robust and accurate analytics more rapidly than their peers will power strategic decision-making that sustains a competitive advantage.
FAQ
What are the key points in this blog?
For a data team, how the work gets done matters more than which tool or model gets picked. A 2020 Gartner survey put innovation at 22% of a data team’s time, and Deming’s finding that 94% of problems are common cause variation says the fix is the system, not the person. DataOps aligns people, process and technology to manage the data factory.
Why do data teams spend so little time on innovation?
Because operational work crowds it out. A 2020 Gartner survey put the share of a data team’s time spent on new initiatives and innovation at 22%, with the rest absorbed by data management, production support and supporting business functions. Complex toolchains, late or error-prone incoming data and coordination across teams, locations and data centers consume the remainder.
What is common cause variation, and why does it matter for data teams?
Common cause variation is the variation a system produces by itself rather than through one-off events. W. Edwards Deming held that 94% of problems come from it, so cutting errors means changing the process instead of finding someone to blame. For a data team that shifts attention from the last bad report to the pipeline that produced it.
What does treating the data pipeline like a factory mean?
It means managing the thing that produces the analytics, not only the analytics. Musk’s version of the idea is “thinking of the factory like a product”: the machine that builds the machine is where the leverage sits. For a data team that means investing in how analytics get developed, deployed, tested, monitored and measured, rather than in the next feature.
Is the number of data-driven companies growing?
Not according to Gartner figures cited in 2020, which showed the share of data-driven companies falling from 37% to 31% since 2017 despite increased investment. Recognizing the need to be data-driven and building the process to become one are different problems, which is why spending rose while the count went down.
What changes when a data team adopts DataOps?
Cycle time, error rates and collaboration all move. Teams deploy new features in days or hours where they previously needed weeks or months. Costly and embarrassing errors drop, which is what builds trust with data customers. Better collaboration inside and between teams cuts the time lost to meetings and bureaucracy, and the time recovered goes back into building analytics.
