In today’s on-demand economy, the ability to derive business value from data is the secret sauce that will separate the winners from the losers. Data-driven decision making is now more critical than ever. Analytics could mean the difference between finding the right mix of strategic moves or falling behind. In fact, Forrester Research predicted that insight-driven companies would grow seven to 10 times faster than the global GDP through 2021.
Most enterprise companies recognize the need to be data-driven, yet 60% of data projects fail to move past preliminary stages, and 87% of science projects never make it to production. More surprisingly the number of data-driven companies has actually fallen from 37% to 31% since 2017, despite increased investment, according to Gartner.
What gives?
Becoming data-driven is hard and data teams are suffering. They are caught between the competing demands of data consumers, data providers, and supporting teams. Typically, data consumers live in an Amazon world and expect trusted, original insight on-demand. Yet data providers often send inaccurate, late, or error-prone data sets. The flawless collaboration and production required from teams in other parts of the organization often just isn’t there.
Taken together, the need to manage complex toolchains and data, as well as collaborate with other organizations, roles, locations, and data centers, saps the data team’s time. In fact, most data teams spend more time fixing errors and addressing operational issues than innovating and providing business value. According to Gartner, only 22% of a data team’s time is spent on new initiatives and innovation. As a result, many data teams are not meeting expectations, or worse, are beaten down and disempowered.
Figure 1: According to Gartner, only 22% of a data team’s time is spent on new initiatives and innovation.
Focus on Operations, Not the Next Feature
Data teams can learn important lessons from other industries. According to management guru Dr. Edward Deming, 94% of problems are “common cause variation,” and to decrease this variation you must focus on the system or process, not look for a person to blame. A relentless process focus has led to dramatic improvements in the auto industry, where lean manufacturing principles have led to dramatically higher levels of productivity and quality. Or more recently in software development, the principles of DevOps have enabled companies to perform millions of software releases each year.
“We realized that the true problem, the true difficulty, and where the greatest potential is – is building the machine that makes the machine. In other words, it’s building the factory. I’m really thinking of the factory like a product.” – Elon Musk
This mind shift was more recently highlighted by Elon Musk who said “we realized the true problem, the true difficulty, and where the greatest potential is – is building the machine that builds the machine. In other words, it’s building the factory. I’m really thinking of the factory like a product.” Successful data organizations are also wise to think of their data pipelines like a factory where quality and efficiency must be managed. But how can a data team shift its focus from the next big tool, technology or data feature to the people and process?
A Solution to the Suffering
In data analytics, DataOps provides the path forward. DataOps aligns the people, processes, and technologies of the data analytics organization. Supported by automation, it puts the focus on the underlying systems and managing the ‘data factory.’ Companies that follow DataOps principles spend less time worrying about the next model, algorithm, tool, visualization or even the data itself, but instead focus on how to develop, deploy, test, monitor, collaborate and measure their analytic operations.
By doing so, these companies realize multiple, simultaneous benefits.
- They experience orders of magnitude improvements in cycle time. They are able to deploy new features quickly and confidently, often improving from months/weeks to days/hours.
- They lower or even eliminate costly and embarrassing errors, enabling them to build a strong culture of trust with their data customers.
- They dramatically increase productivity. Better intra- and inter-team collaboration means less time spent on meetings and bureaucracy.
All of this creates the time and space for the data team to focus on what they signed up for in the first place – creating innovative analytics and delivering business value. 
Figure 2: DataOps reduces time spent on errors and operational tasks and increases innovation.
Data organizations that neglect to modernize their processes, risk being left behind in an increasingly on-demand economy. DataOps enables teams to reclaim control of their data pipelines, reduce time- and soul-sucking errors and minimize the time from new ideas to the deployment of working analytics.
In any economy, but especially in challenging times, the most innovative companies will be those that can quickly adapt to rapidly evolving market conditions. The data teams that adopt DataOps and produce robust and accurate analytics more rapidly than their peers will power strategic decision-making that sustains a competitive advantage.
