Introduction
Whatever you were planning to accomplish this week — forget about it!
Chances are good that you’ll be interrupted by data errors. That is the clear message communicated by the respondents to a joint DataOps survey conducted by DataKitchen and Eckerson Research.
The survey contains feedback from 300 data-analytics professionals who work for medium to large-sized companies across multiple industries in the US and Europe. The full report will be available in June 2019, but here are the key results. The survey sheds light upon three issues that embody the analytics industry’s challenges. From a recent NewVantage Partners Report, we know that despite the hype and investment, the number of companies that identify as data-driven is declining. Gartner estimated that 60% of big data projects fail. We see results in our survey that may help explain this.
NOTE
“The full report will be available in June 2019” is a 2019 sentence, and no separate full report matching that promise has been identified. It is not Best Practices in DataOps — that is a different Eckerson Group report, by Wayne Eckerson, dated June 2019, built on practitioner interviews rather than survey data, and it contains no respondent counts or survey tables at all. The results on this page, together with 2019 DataOps Survey: Errors and Slow Innovation Abound, are the published record of this survey. DataKitchen’s later survey research is the 2021 Data Engineering Survey.
The Impact of Data Errors
One egregious issue is that analytics too often contain errors which erode the credibility of the data team. 30% of respondents to the DataOps survey reported more than 11 errors per month. This is a staggering figure. That means that the data team is probably moving from fighting one fire to the next with little time for any value-add activity. The managers of these enterprises learn not to trust the data. Is it any wonder that companies are becoming less data-driven?
The DataOps enterprises that DataKitchen works with have less than 1 data error per year. Only 3% of the companies surveyed approached that level of quality. Another 18% reported 1-2 errors per month. Would W. Edwards Deming have considered that an acceptable failure rate? Would Toyota? In a manufacturing setting, each one of these errors could be the equivalent of a product recall. For the sake of discussion, let’s presume that 1-2 errors per month in an enterprise analytics context are tolerable (it’s really not). That still leaves nearly 80% of companies surveyed reporting 5, 10 or even more errors per month. That has a big impact on how an organization views its data and may even explain why the average tenure of a Chief Data Officer is only around 2 years.
| Errors per month | Share of respondents |
|---|---|
| No errors | 3% |
| 1 to 2 errors | 18% |
| 3 to 5 errors | 29% |
| 6 to 10 errors | 20% |
| 11 or more errors | 30% |
Table 1: “On average, how many errors (e.g., incorrect data, broken reports, late delivery, customer complaints) do you have each month?” Asked in the spring of 2019 of 300 data-analytics professionals at medium and large companies across multiple industries in the US and Europe. 79% of respondents reported three or more errors a month; 21% reported two or fewer.
DataKitchen interpretation: most companies have way too many errors per month.
Data errors negatively impact the productivity of the analytics teams in several ways. They flood Kanban boards with new tasks. They cause unproductive context switches. Wary of making further errors, the data team may become overly cautious, working more slowly. In short, data errors are a major bottleneck that affect the entire workflow of new analytics development. We call this analytics cycle time, and it is one of the critical bottlenecks that slow the ability of analytics to create value for an enterprise.
A short cycle time enables an analytics team to respond quickly to requests for new analytics. When analytics are produced quickly, the data team can keep pace with the endless stream of requests from the business unit. A short cycle time fosters close collaboration with business users and in our experience this unlocks an organization’s creativity. However, the cycle time in most data organizations is plagued with inefficient manual processes, bureaucracy, lack of task coordination and dependencies on bottlenecks. For example, survey respondents provided interesting feedback about the time it takes to create an analytics development environment.
Delays in Environment Creation
Isolated development environments are an important way that analytics development can proceed without impacting data operations. If it takes weeks or months for IT to provision hardware and software or to make data available, the queue time severely impacts the productivity of analytics professionals.
78% of respondents indicated that it takes days, weeks or months to create a development environment. 38% of users surveyed report that it took weeks or months. That wait time prevents the data analytics team from even beginning to work on the critical analytics that the organization has requested. This means that their time-to-value is much slower than it should be.
| Time to create a new development environment | Share of respondents |
|---|---|
| Minutes | 12% |
| Hours | 10% |
| Days | 40% |
| Weeks | 28% |
| Months | 10% |
Table 2: “On average, how long does it take your team to create a new development environment with the appropriate test data, servers, and tools?” Same 2019 respondents. Days, weeks and months together account for 78% of responses.
DataKitchen interpretation: most companies are very slow creating new development environments.
Lengthy Deployment Times
We asked our survey respondents directly about the end-to-end cycle time of creating new analytics. In light of the above, it is not surprising that we see that far too many organizations undergo lengthy periods of time to create and deploy analytics. 76% of organizations surveyed take days, weeks or months to move from analytics development to production. If one data engineer needs to support a dozen data analysts and each data analyst needs to support hundreds of sales people, the cycle time for new analytics must be reduced to hours or better yet, minutes. Enterprises can reach these performance targets with a DataOps approach to analytics development and deployment.
| Time from development to production | Responses | Share of responses |
|---|---|---|
| Minutes | 16 | 9.3% |
| Hours | 26 | 15.1% |
| Days | 61 | 35.5% |
| Weeks | 47 | 27.3% |
| Months | 22 | 12.8% |
| Total | 172 | 100% |
Table 3: “On average, how long does it take to move a new or modified data analytic pipeline from development to production?” Same 2019 respondents. The source chart plots respondent counts against an axis beginning at 10, so the shares here are of the 172 responses to this question. Days, weeks and months together come to 75.6%, which is the 76% reported in the paper.
DataKitchen interpretation: most companies are too slow to deploy changes into production.
Why These Three Bottlenecks Matter
The three survey responses above help explain why enterprises are not able to respond to user requests for new and updated analytics in a reasonable time frame. For organizations that suffer from these constraints, the case could be made that nothing else matters but addressing these bottlenecks. The ability to rapidly produce and deploy analytics is at the heart of a data team’s ability to add value. If viewed from the perspective of the Theory of Constraints, these bottlenecks limit the overall throughput of value creation. Any improvement in analytics development cycle time improves the overall throughput of the system. An improvement other than in the bottleneck is an illusory accomplishment when an analytics team suffers from long development cycle times.
DataOps offers a way to reduce errors, shorten the time it takes to set up a development environment, and minimize analytics development cycle time. Nothing could state more clearly why analytics organizations need a DataOps initiative now.
FAQ
What is the main point of this paper?
Data teams are interrupted by errors and slowed by queues, and the survey puts numbers on both. Among 300 data-analytics professionals, 30% reported more than 11 data errors per month, 78% waited days or longer for a development environment, and 76% took days or longer to deploy a pipeline change. The three results describe one bottleneck: analytics cycle time.
Who ran the 2019 DataOps Survey, and who responded?
DataKitchen and Eckerson conducted the survey jointly, fielding it in the spring of 2019 and publishing these key findings that June. The 300 respondents were data-analytics professionals working for medium to large-sized companies across multiple industries in the United States and Europe. The findings covered three subjects: the impact of data errors, delays in environment creation, and the causes of lengthy deployment times.
How many data errors do data analytics teams have each month?
30% of respondents reported 11 or more errors per month. Another 20% reported 6 to 10 errors, 29% reported 3 to 5, and 18% reported 1 to 2. Only 3% reported no errors at all. Errors here means incorrect data, broken reports, late delivery or customer complaints, so 79% of companies surveyed were fielding at least three a month.
What share of companies had an acceptable data error rate?
Very few. The DataOps enterprises DataKitchen works with have less than one data error per year, and only 3% of companies surveyed approached that level of quality. Even granting that 1 to 2 errors a month is tolerable, which it is not, that still leaves nearly 80% of respondents reporting three, five, ten or more errors every month.
How long does it take to create a new analytics development environment?
78% of respondents said days, weeks or months. Broken down: 40% said days, 28% said weeks and 10% said months, so 38% waited weeks or longer. Only 12% could create a new development environment with the appropriate test data, servers and tools in minutes, and 10% in hours.
Why does slow environment creation matter?
Isolated development environments are how analytics development proceeds without disturbing data operations. When provisioning hardware, software and test data takes weeks, the queue time lands entirely on the analytics team, which cannot start work the business has already asked for. Time-to-value stretches for reasons that have nothing to do with the difficulty of the analytics.
How long does it take to deploy a new or modified data pipeline into production?
76% of organizations surveyed take days, weeks or months. Reading the responses to that question, 35% said days, 27% said weeks and 13% said months, against 9% who said minutes and 15% who said hours. A quarter of organizations could move a change into production within a working day.
What is analytics cycle time?
Analytics cycle time is the end-to-end interval between a request for new or changed analytics and its arrival in production. The survey shows the three things that lengthen it: unplanned work created by data errors, queue time waiting for a development environment, and manual deployment. A short cycle time lets a data team keep pace with the requests the business generates.
What does the Theory of Constraints have to do with data analytics?
It says that a system’s throughput is set by its bottleneck, so improving anything else changes nothing. For an analytics team with long development cycle times, the survey argues that addressing those bottlenecks is the only improvement that raises the throughput of value creation. Any other gain is an illusory accomplishment.
What is DataOps?
DataOps is a collection of technical practices, workflows, cultural norms and architectural patterns that enable rapid innovation and experimentation, very low error rates, collaboration across complex arrays of people, technology and environments, and end-to-end observability with clear measurement and transparency of results. It applies Agile development, DevOps and statistical process control to data analytics.
How does DataOps reduce data errors?
By testing data and analytics continuously instead of inspecting them after delivery. Tests run against data as it arrives, against the logic that transforms it, and against the output before customers see it, so a bad load is caught at the point it breaks rather than in a dashboard. The result is unplanned work that never reaches the team.
Do the 2019 findings still hold?
The specific percentages are from 2019 and should be cited with that year attached. The pattern has proved durable: DataKitchen’s 2021 survey of 600 data engineers found 97% reporting burnout, with too much time spent finding and fixing errors named as a leading cause. Two years and a different population produced the same diagnosis.
What should a data leader do first with these numbers?
Measure the three the survey measures. Count errors per month, time how long a development environment takes to stand up, and time how long a change takes to reach production. Those three numbers locate the bottleneck, and they are the baseline against which any DataOps initiative gets judged. Nothing else improves throughput until they move.
Get the PDF
The full survey brief is on this page. Fill in the form for a PDF copy to keep or share.