On-Demand Webinar · 57 min

Why Do DataOps?

Chris Bergh on what DataOps actually is and how a data team uses it to cut development time, reduce errors to near zero, and improve collaboration within and across teams. Recorded March 2020; updated August 2026.

Presented by Chris Bergh

What you'll learn 6 points
  • The failure numbers are the argument: 87 percent of data science projects never reach production, 60 percent of all data analytic projects fail, and 79 percent of data projects have too many errors, while the share of organizations calling themselves data-driven fell from 37 percent to 31 percent.
  • The diagnosis is that this is a people and process failure, not a technology one. Data analytics in 2020 is compared to the US auto industry in the 1970s: high production errors and deployment latency measured in weeks and months.
  • Data teams sit between three competing forces — unaware data providers sending late and error-prone data, demanding consumers expecting Amazon-speed delivery, and critical production operators needing flawless output. The result is a work environment where teams cannot innovate and cultures run on heroism or fear.
  • Conway's Law applies to data: the structure of how teams are organized for engineering, science, analytics, and production shows up directly in the shape of the data pipelines, and a common platform is what lets a team escape that constraint.
  • The DevOps precedent is quantified. High-performing IT organizations deploy 200 times more frequently, recover 24 times faster, have three times lower change failure rates, and spend 22 percent less time on unplanned work and rework.
  • The prescription is seven steps plus three: orchestrate two journeys, add tests and monitoring, use version control, branch and merge, use multiple environments, reuse and containerize, parameterize processing, then add architecture, metrics, and inter- and intra-team collaboration.

Prefer to read it? The written version is in Why Do DataOps.

Slides

54 slides

Questions from this session

Why do DataOps at all?

Because the base rates are bad and they are not improving on their own. 87 percent of data science projects never get to production, 60 percent of data analytic projects fail, and 79 percent have too many errors — while investment rises and the share of self-described data-driven organizations falls from 37 percent to 31 percent. Spending more on the same way of working has not moved those numbers.

Is the problem technology or people and process?

People and process. Teams already have capable tools — ETL, warehouses, notebooks, catalogs, BI. What they lack is a way to deploy safely, catch errors before customers do, and coordinate across teams and locations. DataOps is framed here as technical practices, cultural norms, and architecture rather than a product category.

What does Conway's Law have to do with data pipelines?

The structure of the teams doing data engineering, data science, analytics, and production is reflected in the pipelines they build. Three teams with separate handoffs produce three disconnected pipeline segments with brittle joins. A shared platform with common pipeline and environment constructs is what allows collaboration to cross those team boundaries instead of being shaped by them.

What evidence is there that this approach works elsewhere?

DevOps in software and Lean in manufacturing. The State of DevOps Report finds high performers deploy 200 times more frequently, recover 24 times faster, have three times lower change failure rates, and spend 22 percent less time on unplanned work. Lean manufacturing produced higher quality, less rework, better employee satisfaction, and higher profit.

What are the seven steps to DataOps?

Orchestrate two journeys, add tests and monitoring, use a version control system, branch and merge, use multiple environments, reuse and containerize, and parameterize your processing — plus three more covering architecture, metrics, and inter- and intra-team collaboration.

Why focus on operations rather than the next feature?

The session quotes Musk on building the machine that makes the machine, and Nadella's rule that an engineer choosing between a feature and developer productivity should always choose productivity. Google has over 2,000 engineers contributing to engineering productivity. The claim is that capacity to deliver compounds, while any single feature does not.

Where to go next