On-Demand Webinar · 59 min

DataOps For Beginners: The Art of the Possible With DataOps

Chris Bergh and Beth Pfefferle on making the internal case for DataOps: how it supercharges rather than replaces the staff and technology you already have, how to build the business case, and how to collect the small wins that show its impact. Recorded February 2022; updated August 2026.

Presented by Chris Bergh

What you'll learn 7 points
  • The 2021 DataKitchen and data.world survey of data engineers found 52 percent hope and pray that things do not break, 78 percent are stressed enough to want a therapist, 70 percent expect to change jobs within a year, and 79 percent have considered leaving the career entirely.
  • Gartner put innovation at 22 percent of a data team's time in 2020. The other 78 percent goes to errors and manual execution.
  • At Celgene, later Bristol Myers Squibb, seven data and DataOps engineers ran hundreds of integrated data sets with more than 10,000 automated tests and more than 100 schema and data changes a week, with very few errors or missed SLAs.
  • Production error rate is not only a data quality number. Late data that misses an SLA, a data processing failure, a code change that broke something, and a broken report or model all land on the customer the same way.
  • Production testing covers five test types: traditional data quality, statistical process control, location balance, historic balance, and business-based tests.
  • A DataOps engineer works on the pipelines rather than in them, taking nuggets of ETL, SQL, Python, and XML from other roles and putting them into pipelines, tests, deployments, and process measurement.
  • Bringing DataOps to an organization follows six steps: educate, find a first project, establish a community of interest, demonstrate value in a month or two, iterate onto more use cases, and expand with a staffed center of excellence.

Slides

65 slides

Questions from this session

What is DataOps?

DataOps is the set of technical practices, cultural norms, and architecture that enable rapid cycles of experimentation and delivery of new insight, low error rates, collaboration across complex sets of people, technology, and environments, and clear measurement of results. Gartner's Sumit Pal wrote in November 2018 that organizations adopting a DevOps and DataOps approach are more successful at implementing end-to-end, reliable, scalable, repeatable solutions.

How much of a data team's time goes to errors instead of new work?

Gartner put it at 78 percent in 2020, leaving 22 percent for innovation. The 2021 DataKitchen and data.world survey found 52 percent of data engineers named errors as a major source of burnout, and 78 percent said they were stressed enough to want a therapist.

What does a DataOps engineer do?

A DataOps engineer owns the assembly line rather than the product moving along it, working on the pipelines rather than in them. The job is to take nuggets of code from data engineers, scientists, analysts, and governance staff and put them into pipelines, create tests, run the factory, automate deploys, and measure success. The goal is making invisible process visible.

What should a data team automate first?

Eight things, in the order a team usually needs them: production orchestration, production data monitoring and testing, self-service environments, development regression and functional tests, test data, deployment, shared components, and process measurement. The common failure is not technical. Nobody owns automation, so everyone stays on the task in front of them.

What tests belong in a production data pipeline?

Five types: traditional data quality checks, statistical process control, location balance tests, historic balance tests, and business-based tests. They should run automatically in production, across the whole toolchain rather than one tool, send alerts, keep history, and be easy enough to create that the team keeps adding them.

How do you get a large organization to adopt DataOps?

Six steps. Educate on the ideas. Find a first project by talking with individual teams about where the pain is. Establish a community of interest and a shared resource center. Demonstrate real value in a project of a month or two. Iterate onto more use cases. Expand by staffing a full-time center of excellence with common tools and metrics.

Where to go next