On-Demand Webinar · 50 min

Jumpstart Your DataOps Program with DataKitchen’s Lean DataOps

Beth Pfefferle and Chris Bergh walk through Lean DataOps, DataKitchen’s four-phase rollout: Production DataOps for an observable, error-free analytic factory, then Development, Measurement, and Enterprise DataOps. Recorded September 2021; updated August 2026.

Presented by Chris Bergh

What you'll learn 8 points
  • The numbers behind the case for Lean DataOps: Gartner reports 60 percent of data analytic projects fail altogether and 87 percent of data science projects never reach production, while NewVantage Partners recorded organizations calling themselves data-driven falling from 37 percent to 31 percent.
  • The 2021 DataKitchen and data.world data engineer survey found 52 percent hope and pray that things do not break, 78 percent are stressed enough to want a therapist, 70 percent expect to change jobs within a year, and 79 percent have considered leaving the career entirely.
  • Lean DataOps stages adoption in four phases so no reinvention is required: Production DataOps for a team of one to three with no process change, Development DataOps for a team of three to ten, Measurement DataOps across multiple teams, and Enterprise DataOps across multiple groups, which is the only phase demanding significant process change.
  • Data observability is defined here as the technical practices, cultural norms and architecture that enable low error rates, and as a subcomponent of DataOps. It covers four kinds of failure: data quality, lateness against an SLA, a system processing issue, and a code change that broke something.
  • Production DataOps needs no process change and no tool change. Automated tests are added to existing pipelines and written in whatever the team already uses, SQL, Python or a tool UI, with errors classified by severity so alerts match the failure. The deck shows a move from zero automated tests and frequent errors to 1,000 tests per build and none.
  • One real deployment ran through four manual stages across data science, visualization and governance teams and took four months end to end. Seventy percent of companies take weeks or months to push a change to production.
  • Rajesh Gill, former Celgene senior manager of business analytics, forecasting and DataOps strategy, reported errors falling to about one per quarter when DataOps was first implemented, and several years without a major glitch as tests kept being added.
  • Enterprise transformation follows six steps: educate on the ideas, find a first project, establish a community of interest, demonstrate real value in a month or two, iterate onto more use cases, and expand into a staffed center of excellence or dojo.

Prefer to read it? The written version is in Start DataOps Today with 'Lean DataOps'.

Slides

56 slides

Questions from this session

What is Lean DataOps?

Lean DataOps is an incremental way to adopt DataOps that asks what the least amount of effort is that yields the greatest benefit. Rather than rebuilding a data platform, it starts by adding DataOps to the pipelines already running, then breaks the rest of adoption into four phases that widen from a single small team to the whole organization.

What are the four phases of a Lean DataOps program?

Production DataOps lowers error rates through testing, statistical process control and observability, with a team of one to three and no process change. Development DataOps automates cycle time, productivity and collaboration for a team of three to ten. Measurement DataOps adds process measurement across multiple teams. Enterprise DataOps drives lasting organizational change across multiple groups and is the phase that requires significant process change.

What is data observability?

Data observability is the set of technical practices, cultural norms and architecture that enable low error rates, and it is a subcomponent of DataOps. It is broader than data quality: it also covers lateness against an SLA, system processing issues, and a code change that broke something. Every step in production gets tested, from whether inputs are clean to whether business logic is still correct and outputs are consistent.

Do you have to change your tools to start doing DataOps?

No. The first phase adds automated tests to existing pipelines with no process change and no tool change, so tests can be written in SQL, in Python, or in a favorite tool's UI. Errors are then classified by severity level so an alert matches the seriousness of the failure. Tests go up, errors and unplanned work come down.

How long does it take most companies to deploy a change to production?

Seventy percent take weeks or months. One real deployment described in the session moved through four manual stages across data science, visualization and governance teams and took four months from development to production. The manual handoffs are what add the complexity, the slowness and the errors, and environment creation is usually the bottleneck inside them.

What are the six steps of an enterprise DataOps transformation?

Educate on the ideas of DataOps through presentations, video and books. Find a first project by talking with individual teams about where the pain is. Establish a community of interest with shared wikis and channels, aligned with data and agile leaders. Demonstrate real value in a project of a month or two. Iterate onto more use cases. Expand by staffing a full-time center of excellence or dojo with common tools and metrics.

Where to go next