On-Demand Webinar · 1 hr 4 min

Orchestrate Your Production Pipelines for Low Errors

Part one of Orchestrating the Three Pipelines of DataOps. Chris Bergh covers why and how to orchestrate a multi-environment, multi-tool production pipeline along the whole journey from data access to value delivery, and how to build testing and monitoring into it. Recorded April 2020; updated August 2026.

Presented by Chris Bergh

What you'll learn 7 points
  • DataOps has three pipeline orchestrations: the value pipeline that runs production, the innovation pipeline that moves changes to production, and the environment pipeline that both of the others stand on. The value pipeline is the one that runs day in and day out, and it is where the production errors live.
  • Gartner's March 2020 survey, Data Management Struggles to Balance Innovation and Control, put only 22 percent of a data team's week on new initiatives and 56 percent on operational execution.
  • Deming found that 94 percent of causes were common cause, which is the argument for fixing the process rather than finding a person to blame. Elon Musk's version on the same slide is that the real difficulty and the greatest potential is building the machine that makes the machine.
  • Every tool in a data toolchain has its own workflow or DAG, and each category has 50 or more tools to choose from, so a production value pipeline is a meta workflow: a DAG of DAGs sitting over Informatica, Airflow, Redshift, Python, Tableau, Alation, and the rest.
  • Testing data is not just pass or fail. Three severities carry different responses: error stops the line, warning gets investigated later, and info is a list of changes. Test history is kept so statistical process control can spot a trend break.
  • A location balance test compares the same quantity at each point on the pipeline: a million rows at source, a million in the database, 300,000 facts and 700,000 dimensions in the report. A historical balance test compares this run's aggregates against the last run's to catch a shift no single-step check would see.
  • The practice to start with is a production quality circle: keep track of every error and failure, meet periodically to review them, find the patterns and root causes, and write a new test or procedure for each one. The framing throughout is no shame, no blame, and it is not about data quality, it is about low rates of error.

Slides

55 slides

Questions from this session

What are the three pipelines of DataOps?

The value pipeline is production, the work that runs every day to turn source data into reports and models. The innovation pipeline is deployment, moving new code and configuration into production. The environment pipeline creates and manages the environments the other two run in, which is why it is described as the foundation for both.

What is a location balance test?

A location balance test checks that the same quantity holds at every point along the pipeline. If a million rows leave the source, a million should land in the database, and the report built from them should account for all of them, for example 300,000 facts and 700,000 dimensions. It catches silent losses that a check on any single step would miss.

What is a historical balance test?

A historical balance test compares an aggregate from the current run against the same aggregate from a prior run. If product group G1 totalled 225 last run and 358 this run while G2 moved the opposite way, the numbers have swapped groups or the mapping has changed. The check compares pre-production data against production data rather than validating a single value in isolation.

What kinds of tests belong in a production data pipeline?

Five types are named: traditional data quality checks, statistical process control, location balance, historic balance, and business-based tests. They should run automatically on top of the whole toolchain, send alerts to Jira, email, or Slack, keep their history, and be easy enough to create that people actually add them.

What is a production quality circle?

A production quality circle is a recurring meeting where a team reviews every error and failure it has logged, looks for patterns and root causes, and writes a new test or procedure to stop each one recurring. It includes both development and operations people, and it is run for error reduction and psychological safety rather than for accountability.

Why is a production data pipeline called a DAG of DAGs?

Because each tool along the way already has its own workflow or DAG, and each group of people owns one part of the value chain. Data engineering runs Airflow or Informatica, data science runs Python, visualization runs Tableau, and none of them see the others. A meta workflow above all of them is what organizes those separate DAGs into one production system.

Where to go next