On-Demand Webinar · 51 min

DataOps Risk Insurance & Mission Control

Chris Bergh on treating DataOps as risk insurance for a data investment: how you know the data is right, where the risk hides in models and reports, and what a Mission Control for the data organization does. Recorded June 2022; updated August 2026.

Presented by Chris Bergh

What you'll learn 7 points
  • At a top-20 US company by revenue, the head of data took a call from the CEO about a compliance report that came back empty. The cause was one blank field passing through the pipeline, 26 people spent a day chasing it, the deploy cycle was six weeks, and a thousand other pipelines sat in the same hope-it-works position.
  • The 2022 DataKitchen and data.world data engineer survey found 52 percent hope and pray things don't break, 78 percent are stressed enough to want a therapist, 70 percent expect to change jobs within a year, and 79 percent have considered leaving the career entirely.
  • DataOps Mission Control takes its model from how NASA and SpaceX handle the risk of space flight: one interface carrying information about every aspect of the flight, decisions and communication driven from that information, history stored for after-the-fact analysis, and automatic alerts.
  • The unit Mission Control watches is the observational meta-pipeline, a layer above the DAGs, jobs, and schedulers already running. It records only the major steps a team cares about, treating the dozens or hundreds of sub-steps inside each tool as noise, and it holds fan-in, fan-out, causal, temporal, manual, periodic, evented, and inferred relationships that most organizations keep as tribal knowledge.
  • IT hardware and application performance monitoring are lagging indicators of data problems. APM tools do not check the data, the integrated data, or the reports and models built from it, and they give no context for the pipelines, jobs, and tools acting on that data.
  • A data catalog and data lineage answer what the data is and where it came from. Two governance questions remain: can I trust the data, answered by test results attached to each artifact, and is the data fresh, answered by process lineage recording when it was last updated and at what version.
  • The stated adoption sequence is Mission Control first as the pain pill that reduces risk, then DataOps Automation as the vitamin that drives customer value. The closing framing is that it is not about data quality, it is about a low rate of errors.

Prefer to read it? The written version is in DataOps Mission Control And Managing Your Data Infrastructure Risk.

Slides

58 slides

Questions from this session

What is DataOps Mission Control?

DataOps Mission Control gives visibility of every journey data takes from source to customer value, across every tool, environment, team, and customer, so problems are detected, localized, and raised immediately. It works by testing and monitoring every data analytics pipeline in an organization, in development and in production. The model is borrowed from NASA and SpaceX: one interface with the whole flight on it, history stored, and automatic alerts.

What is an observational meta-pipeline?

An observational meta-pipeline sits above the operational pipelines, DAGs, jobs, and schedulers a company already runs, and represents the process at the level a team actually cares about. A warehouse build might reduce to five steps: ingest data, create dimensions, build the fact table, run predictions, extract and report. The sub-steps inside each tool are noise until you need to drill down and diagnose.

Why is IT infrastructure monitoring not enough for data pipelines?

Disk and CPU metrics are lagging indicators of a data problem, and by the time they move the report is already wrong. Application performance monitoring tools do not check the data, the integrated data, or the reports and models created from it. They also give no context that ties pipelines, jobs, and tools into a coherent picture of production and development work.

What production questions should a data team be able to answer?

Whether source files arrived on time, whether the source data is the right quality, whether the report being read is fresh, and whether a particular supplier or pipeline is a repeat offender. On the job side: did every job that was supposed to run actually run, did job X run after group Y finished, and how long did yesterday's jobs take. Most teams cannot answer these without a manual hunt.

What is the Andon cord, and how does it apply to data?

The Andon cord came out of the Toyota Production System: a cord or button any worker could pull to stop production. The related idea is jidoka, which empowers operators to detect an abnormal condition and stop work immediately. Applied to data, it is the mechanism for halting a pipeline when the data or the processing is bad enough that continuing would push the error to customers.

How does mission control relate to data governance?

A data catalog answers what data exists and what it means, and data lineage answers where it came from and where it moved. Two questions are left over: can I trust this data, which test results attached to each artifact answer, and is it fresh, which process lineage answers by recording when the data and the reports built from it were last updated. The combination is called DataGovOps.

Where to go next