On-Demand Webinar · 54 min

How DataKitchen DataOps Automation Delivers End-to-End Data Observability

Chris Bergh on the testing and monitoring that make data observability real: the duality of tests in production and development, the kinds of tests a data team needs, and how to orchestrate them across a whole analytic system rather than one tool. Recorded March 2021; updated August 2026.

Presented by Chris Bergh

What you'll learn 7 points
  • Data observability is defined here as the set of technical practices, cultural norms, and architecture that enable low error rates, and as a subcomponent of DataOps. The framing is deliberate: it is not about data quality alone, it is about the error rate, which includes data quality, lateness against an SLA, a system processing issue, and a code change that broke something.
  • About 90 percent of your tests should run in both production and development. Tests have a dual nature, serving as production monitoring in one place and as regression, functional, and end-to-end tests in the other, and the majority do double duty.
  • Location balance tests confirm that data properties match business logic at each stage of processing. Historical balance tests compare current data against previous or expected values to judge whether new values are within a reasonable range. Both are statistical process control applied to a data factory.
  • Test results carry three severities and they do different work. An error stops the line, a warning is flagged for investigation later, and info produces a list of changes. Treating every failure as a stoppage is how alerting gets ignored.
  • Saving data is what makes development testing possible. After a production run finishes, keep the input production data, the test history, and the output data and artifacts, so the development team can run regression and historical balance companion tests against real data rather than invented data.
  • The Andon cord from the Toyota Production System is the model for stopping on a serious data error. Jidoka empowers an operator to detect an abnormal condition and immediately stop work, and the data equivalent needs a single pane of glass across every tool before anyone can pull the cord.
  • A data engineer can add testing without replacing an ETL tool: build in the tool you already use, call DataKitchen from anywhere in the process over a non-blocking REST API with parameters, and get tests run, alerts sent, and a control panel back.

Slides

53 slides

Questions from this session

What is data observability?

Data observability is the set of technical practices, cultural norms, and architecture that enable low error rates in data and analytics, and it is a subcomponent of DataOps. The scope is wider than data quality: it also covers lateness against an SLA, system processing failures, and code or configuration changes that break something downstream. The goal is to catch a problem before the people who depend on the data see it.

What is a location balance test?

A location balance test confirms that data properties match business logic at each stage of processing. As data moves from source to raw store to transformation to report, the test checks that counts, sums, or other properties still agree with what the business logic says they should be at that point. It catches internal inconsistency between steps that individual step-level checks miss.

What is a historical balance test?

A historical balance test compares current data to previous or expected values, using history as the reference for whether today's values are reasonable or within a reasonable range. It is how you catch a feed whose format quietly changed, or a volume that dropped by half, without anyone having written a fixed threshold in advance.

How many tests should run in production versus development?

Around 90 percent of tests should run in both places. A test that checks whether inputs are clean, whether business logic still holds, and whether outputs are consistent is a production monitor when the code is fixed and the data changes, and a regression test when the data is fixed and the code changes. Writing two separate suites duplicates the work and lets them drift apart.

What is an Andon cord for data pipelines?

The Andon cord comes from the Toyota Production System: a cord or button any worker can pull to stop the production line when something is wrong, part of the wider Jidoka principle that operators should detect abnormal conditions and stop work immediately. Applied to data, it means a test failure severe enough to halt processing before bad data reaches dashboards and models, which requires a single view across every tool in the chain.

How can a data engineer start data observability without replacing their ETL tool?

Keep building in the ETL or ELT tool you already use, then call DataKitchen over a REST API with parameters from anywhere in that process. The call is non-blocking, so it does not slow the pipeline; DataKitchen runs the tests, sends the alerts, and gives you a control panel with the results and their history. A fuller integration lets DataKitchen orchestrate the tool itself and stop on error.

Where to go next