A Guide to DataOps Tests

Testing Is the Foundation of DataOps

If there’s an error in data or analytics, the manager in charge is on the hook to report what happened, when, why, and how to make sure it doesn’t happen again. Observability is how easily the data team can get those answers. An observable system is built with transparency and instrumentation, so you get a complete picture of the data pipelines. In DataOps, tests provide the status at each stage of processing. Testing is the foundation of observability, and of DataOps itself.

Most talk about observability means observability of the data. DataOps testing covers that, and adds a second dimension: process observability, meaning visibility into every tool and every step of the data journey, from source to report. Data can be perfectly valid and eight hours stale. You need tests that see both.

Data changes continuously as it flows through the system. It drifts out of statistical range, defies data prep algorithms, and confuses machine learning models. DevOps testing verifies code. DataOps testing has to cover both analytics code and data.

Manual testing doesn’t keep up. A person runs tests one at a time, it costs a scarce resource like a data scientist, it’s error-prone, and it’s too slow to run often. DataOps automates testing, so tests run under orchestration every time the pipeline does. Whenever a problem occurs, you add a test. New tests accompany every analytics change. The suite only grows.

The stakes went up when AI arrived. When analysts and agents point LLMs at your tables, errors multiply rather than add: data that is 80% correct, read by an LLM that is 80% accurate, gives 64% accuracy. Any table can end up in front of a model now, so any table needs tests.

Some people write a few unit tests and proudly call it “DataOps.” Unit tests are a start, and plenty of teams never get that far. A data organization reduces its errors to virtually zero by implementing a breadth of tests at every step in its pipelines, and the rest of this paper is what that breadth looks like.

This 2026 edition rewrites the 2021 paper using our book, The DataOps Way to Data Quality and Data Observability, which is free. The book goes deeper on every section below.

The Four Places to Test

A data quality test is code that checks data. It’s scriptable, automated, and reusable. Take the simplest one you can imagine: check a column for nulls. Run it on source data and it’s a quality metric. Run it at ingestion and a column that was full yesterday and half-null today is an anomaly. Run it in production and a null in your gold fact table is a reason to stop the line. Run it in development and a column that goes null after a code change means you broke something.

Same test, four jobs, four places. If you only run it in one of them, you’re covered for one failure mode and blind to the other three.

Place	Metaphor	Action	Where it finds errors	Benefit
Data quality	Pets	Persuading the source	At the point of origin	Simple data-level integration
Data observability	Cattle	Polling for perversity	At the point of data ingestion	Simple data and data lineage-level integration
Data production	The data production assembly line	Protecting production	During data production, with a tripwire to stop processing	Stops customer-visible errors in data, reports, and models
Deployment to production	The code production assembly line	Protecting the push to production	In code, during development and deployment	Stops coding regressions before they reach production

The first two places look at data where it starts and where it lands. The third watches data as it’s made. The fourth watches the code that makes it.

Shift Left and Shift Down

A data operation runs two pipelines that cross. Data flows left to right through production, from source to the reports and models people use. Code and configuration flow top to bottom, from an idea through development into production. Each pipeline breaks in its own way, and each gets its own shift.

Shift left moves tests earlier on the value pipeline, toward the source. It’s the manufacturing idea of catching a defect at the station that created it, with an Andon cord any worker can pull to stop the line. The first three places to test are shift left: data quality on the source, observability at ingestion, and tripwires between production steps. Shift left protects the data you already have.

Shift down moves tests into the deployment path, before a change merges. Run the changed pipeline on data pulled from production, compare its outputs with the current version’s, and fail the build when a number moves that shouldn’t. That’s the fourth place. Shift down protects the data your next deployment is about to produce.

Most teams do neither. Their tests sit at the end of the line, validating the gold table right before a report runs, which is the most expensive place to find a problem and the worst place to learn which step caused it. DataKitchen estimated in 2025 that about 80% of data teams rely on hope-based testing in development, with no automated validation before changes reach production.

The reason to do both is the 1:10:100 rule from George Labovitz and Yu Sang Chang. Prevention costs 1, remediation costs 10, and a failure in front of a customer costs 100. The rule holds on both pipelines: a defect is cheapest to fix where it enters, whether it enters as a record or as a commit.

Data Quality: Persuading the Source

At a glance	Data quality
Metaphor	Pets
Action	Persuading the source
Problem	Poor source data quality, not fit for purpose
Where	Find errors at the point of origin
Goal	Improve source data quality, one table at a time
How	Lots of specific tests, data quality dimensions, data quality dashboards, and detailed checks. Focus on testing and scoring, then influence your data sources
Limitation	Errors in downstream tools and deliverables are invisible, so the analytic results delivered to customers may still be incorrect
Benefit	Simple data-level integration

In DevOps, a pet is the server you set up by hand and nurse back to health when it’s sick. Cattle are the herd you manage by the numbers. Source data is pets. You’re trying to make specific tables fit for purpose, one table and one column at a time, and you can’t automate your way out of it, because you usually don’t own the source.

That’s the hard part. The data you need to fix lives in someone else’s system. You can’t change it. You can only prove it’s wrong and persuade the owner to fix it. So data quality is an influence game, and the tests are your evidence. That’s the approach in Data Quality: The DataOps Way: focused, test-driven data quality dashboards, and the influence skills to get a source owner to act on them.

Specific Tests on Every Data Quality Dimension

Pets get lots of specific, detailed tests. Organize them by data quality dimension, so a score means the same thing on every table. TestGen groups its tests into six: accuracy, completeness, consistency, timeliness, uniqueness, and validity.

Dimension	What it asks	Examples
Completeness	Is everything that should be here, here?	Required fields are present, with no placeholder values. Row counts are in the expected range
Validity	Is each value a legal value?	US Zip5 codes are five digits. An NPI passes its check digit, not only its length. Dates fall in a believable range
Consistency	Does the data agree with itself and with other data?	90 percent of values match entries in a dimension table. End dates are after start dates
Uniqueness	Is each thing recorded once?	No duplicate customer IDs. No duplicate rows
Timeliness	Is it current?	Transaction dates are in the past. The newest record is from today
Accuracy	Is it right?	Body temperature is within a range around 98.6F/37C. A medical practice can’t have more appointments than doctors

A pattern test proves a value looks like an identifier. It doesn’t prove it is one. The transposed NPI is the standard example: ten digits, all numeric, two of them swapped, and a join that silently dropped one prescriber for months while the pattern test stayed green. Where a spec defines a check digit, test it.

Add usability to the list too. Data can be accurate, valid, and unusable because of inconsistent category spellings, mixed casing, invisible characters, or several different ways of writing “missing.” None of those checks needs business context, so they’re the cheapest fixes you’ll ask a source owner for.

Dashboards That Persuade

Scoring is how you win the influence game, and one generic dashboard doesn’t do it. Different stakeholders need different views: critical data elements, quality tied to a business goal, quality by data source so you can hold a supplier accountable, or the eight fields one data scientist actually uses.

The rule that matters most: every dashboard score traces to a data test. A score without a clear “fix this” path is noise. Each score breaks down into tickets, and each ticket is backed by a test that both demonstrates the problem and verifies the fix. When you hand a source owner a score, you’re handing them a list of failing tests they can rerun after they fix it.

Writing it down helps too. A data contract records what the source agreed to deliver, and the tests are what give the contract teeth.

Data Observability: Polling for Perversity

At a glance	Data observability
Metaphor	Cattle
Action	Polling for perversity
Problem	Undiscovered data errors in large sets of data tables, and getting blamed for “obvious” data problems
Who	The data ingestion engineer
Where	Find errors at the point of data ingestion
Goal	Find errors in large databases with thousands of tables before your customer does, and reduce data downtime
How	Poll the database and ping people when there may be an error. Watch hundreds or thousands of tables for variance in frequency, volume, schema, and drift, using time series anomaly detection
Limitation	Finds data errors after they get into production, sends some irrelevant pings, and leaves partial test coverage
Benefit	Simple data and data lineage-level integration

Now stop thinking about individual tables and start thinking about the herd. When source data lands in bronze, somebody owns that landing zone, and it isn’t six tables. It’s 600 or 6,000. That person doesn’t have time to care for each one. They need to know whether data arrived, whether it’s all there, and whether anything looks off. They’re scanning a thousand cattle for the one with the lame leg, or the one that isn’t there anymore.

Four Monitors for Every Table

Observability learns each table’s normal behavior over time and flags the runs that don’t fit. Four monitor types do most of the work.

Freshness asks when the table was last updated. If a table updates every Tuesday and it’s Wednesday with no data, something’s wrong. Volume asks whether the right number of records showed up. A table that normally grows by 1,000 rows and grows by 10 is a silent failure. A table that doubled overnight was double-ingested. Schema asks whether someone altered a column without telling you, and it’s the one monitor with no prediction model, because any change at all is an anomaly. Drift and custom metrics watch the values themselves.

Timing deserves its own check. A load can be complete and on schedule and still be late relative to what depends on it. Test arrival against the schedule the downstream consumers expect, not only against the previous run.

Statistical Process Control

Lean manufacturing measures every aspect of its process to catch issues as early as possible. These are time balance tests or, more commonly, statistical process control (SPC). SPC tests measure an aspect of the data pipeline repeatedly and screen the measurements for error or warning patterns, so a data team can catch failures before users see them in reports. Time series anomaly detection is SPC applied to thousands of tables at once.

The math is a commodity. The part that decides whether anyone keeps the monitor turned on is the calendar. Volume drops on weekends. Claims spike at month-end. A detector that hasn’t learned your seasonality fires every Saturday and every close, and your team filters the channel inside a month. Let a monitor learn a column’s shape over 90 days before you trust it. In our experience, fixed bands like “sales should not vary more than 10% week over week” are where this goes wrong, because they don’t know about your weekends or your month-end close.

What Observability Can’t Do

Observability tells you the data is late, short, or reshaped. It doesn’t stop anything. By the time you read the ping, the bad data is already in production. You’ll also get the occasional irrelevant ping, which is the price of anomaly detection, and your coverage is only as deep as the monitors. A table can pass every observability check with wrong values in it.

There’s no production tripwire and no view of the tools acting on the data. That’s the next place. Still, in the land of the blind, monitoring every bronze table beats hoping, and you can stand it up in about an hour.

Data Production: Protecting Production

At a glance	Data production: data journey observability
Metaphor	The data production assembly line
Action	Protecting production
Problem	How do you stop errors in data and deliverables before they get into production?
Who	The data engineering team
Where	Find errors during data production, and place a tripwire to stop processing
Goal	Find all errors during data production, and stop processing so that no errors get into production
How	Put tests and monitors in the data production processing steps, with a test-aware orchestrator or a data journey observability tool, and full test coverage
Limitation	Requires both data and tool integration
Benefit	Stops customer-visible errors in data, reports, and models or AI

Now we get to the production process itself, where data moves through bronze, silver, and gold with code transforming it at every step. Think of data analytics as a manufacturing pipeline. There are inputs (data sources), processes (transformations), and outputs (analytics). A manufacturer puts a quality control station at every step, because catching a problem at incoming inspection costs far less than catching it in finished goods, and it stops the line when it finds a defect.

DataKitchen’s view is blunt. You need to test every table in every zone. You need to monitor every tool and pull them into a single view of your process. And you need tripwires that stop production at every step of the data journey. Data lineage is not enough: lineage is a blueprint, and production needs a fire alarm panel.

The figure below shows a data analytics pipeline. Analytics access databases, transform data for input into models, and the models feed the visualizations and reports people use.

Along the way, tests ask three questions. Are data inputs free from issues? Is business logic still correct? Are outputs consistent?

Data input tests keep bad data out of later stages, the way incoming quality control does in a factory. Letting bad data through wastes processing and raises the odds nobody ever catches it. Data output tests verify that a processing step ran correctly. Business logic tests check data against things you know are true about the business. Say every European customer is assigned to a member of the Europe sales team. Test results saved over time let you check quality against historical levels.

Tests validate data inputs and outputs, and verify that data is consistent with business logic

Test category	What it checks	Examples
Inputs	The data arriving at a processing step	Count verification: row counts are in the right range. History: the number of prospects always increases. Balance: totals reconcile with the prior step. Temporal consistency: transaction dates are in the past. Field validation: all required fields are present and correctly entered.
Business logic	Whether the data matches what you know about the business	Customer validation: each customer exists in a dimension table. Domain rules: a medical practice can’t have more appointments than doctors.
Output	The result of an operation, such as a cross-product join	Completeness: the number of customer prospects increases with time. Range verification: the number of physicians in the US is less than 1.5 million. Value set: every region in the source appears in the output, with matching distinct counts.

Tests That Find Problems Before Users Do

The data team’s work product is under a microscope. Business users are experts in their own domain and see problems in analytics at a glance. Learning about errors from your internal and external customers is an unpleasant experience, and it’s the default for a team without production tests. Two tests from the 2021 edition of this paper still do most of the work: location balance and historical balance.

Location Balance

Location balance tests make sure data properties match business logic at each stage of processing. An application may expect 1 million rows to arrive via FTP. The location balance test confirms the right quantity arrived, that the same quantity is in the database and in later stages of the pipeline, and finally that it’s in the reports.

In a Medallion architecture, this is row-count reconciliation between layers: bronze to silver, silver to gold. Don’t hardcode the expected count. Compare against the control count the source sends, or against the prior layer. Where no control count exists, test that the increment falls in the range history predicts. And match sums as well as counts, because a load can keep every row and still lose the cents.

Historical Balance

Historical balance tests compare current data to previous or expected values. History is the reference for whether today’s values are reasonable. A test can check the top fifty customers or suppliers. Did their values go up or down unexpectedly compared with history?

Correct analytics aren’t enough. Accurate analytics that “look wrong” to users raise credibility questions. The figure below shows how a change in SKU allocations, moving from pre-production to production, changes the sales volumes for product groups G1 and G2. The VP of sales will notice this immediately and report that the analytics look wrong. The report is correct, and it still reflects badly on the data team, because it seems wrong. What changed? The data team has no ready answer. Guess who is in the hot seat.

A historical balance test would have warned the data team ahead of time that product group volumes had shifted. That warning gives you a chance to investigate and tell users before they find it. Instead of hurting your credibility, the episode builds it, because it shows users the reporting is under control. “Dear sales department, you may notice a change in the sales volumes for G1 and G2. This difference reflects a reassignment of SKUs within the product groups.”

The modern form is an aggregate balance test. Five years after this figure was drawn, it shows up in commercial pharma almost unchanged: on a monthly refresh, one product group’s dispense totals jump while another drops by about the same amount, because an NDC was remapped from one market basket to another. Nothing failed on load. Only the balance test sees it.

Tripwires Stop the Line

A test is only worth something in production if it can stop the pipeline. A tripwire is a test task placed between two processing steps. It runs a test suite and exits with an error on failure. The orchestrator treats that as a failed task, so the next step never runs, and production keeps serving the last good data.

Stopping matters more than alerting because an alert arrives after the data has already moved. A model trained on a column that silently lost 30 percent of its values can’t be undone by a notification. Write the test results back next to the data, as tags on the tested tables and columns in your catalog, so the whole team sees quality status where it already looks.

Monitor Every Tool, Not Just the Data

Data is only half of production. Tools act on the data, and tools break in ways a data test won’t catch. An Airflow job hangs for three hours for no obvious reason. A report’s scheduled refresh silently doesn’t run. The necessary and sufficient condition for no customer-visible errors is to check the data and monitor the tools acting on it, for errors and for timing.

You have two ways to do it. A test-aware orchestrator runs the checks as part of the tool itself. When you have several orchestrators, one loading data, another running bronze, another handling gold, and a script kicking off the BI refresh, you need something sitting above all of them. That’s data journey observability: one view of every step across every tool, with tripwires in it. This is the place that takes the most integration work, and it’s the one your customer actually feels.

Failure Modes

A disciplined data production process classifies failures by severity. Some failures are fatal and the pipeline has to stop. In a factory, the most severe errors “stop the line.” Some failures are warnings: a member of the data team investigates whether a source changed, or a definition. Many test outputs are informational. They record a number so the data engineer can read the trend later.

Actions required for different failure modes

Severity	What the pipeline does	Who hears about it	How many you should have
Hard fail (error)	Stops. The next step never runs	The on-call engineer, right away	Very few
Soft fail (warning)	Lets the data through	The rule’s named owner, who investigates	Most rules that matter
Informational	Lets the data through	Nobody. The number is recorded for the trend	As many as are useful

The rules for assigning a tier are short. Tag every rule when you create it. Hard constraints, like a primary key that must be unique, can fail loudly from day one. A learned or statistical rule always starts as a soft fail, because you don’t stop a production load on a statistical hunch. Review the list once a quarter with the people who use the data: retire the dead rules, tighten the loose ones, and add the ones last quarter’s incidents taught you.

A complex process can have thousands of tests running continuously. When a hard or soft fail occurs, the right person should hear about it in real time, through email, text, or a service like Slack.

The way test suites die is noise. A rule written for a discontinued product line fires every night. Somebody mutes the alert. Then somebody mutes the channel. Six months later, a real failure lands in a channel nobody reads. Route each alert to a named owner, not a shared channel. Treat an alert that keeps firing without a real incident as a broken threshold and fix it, or demote the rule to informational. Never mute it.

Deployment to Production: Protecting the Push to Production

At a glance	Deployment to production: regression and impact testing of new code
Metaphor	The code production assembly line
Action	Protecting the push to production
Problem	Poor code gets into production, and your customers notice
Who	The data engineering team, and the DataOps engineer who controls the deployment process
Where	Find errors in code during development and deployment to production
Goal	Stop coding regressions before they get deployed to production
How	Regression and impact testing in development and CI/CD
Limitation	Requires test data, tools, and CI/CD integration
Benefit	Stops coding regressions before they reach production. Under the 1:10:100 rule, a regression costs up to 100 times more once it reaches a customer

The first three places are about data. This one is about code. You have a second manufacturing line, and it doesn’t make data. It makes the transformations that produce the data. The job is to prove that the new code works and that it didn’t break the code that was already there.

At this point some of you are thinking software development methods have nothing to do with me. I’m a data analyst or scientist, not a coder. I’m a tool expert. What I do is a sophisticated form of configuration. It’s a common view in data analytics, and it slows analytics cycle time. Tool vendors have a business interest in the idea that staying inside their tool protects you from the complexity of software development. Don’t buy it.

Don’t get us wrong. We love tools. But data analytics is code. Python, SQL, R: all code. The tools of the trade (Informatica, Tableau, Excel, and the rest) are code too. Open an Informatica or Tableau file and it’s XML, with conditional branches, loops, and sometimes embedded Python or R. And in 2026, AI agents write a growing share of that code. The data professional creates code and has to own the complexity that comes with it.

Regression and Impact Tests Come First

Development testing has a maturity ladder. At the bottom is hope-based testing: ship it and see. Next is static analysis, such as SQL linting. At the top is dynamic testing, which runs the changed code against real data. Most teams sit at the bottom.

Here’s the failure that regression testing catches. You rename a column in silver to fix a misspelling. A process in gold reads that column, so it breaks, and the reports that read gold break with it. A unit test never sees it. A manual business review doesn’t either. Your customer does.

So run the same bronze, silver, and gold pipeline you run in production, in development or QA, with test data pulled from production: a filtered slice or a zero-copy clone. Your production data tests pull double duty here, because the same checks that watch the data now tell you whether your change moved a number it shouldn’t have. That’s impact testing. Post the before-and-after on the pull request so the reviewer sees it.

The marker of a high-performing team is that you can hand a new engineer the keys, let them rename a column, and have a red light go off before it reaches production. Unit tests and manual reviews won’t get you there. A fully automated system with adequate test coverage will.

Types of Tests

The software industry has decades of experience making sure code behaves as expected, and each type of test has a specific goal. With cloud infrastructure and infrastructure-as-code, data teams face much less resistance to building large test suites. If you discuss testing with your peers, these terms come up:

Test type	What it verifies	In a data pipeline
Regression	Previously working features still work after a change	Run the full pipeline on production test data and compare the outputs. Build this first
Impact	What a change does to downstream tables and reports	Compare row counts, null rates, and key metrics before and after the change
End to end	The pipeline as a whole works	Bronze through gold through the report, in a QA environment
Integration	Components work together	A transform and the table it feeds, or two tools handing off a file
Unit	One section of code does its job, including corner cases	A function or a single SQL model. Parameterize it so one test covers many cases. Useful, and not a data quality check
Functional	A feature meets its specification, usually as a black box	A new metric matches the definition the business signed off on
Performance	Speed, stability, and scalability under load	How long environment creation, deployment, and error resolution take
Smoke	Major functions are up	A quick check before the full suite runs

Don’t start with unit tests. In about 850 comments from data engineers about why they don’t test, “unit tests miss real failures” was one of the reasons given, and the practitioners who did test started elsewhere: uniqueness, foreign keys, nulls, and dates, then reconciliation. Put regression testing on top of those, and add unit tests once they’re in place.

Testing AI-Generated Pipelines

Producing pipeline code got cheap. Verifying it didn’t. When an agent writes a 200-line transform in nine seconds, the push to production is the bottleneck, and every weakness in your test suite gets ten times the traffic.

The seven deadly sins of AI-generated data pipelines each map to a test you can write:

Sin	What goes wrong	The test
Gluttony	A retry loads every row twice	Idempotency: run the job twice and get the same table
Pride	The model assumes it knows your business semantics	Business rules from people who do, checked against real column profiles
Greed	4,000 new lines of SQL behind 40 old tests	Coverage that grows with the code, generated from the profile
Envy	A lineage graph instead of tests	Value-set tests: add ‘APAC’ to a CASE statement and the graph doesn’t move, but the distinct-count test fails
Sloth	Writing tests takes three times longer than generating the pipeline did	Generate the mechanical tests, write only the business ones
Vainglory	The agent grades its own homework	An external test harness running against real rows
Lust	Speed over verification	Tests inside the agent’s loop, called on every change

The rule underneath all seven: an agent never grades its own homework. Four things hold you to it. A FITT data architecture (functional, idempotent, tested, two-stage), so every step can be rerun and checked. Real column profiles as the agent’s context, so it isn’t guessing what the data looks like. Generated test coverage, so the agent has a definition of done. And the testing tool wired into the agent’s loop over the Model Context Protocol (MCP), so the agent runs tests it didn’t write, through a harness it doesn’t control. TestGen’s MCP server exposes 112 tools across the whole data quality loop.

Agents can also help fix what the tests find, with a person in control. One program uses the test results to find problems and propose exact SQL fixes, with the affected row counts, in a ticket. A person approves or rejects each one. A second program applies only what was approved. The SQL an agent writes is usually right in mechanics and sometimes wrong in intent, and intent is the part that damages production data.

Where Tests Come From

The coverage standard in the next section works out to thousands of tests across the four places. Nobody writes thousands of tests by hand, and that’s where the 2021 edition of this paper needs the biggest correction.

In software engineering, a common rule of thumb is that developers spend about 20% of their time writing tests. The 2021 edition asked the same of data teams. That’s still the right instinct, and the wrong budget. With AI and a test generator like DataOps TestGen, you spend far less. TestGen profiles each column across 55 characteristics, flags 32 kinds of data hygiene issues, and generates 34 test types from the profile, along with every freshness, volume, and schema monitor. That’s about 80% of the tests you need. Your time goes to the other 20%: the business rules.

That split matters, because generated tests have a blind spot. Generic tests catch broken pipelines. Business rules catch broken business. Statistical detection catches what nobody thought to check. You need all three.

Here’s the blind spot in numbers. In August 2026 we profiled NYC taxi rides, starting with a 300,000-row sample, and generated tests from the profile. Then we gave an AI model the profile through TestGen’s MCP server and had it write business rules.

What the 300,000-row sample held	Generated tests	Business rules
3,763 negative fares, one as low as -$367	Passed	Failed
14,252 payment codes outside the documented range	Passed	Failed
17,477 trips with zero passengers	Passed	Failed
9 trips dropped off before they were picked up	Passed	Failed

All 62 generated tests passed. All 7 business rules failed. A test generated from your data learns what your data already looks like, so impossibility reads as normal. The whole run took about five minutes.

Then we ran the full month, all 2,964,624 rows. The business rules failed again at almost exactly ten times the sample counts: 171,627 trips with zero passengers, 140,162 bad payment codes, 37,448 negative fares. And four of the generated tests failed too. They were minimum-value tests, and the full month held a few fares and totals below anything in the sample, down to -$899. That’s the other half of the lesson. Generated tests don’t know what’s impossible, but they do notice when the data moves past what it used to be.

The best business rules live in someone else’s head, usually the analyst who knows which source feeds a number differently from the rest. Don’t ask that person to write rules from scratch. Show them the existing rules and ask what’s wrong. That’s a 20-minute review instead of a project nobody finishes.

Spend the hand-written effort where a wrong number reaches someone who can escalate it. Generated coverage goes on every table, because it costs almost nothing. Business rules go on the tables behind the reports that matter.

How Many Tests Is Enough

Partial coverage is better than nothing, and it keeps the problems coming. If you test one table out of a hundred, you’re still praying about the other ninety-nine. The standard is common sense: cover every table in every layer, and cover every tool.

The coverage standard:

What	Minimum tests
Every table	Two to three, covering freshness, volume, and schema
Every column	At least two
Every significant business metric (sales by region, production volume)	At least one custom test
Every tool that touches data	One check for errors in its logs
Every job	One status check and one timing check

Now do the math on a real Medallion with three database levels, three tools, and six jobs. Bronze has 100 tables averaging 10 columns: 200 table tests and 2,000 column tests. Silver is the same. Gold has fewer tables and more columns: about 20 table tests and 600 column tests, plus the business metric tests. The tools add three error checks, and the six jobs add twelve status and timing checks. That’s roughly 5,000 tests.

Write half of them by hand at 30 minutes each and you’ve spent 1,250 hours, or 7.2 months of a senior engineer doing nothing else. That arithmetic is why “I don’t have time” is the most common reason data engineers give for not testing, and it’s why generation is the only way to reach the standard.

Watch Coverage Over Time

Test metrics show whether test coverage is keeping up. The figure below shows the number of tests at each node of an analytics pipeline, drawn as a directed acyclic graph. A per-node count is still the fastest way to see thin spots.

Track the total number of tests and the tests per unit of pipeline together. In the next chart, the total climbs from 36 to 125 over seven months while tests per node stays flat between 8 and 10. The pipeline grew as fast as the suite, so coverage never improved. With AI writing code, this is the chart to watch every month.

Counts tell you whether tests exist. Two more measures tell you whether they work. First, of your last five production incidents, how many did a test catch before a user did? Second, how many tests fire without a real problem behind them? The first number should go up. The second should go down.

The Role of TestGen and DataOps Observability

A data pipeline uses many tools, each doing one job. Imagine learning every tool in the pipeline and writing tests inside each one for its own slice of the work. A common framework that spans the pipeline, from data sources to published analytics, lets you write tests for every step in one place. DataKitchen splits that job across two open-source products, because the four places need both: tests on the data, and monitors on the tools.

Place	DataOps TestGen	DataOps Observability
Data quality	Tests, hygiene detection, and quality scorecards on source tables
Data observability	Freshness, volume, schema, and metric monitors on every table
Data production	Test suites run as tripwires between processing steps	One view of every tool and step in the data journey, with timing and error checks
Deployment to production	The same test suites run in CI/CD as a quality gate	Journeys watched in QA as well as production

DataOps TestGen checks the data. It profiles your tables, generates the tests and monitors in this paper, and runs them as SQL inside your database, so no rows leave your network. Your assistant or agent can call it over MCP.

DataOps Observability checks the tools. It watches every step of the data journey across Airflow, Databricks, dbt, Fivetran, and BI tools like Power BI and Tableau, and catches the job that hung for three hours or the report refresh that silently didn’t run.

DataOps Automation is where the development tests run. It gives each engineer an isolated Kitchen workspace with its own tools, datasets, and tests, so today’s code can run against yesterday’s data without touching production.

Start small. A read-only connection to your bronze database and an hour of setup gets freshness, volume, and schema monitoring on every table you have. That’s data observability, the fastest of the four places to stand up, and it costs nothing.

Conclusion

A unified, automated test suite that checks both production data and analytics code is what makes DataOps work. There are four places to test, and you need all four. Persuade the source so your data is fit for purpose. Poll for perversity so you catch what arrives broken. Protect production with tripwires so nothing bad reaches your customer. Protect the push to production so your code changes don’t break what already works. Cover every table, every column, and every tool, generate most of the tests, and write the business rules yourself.

For the full argument behind every section of this paper, read the free book The DataOps Way to Data Quality and Data Observability. For more on the practices this paper depends on, see Data Quality: The DataOps Way on running data quality as a continuous practice, DataOps Observability: Taming Chaos on tracking every journey data takes from source to customer value, DataOps TestGen on generating these tests automatically from a data profile, and 7 Steps to Implement DataOps, which opens with the same instruction to add tests.

---

FAQ