An analyst on the finance team knows that when the active member count drops while claim volume holds flat, the eligibility file truncated. She has known this for six years. She checks the two numbers against each other by hand every month before the close.
Your data quality tests do not check it.
They cannot. The total member count fell 3 percent, which is under every threshold you have. The entire drop sits in one plan that lost 60 percent of its members, and nobody tests at the plan level. Both tables pass. Nothing is null, nothing is duplicated, no schema changed. The failure lives in the relationship between two feeds and inside a segment nobody thought to slice. Both of those facts exist only in her head.
Your tests check nulls. They check uniqueness on the primary key. They check that the row count did not drop more than 10 percent overnight. Those tests are real and they catch real problems. They also came out of a table profile, not out of anyone’s head. A profiler reads the data. It cannot read the business.
Generic tests catch broken pipelines. Business rules catch broken business. Statistical detection catches what nobody thought to check. You need all three, and you have one.
Asking for requirements does not work
The standard fix is to go ask the business. Send a spreadsheet. Ask each domain owner to list their data quality rules and return it by Friday.
You will get four rows back. Two of them will say “data should be accurate.”
This is not because business users are lazy. It is because nobody is good at authoring rules from a blank page. The analyst does not carry a list of rules in her head. She carries a reflex. She looks at a report, something is off, and she knows why within five seconds.
Reacting is easy. Authoring is hard. So stop asking people to author.
Show someone a rule that already exists and they will tell you immediately whether it is right. Show them a threshold of 5 percent and they will say the number should be 1 percent, because anything above that means the vendor file arrived twice.
You get better rules in a 20-minute review than in a six-week requirements gathering exercise.
Nobody has the whole elephant
Three people look at the same broken table and describe three different animals.
The engineer touches the pipeline. He knows the load ran at 2:14am, he knows the join fans out, and he knows the table came in at 4 million rows where it usually lands at 4.1 million. He does not know that a duplicate contract ID means somebody re-keyed a renewal instead of amending the original, and that finance will restate a number because of it.
The analyst touches the report. She knows the renewal number is wrong within five seconds of opening the dashboard. She cannot tell you that the merge statement matched on a nullable column. Ask her to write the fix and she will describe the symptom, correctly, in language nobody can turn into SQL.
The model touches the data. It sees that the column is 3 percent null this week and was 0.2 percent last week, and it will write you a clean test for that in one shot. It does not know whether anyone cares. It does not know that rule 212 already covers this table, that the last three times this fired it was the same vendor sending a duplicate file, or that the team decided in March to stop testing this column at all.
Each one is right about the part they are holding. None of them can describe the animal.
The usual answer is a meeting. Get everyone in a room, walk the rules, agree on the fixes. That works once. Then the meeting ends, the context goes back to living in three heads and a set of notes, and six weeks later you hold the meeting again with two new people who were not there the first time.
Who touches a data quality rule
The cast is also bigger than three, and it does not sort neatly into authors and readers.
Your business analyst knows what wrong looks like. She has seen the specific failure, in the specific column, on the report a VP opened at the wrong moment. She will never write SQL and she will absolutely tell you a threshold is wrong. Your data steward owns the definition. When sales and finance disagree about what counts as an active customer, the steward decides and the rule reflects that decision. The steward also retires rules that stopped applying two reorgs ago, which matters more than it sounds.
Your data quality lead lives in this all day. She wants to know what changed since last week and which failures are still open, and in most shops she is technical enough to write the rule herself, so give her the write path instead of making her file a ticket. Your data engineer implements and runs. He wires the rule into the pipeline, sets the schedule, decides whether a failure stops the load or just records itself, and owns the ten minutes after it fires. Your analytics engineer has a different title and the same job.
Your data product owner wants her tables and nothing else, because she is deciding whether she can promise a freshness commitment to the teams consuming her product. She will add a rule the moment she has to make that promise in writing. Your data scientist wants to know whether the features feeding his model were clean on the dates he trained on. That is a question about history, not about this morning, and today he cannot answer it at all. He finds out the model drifted, then spends three weeks reconstructing what March looked like. After that happens once, he writes his own tests.
Your chief data officer asks the broadest question and the one nobody can answer. How many tables do we test. How many rules do we run. How many fired last month, how many turned out to be real, and is that number moving the right way. Every quarter somebody hand-builds her a spreadsheet that goes stale before the meeting ends.
The line between the people who change rules and the people who only look at them does not follow job titles. It follows how technical someone is and what happened to them that week. The analyst who has never edited anything will edit a threshold the morning after a bad close. Build for that. A tool that ships a business view and an engineer view as two separate products has already made the wrong bet, because it froze a boundary that moves.
What these people share is the need to see the same thing. State, status, type, count, and whether the rule is still correct. Six people assembling that picture by hand, on different days, from different sources, will not agree on how many tests you run. That is the argument for one store, and it is a better argument than collaboration.
Most organizations skip all of this and hand the whole job to the engineer. He ends up guessing at business logic he does not know, defending thresholds he picked because they were round numbers, and fielding Slack messages from analysts who found a problem his tests missed.
One database of record for rules and results
Ask your team a simple question. How many data quality rules do you run?
Nobody will know. The rules are in a dbt YAML file, plus a legacy tool somebody bought in 2019, plus three stored procedures, plus a Python script on a cron job that a contractor wrote and never documented, plus a spreadsheet a steward keeps on her desktop.
Because the rules are scattered, you cannot answer the questions that matter. Which tables have no coverage at all? Who owns this rule? Why is the threshold 5 percent? When did this rule last pass? Did anyone review it after we changed the source system?
Put every rule in one place. Keep the plain language description, the table and column it runs against, the owner, the threshold and the reasoning behind it, the run history, and the current state. Make it readable by people without a Git account.
Keep the results next to the rules. Not in a separate logging system, not in a Slack channel scrollback that ages out after 90 days. Every run, every failure, the rows that failed, who looked at it, and what they decided. The rule list tells you what you check. The result history tells you what actually breaks, and those are two different lists. A rule that has failed 40 times this year and gets waved through every time is not a rule. It is a documented tolerance, and somebody should either fix the source, move the threshold, or admit it was never a hard failure.
That last one is how test suites die. A rule written for a discontinued product line keeps firing every night. Somebody mutes the alert. Then somebody mutes the channel. Six months later, a real failure lands in a channel nobody reads. Suites do not die from missing rules. They die from the noisy ones nobody owns.
Severity is the mechanism people skip. Every rule gets a tier. A hard fail stops the load and wakes somebody up, and you should have very few of those. A soft fail lets the data through and posts a warning to the rule’s owner. An informational check records the number and says nothing, so you can look at the trend in six months. Most of what a profiler writes for you on day one is soft-fail material. Tag every rule when you create it, revisit the tag when it misbehaves, and demote the ones that fire nightly instead of muting them. That is the whole difference between a suite people trust and a channel people filter.
Then review on a schedule. Once a quarter, put the list in front of the people who use the data and ask which rules still matter. Retire the dead ones. Tighten the loose ones. Add the two or three that came out of last quarter’s incidents.
Your data quality and test tool should open three doors onto that database. The business user gets a UI she can search, read, and comment in without a Git account. The engineer gets an API and version control, so rules move through CI like any other code. The model gets an MCP interface. MCP is the Model Context Protocol, an open standard for letting an assistant read a system’s data directly. Give the model one, and it can read the rule definitions, the profiles, and the full result history before it opens its mouth.
IMPORTANT
Decide which door is authoritative before you open any of them. Two writable copies of the rule set are the same disease you started with, and it is worse this time because both copies look official.
The pattern that works: the database holds the live rule set, and every change is exported to Git, so you get diffs, review, and rollback. A business user’s edit is a proposal until the steward approves it. The pattern that fails: rules in Git, rules in a UI, both writable, syncing on a schedule. That is how a threshold reverts every Wednesday, and nobody can explain why.
The rules nobody wrote
Everything so far assumes a human decided what to check. That covers what somebody thought of, which is not the same as what breaks.
Statistical detection covers the rest. Point a profiler at a column, let it learn the shape over 90 days, and it flags the week that does not look like the last twelve. Distribution shifts. Cardinality jumps. A field that ran 2 percent null for a year and came in at 9 percent this morning. Nobody wrote those rules, and nobody was going to. You have 4,000 columns in that warehouse, and no team is hand-authoring a distribution check on each one.
Seasonal baselines decide whether anyone keeps it turned on. Volume drops on weekends. Claims spike at month-end. The last week of the quarter looks nothing like the first. A detector that has not learned your calendar fires every Saturday and every close, and your team filters the channel inside of a month. A detector that has learned it stays quiet on Saturday and speaks up on the Saturday that is actually wrong.
Here is the part vendors skip. Learned rules need more human attention than authored ones, not less.
An authored rule arrives with a reason. Somebody wrote it because a specific thing broke, and they can tell you what. A learned rule arrives with a z-score and no story. “Distribution of plan_code shifted by 3.4 standard deviations.” Your engineer cannot tell you whether that matters. Neither can the model. Your analyst can, in about five seconds, because she knows the new plan launched on the first.
So learned rules get the same treatment as everything else. An owner, a severity tag, a line in the catalog, and a slot on the quarterly review. Two things differ. A learned rule starts as a soft fail, always, because you do not stop a production load on a statistical hunch. And the first review question changes. For an authored rule, you ask whether it is right. For a learned rule, you ask whether it is real.
That question is the whole loop. Detection finds something once, a human names it, and it becomes a rule with an owner and a reason attached. The plan_code shift gets looked at, turns out to be the eligibility file dropping a plan, and now you have a named rule that fires on that exact failure and explains itself. The detector found it. The analyst made it permanent. Do that 20 times, and your rule set knows things nobody ever sat down to write.
Point an LLM at the rules
Half the reason business users never review data quality rules is that the rules are written in SQL. Nobody is going to read a case statement with four nested joins and tell you whether the logic matches how sales defines a closed deal.
A model translates it. Point it at the rule, and it writes the sentence. “This test fails when the active member count drops more than 5 percent while claim volume stays flat, and it skips the first week of every month.” Now the analyst has something to argue with. She reads the second half of that sentence and tells you the exclusion is backward because the first week of the month is exactly when the eligibility file lands and exactly when it truncates.
That translation step is small, and it is the whole bridge. It turns a review meeting nobody could hold into one that takes 20 minutes.
The same trick works on the rule set as a whole. Point a model at all 400 rules in your catalog and ask which tables have no coverage, which rules duplicate each other, and which ones have not failed in two years. You get a review agenda in a few minutes. Go the other direction too. Paste in last quarter’s incident write-up and ask for the test that would have caught it. Most teams never write that test because the incident ends, everyone is tired, and the retro action item dies in a ticket.
Root cause analysis is another place this pays. When a test fails at 3am, the first 20 minutes are mechanical. Pull the failing rows. Check what changed. Compare this run against the last 30. Give a model your test definitions, your table profiles, and your full result history, and it comes back with a first guess and the evidence behind it. “Row count dropped 12 percent. The entire drop is in the EMEA plan segment. Rule 212 covers this table but only tests the total, which is why it passed last night.” Your engineer confirms or rejects in two minutes instead of digging for an hour. Your data quality and test tool should feed all of that directly to a model, via MCP or an API, so the assistant reads real definitions and real results instead of guessing from whatever you pasted into a chat window.
Know where this stops. The model gives you a hypothesis, not a verdict, and it will be confident and wrong sometimes. Your test results say what broke and where. They say nothing about why the upstream job ran late or which commit changed the transform, because that lives in Airflow or Databricks or Git, and you have to hand it to those systems too if you want it to reach that far. It does not know that the EMEA feed always runs light on the last business day of the quarter, and it will happily write you a rule that enforces a business definition nobody agreed to. A human signs off on every rule change. What you get is a narrowed search, not an answer.
The whole animal
Nobody on your team is going to wake up holding the entire data estate. Your engineer will keep not knowing what a duplicate contract ID does to the quarterly close. Your analyst will keep not knowing what a merge statement does to a nullable column. The model will keep not knowing what anyone cares about. That is not a training problem, and no amount of documentation solves it.
What you can fix is where the knowledge lands. One catalog holds every rule and every result, including the ones the profiler learned on its own. Generate the first pass automatically, because if your tool makes you hand-author the first hundred tests, you will never get past table one. Give the business user a UI, the engineer an API, and the model an MCP interface, and make one of them authoritative so the copies never fight. Tier the failures so people trust the alerts. Review the list once a quarter with the people who use the data, and name the anomalies that turned out to be real.
Then route each failure to whoever asked for the rule, with the failing rows attached, wherever she already reads. Not a shared channel with 40 people in it. Not a dashboard she has to remember to open. If she has to go look, she is still checking by hand, and you have bought nothing.
None of that is exotic. Teams skip it because rules feel like plumbing, and plumbing never gets a budget until it floods. Then the eligibility file truncates in one plan, the number goes out wrong, and somebody spends a quarter rebuilding trust that a 20-minute review would have protected.
The analyst who reconciles those two numbers by hand every month stops doing that. The check runs at 3am instead, and it fails the load before the report is wrong.
DataKitchen’s TestGen is the database of record
We built DataOps TestGen because we kept writing the same tests. Different client, different warehouse, same twenty checks. And nobody enjoys this work. Nobody has ever woken up wanting to hand-author a null check on the four hundredth column, and nobody wants to still be maintaining it two years later.
TestGen profiles your tables and writes a baseline test set on day one, so nobody has to stare at a blank page. Every test definition and every result land in a single Postgres database, along with the owner, threshold, severity, run history, and outcome. That is the database of record. It is not a side effect of the tool. It is the tool.
Stewards and business users get a UI where they can read what is tested, see what failed, and change a threshold without touching a repo. Engineers get an API, so tests move through CI like any other code, and you can write your own SQL when the generated tests do not cover your logic. The MCP server lets Claude or any other assistant read the profiles, the test definitions, and the full result history before it suggests anything.
TestGen will not invent your business rules. Nothing will. It gets you the mechanical coverage in an afternoon and puts it in front of the people who know the business, in language they can argue with. The arguing is still your job. It is also the part that catches the truncated eligibility file.
TIP
DataOps TestGen profiles your tables, generates a baseline test set, and stores every test definition and result in a single Postgres database. Browse it in the UI, drive it from the API, or point an assistant at the MCP server. Apache 2.0 licensed and free to run on your own hardware. Install Open Source TestGen.
TL;DR
Data quality tests generated from profiling catch structural problems: nulls, duplicates, schema drift, volume swings. They miss business rules, which live in the heads of the people who read the reports. Asking those people to author rules fails. Showing them existing rules to correct works. Business analysts, data stewards, data quality leads, data product owners, data scientists, analytics and data engineers, and chief data officers all need to see and sometimes edit the same rules, and none of them hold the whole picture. Neither does an LLM.
The fix is a single database of record for both rules and results, accessible via a UI, an API, and an MCP interface, with one authoritative copy, severity tiers, failures routed to the person who requested the rule, and a quarterly review. Statistical detection covers what nobody thought to check, and a human names the anomalies that turn out to be real.
