A data quality test fails at 6am. Row count on claims_daily dropped 40%. Now what?
You need to know which job loaded that table, whether it finished or died halfway through, what sits downstream of it, who looks at those reports, and whether anyone opened a ticket about this last quarter. That information exists. It lives in your catalog, your orchestrator, your lineage tool, your git repo, your ticketing system, and in the heads of two people on your team. Nobody has all of it on one screen. So you either guess or spend the morning pulling it together by hand.
The quality of the decision tracks the amount of context behind it. That’s the whole argument of this post. Your LLM is the one thing that can hold all that context at once, so put it at the center of the process.
Find problems before they reach production
That is the job. We’ve been doing data engineering consulting since before DataKitchen existed, and the lesson hasn’t changed. Monitor the data and the tools acting on it. Set tripwires that stop the run when something breaks. A bad file that never lands in the warehouse costs you nothing. The same file, discovered by the CFO on a dashboard, costs you a week.
When data breaks, the useful question is which process ran wrong. Column-to-column lineage tells you what feeds what. It doesn’t tell you that the 3am Airflow task retried twice and then wrote half a table. A test-informed orchestrator gets you part of the way there. We’ve written before about why lineage alone isn’t enough and what a data journey adds.
Even with that in place, the decision is still yours.
The decision is the hard part
Knowing a test failed is cheap. Deciding what to do about it is expensive. Someone has to review the open issues and pick which one to fix first. Someone has to review the existing tests and decide whether they’re the right ones or write new ones. Someone has to know that claims_daily feeds the Tuesday reimbursement report and that finance reads it at 8am. And someone has to remember that this same table did this same thing in March, and the fix was a vendor file that arrived in the wrong encoding.
Part of the reason is how the system is built. Most data architectures are loosely coupled, and the seams follow the org chart. An ingest team, a data production team, a data science team, and a BI team, some reporting to IT and some to the business. Each one picked its own tools in its own decade. Airflow in one department, Informatica in another, Power BI in a third, and Python scripts everywhere in between. Nothing in that stack knows about the rest of it. When something breaks, the failure crosses four teams and three technologies, and the person diagnosing it has credentials for maybe two.
That person is usually your most senior engineer. They’re the bottleneck because they’re the only one with the full picture. The picture isn’t written down anywhere.
Only a central LLM holds all the context
Look at what the decision draws on. Data catalog entries and owners. Profiling results. Test definitions and their history of passes and failures. Data lineage: what connects to what. Process lineage: what just ran and what failed. The SQL and Python that run in the pipeline. Your ticketing system, both the past tickets and the one you’re about to open. And the context nobody can buy: your customers, your domain, your department’s habits and workarounds.
No data quality tool has that. No catalog has that. No orchestrator has that.
Every vendor now ships a cute named agent. Elementary has Ella. Sifflet has Sage. Alation has Allie. Soda has Ask AI and Contract Copilot. Collibra has AI Copilot, Atlan has Atlan AI, and Informatica has CLAIRE Copilot. Ask any of them a question, and it answers well inside its own product. Its scope ends where the product ends. Maybe it has lineage, maybe it doesn’t. Maybe it can read the process code, maybe it can’t. Maybe there’s a catalog behind it, maybe there isn’t. Each one meters its own usage and sees its own slice.
This isn’t a theory. OpenAI wrote up the data agent it runs internally for 3,500 people across 70,000 datasets. Their finding was that even a strong model gave badly wrong answers, like misestimating user counts, until they surrounded it with layers of context: schema metadata and lineage, historical queries, expert-written table descriptions, and a nightly job where Codex reads the pipeline code that produces each table and records what it contains and how fresh it is. The model didn’t change. The context did, and the answers followed.
The LLM your company already standardized on can do the same thing. Wire it to everything, and it sees the whole picture. Claude or ChatGPT, connected to all your tools, has more context than any single vendor’s agent.
Wire it through MCP
Feed your LLM through MCP servers. Your data quality tool’s profiling and test results. Your governance tool. Data lineage. Process lineage from your observability tool. Orchestration steps and the source code behind them. Your ticketing system. And a document, or a set of documents, that describes your business the way you’d explain it to a new hire.
Be honest with yourself about which of those have MCP servers today. Some do. Some have an API you’ll wrap in an afternoon. Some you’ll have to build. The point is that each one is a bounded piece of work, and once it’s done, every future question benefits.
Now when the claims_daily test fails, you ask one question. The LLM pulls the failed run, reads the task that wrote the table, checks the diff in git from last night’s deploy, walks the lineage to the reimbursement report, finds the March ticket, and tells you: this is the same encoding problem, here’s who owns the vendor feed, and the report goes out in two hours. Upstream cause and downstream blast radius, prioritized against what your business actually cares about.
Keep deterministic tests underneath
The obvious objection: LLMs hallucinate, and you want one at the center of your incident response? Yes, and the objection is why the architecture works the way it does.
The most common failure in LLM-based root cause analysis isn’t invented evidence. It’s misread evidence: the model gets real logs and real metrics and draws the wrong conclusion from them. Picking a better model narrows that gap but doesn’t close it. What closes it is giving the model ground truth it didn’t produce. Your data quality tests should be algorithmic and deterministic. Run them twice on the same data, and get the same result. The LLM reasons over those results. It doesn’t generate them. We made the same argument from the other direction last year: an LLM fed untested data produces confident analysis of garbage.
The same split applies to writing tests. A profiler can see that discount_pct ranges from 0 to 100. It can’t know that anything above 40 needs a VP signature. Your LLM can, because you told it. Point it at the profile, describe what correct means for your data in plain English, and let it write the test definitions. Then check them against the raw table. The LLM contributes the business context. The test still runs deterministically.
Automate the repeatable parts with agents
Once the LLM has the context, most of these workflows repeat. Agents and skills can run them. That saves time, and it encodes company knowledge that currently lives in one engineer’s head.
The market hasn’t settled on an agent architecture. Most companies build agents their own way, and that’s fine. That matches the data architecture you already have, which was also built department by department. The cost of building an agent has never been lower. With Claude Code, you can stand up a working one in an afternoon.
Five to start with: data quality remediation, cross-database reconciliation, root cause and blast radius analysis that opens the ticket for you, pipeline change recommendation, and test and hygiene triage. Each one is a workflow your team already does by hand, badly, at the wrong time of day.
Why the center matters
More information in the decision means a better decision. Your security team will ask what leaves the building, and the answer is nothing new: you run this against the model endpoint your company already approved, with the same controls. You want one smart place where the data, code, history, and business context meet. The alternative is what you have now: eight tabs and one tired engineer connecting the dots.
Where DataKitchen fits
Two of the context sources above are from us and are open source. DataOps TestGen ships an MCP server with 96 tools that provide your LLM with its profiling results, test definitions, and pass/fail history for every table. Its rule generation is deterministic, so the ground truth stays ground truth. DataOps Observability supplies the process lineage: what ran, in which tool, whether it finished, and what sits upstream and downstream of the failure. TestGen runs behind your firewall under Apache 2.0, and the MCP server authenticates as you, so the agent gets your permissions and nothing more.
If you want to see what the agent side looks like in practice, we wrote up a night where one fixed 14 data quality failures before anyone woke up, and we ran a webinar on when the dashboard still wins and when it’s safe to turn agents on. For the test-writing part, we did it with 3 million NYC taxi rides in five minutes.
FAQ
What are the key points in this blog?
When a data quality test fails, the information needed to decide what to do is spread across the catalog, orchestrator, lineage tool, git repo, ticketing system, and people’s heads, so the decision is made on partial context. A central LLM wired to every tool through MCP can hold all of it at once, while deterministic tests underneath supply ground truth. Once that context is central, agents automate the repeatable workflows.
Why can’t a vendor’s built-in AI agent do root cause analysis across my data stack?
Because its scope ends where the product ends. Elementary’s Ella, Sifflet’s Sage, Alation’s Allie, Soda’s Ask AI, Collibra’s AI Copilot, Atlan AI, and Informatica’s CLAIRE Copilot each answer well inside their own tool. Maybe one has lineage, maybe it can read the process code, maybe there is a catalog behind it. Each meters its own usage and sees its own slice. A failure that crosses four teams and three technologies needs a view that spans all of them.
What context does an LLM need to decide what to do about a failed data quality test?
Catalog entries and owners, profiling results, test definitions with their pass and fail history, data lineage showing what connects to what, process lineage showing what just ran and what failed, the SQL and Python in the pipeline, past and pending tickets, and the business context nobody can buy: your customers, your domain, and your department’s workarounds. OpenAI found its internal data agent gave badly wrong answers until it had that surrounding context.
How do you connect an LLM to your data quality and observability tools?
Through MCP servers, one per source: your data quality tool’s profiling and test results, your governance tool, data lineage, process lineage from your observability tool, orchestration steps and their source code, your ticketing system, and a document that describes your business the way you would explain it to a new hire. Some tools ship an MCP server today, some have an API you wrap in an afternoon, and some you build. Each is a bounded piece of work that every future question benefits from.
Won’t an LLM hallucinate if it sits at the center of data quality incident response?
The common failure is misread evidence rather than invented evidence: the model gets real logs and metrics and draws the wrong conclusion. A better model narrows that gap but does not close it. What closes it is ground truth the model did not produce. Data quality tests should be algorithmic and deterministic, returning the same result on the same data every time. The LLM reasons over those results and does not generate them.
Which data quality workflows should you automate with agents first?
Five that most teams already do by hand at the wrong time of day: data quality remediation, cross-database reconciliation, root cause and blast radius analysis that opens the ticket for you, pipeline change recommendation, and test and hygiene triage. Each encodes knowledge that currently lives in one engineer’s head. With current coding tools a working agent can be stood up in an afternoon, and it runs against the model endpoint your company already approved.
