The Seven Deadly Sins of AI-Generated Data Pipelines

You shipped four dbt models before lunch. Three of them came out of Claude Code. You read them back, checked the grain, squinted at a join, and merged.

Now count the hours going the other direction. Reading. Checking. Working out whether the load ran twice. Producing code got cheap. Verifying it did not.

The output that hurts you was never the code that threw an exception. It’s the code that ran clean and quietly moved a number.

Seven failure modes, one rule holding them together.

Everybody is using it. Almost nobody trusts it

The 2025 Stack Overflow Developer Survey put 84% of developers on AI tools. Half of professional developers use them daily.

Trust went the other way. Only 29% of respondents say they trust AI output to be accurate, down from 40% the year before. Stack Overflow calls this the AI trust gap, and the number-one frustration behind it is code that is almost right but not quite.

Adoption rose every year. Trust did the opposite.

Almost right is the whole problem in data. Almost right in application code throws a stack trace. Almost right in a pipeline produces a revenue number that is 4% high because the join fanned out on a customer table with duplicate keys. Nothing fails. The dashboard renders. Someone makes a decision.

You already know what you use LLMs for. Window functions you don’t want to write from memory. Legacy stored procedures nobody has touched since 2019. Airflow and Dagster DAG scaffolding. Parsing nested API JSON into something DuckDB will accept. Turning messy OCR text into a Pydantic schema. It works, and it’s fast, and that’s exactly why the failure modes are worth naming.

An engineer on r/dataengineering put it plainly: producing code is now “a very cheap commodity,” and the expensive part is review and testing. GitClear’s code quality research found AI power users hit up to 9x higher code churn, meaning code committed and then reverted or rewritten within two weeks, plus an 8x jump in duplicated blocks. When the pipeline breaks at 3 AM, nobody on call wrote the logic. That failure lands in the same place it always did, inside the four processes every data team runs, except now the code arrived faster than anyone could read it.

Seven ways AI-generated pipelines break. Each one has a mechanism and a fix.

Sins of construction

1. Gluttony, or the pipeline that eats twice

An agent writes an append-heavy load script. The API times out mid-run. The agent retries, because retrying is what agents do, and the target table gets the same rows again.

Four extra words of SQL is the whole difference.

This is what the model hands you, and it looks fine:

INSERT INTO fact_orders
SELECT * FROM stg_orders
WHERE order_date >= CURRENT_DATE - 1;

Run it twice and you have every row twice. Gluttony is consuming the same thing repeatedly, which is also the textbook definition of non-idempotent code. What you want is a load that produces the same table no matter how many times it runs:

DELETE FROM fact_orders
WHERE order_date >= CURRENT_DATE - 1;

INSERT INTO fact_orders
SELECT * FROM stg_orders
WHERE order_date >= CURRENT_DATE - 1;

Idempotency was always good practice. Agents make it mandatory. A human who watches a load fail halfway stops, looks at the target table, and decides what to do. An agent sees a non-zero exit code and runs the command again. Then again. Nobody is watching the row count, and the retry is the whole point of handing the job to an agent in the first place.

The test for whether you have it is one question. Can you run this job a thousand times and get the same table? If the answer is no, or if the answer is “yes as long as it doesn’t fail partway,” you don’t have it.

The pattern that gets you there is FITT: functional, idempotent, tested, two-stage. The I is the letter that matters here. Raw data stays immutable, transformations behave like pure functions, and any job can be rerun without reconstructing what happened last time. Recovery stops being an investigation and becomes a rerun. We wrote up why we won’t build pipelines any other way, and the SQL and orchestration mechanics are in FITT vs. fragile.

2. Pride, or assuming a frontier model knows your schema

The model has your table names. It does not know that status_id = 9 means canceled in the ERP your company bought in 2014. It does not know that ACTIVE_PENDING is what the migration script left behind for records canceled before the migration. It does not know your fiscal year starts in February, or that the natural key on the customer table stopped being unique after the acquisition.

An LLM with database access and no business context is a fast way to generate wrong data. It writes syntactically perfect SQL against semantics it invented.

Every signal the model can see says this column is clean.

The reason this bites so hard is that the semantics were never written down. Your best data quality rules live in someone else’s head, usually the analyst who has been there nine years. A model cannot read that head. It can read column profiles, null rates, cardinality, value distributions, business definitions, and current test results, which is the argument in the equation for AI success. Feed it what correct looks like in your data and it stops guessing.

Sins of testing

3. Greed, or 4,000 new lines behind 40 old tests

You generated a quarter’s worth of SQL in an afternoon. Your test suite is the same size it was Monday.

Greed is hoarding generated code without paying the coverage tax. Ship 10x the SQL against a fixed test suite and you didn’t get 10x productivity. You got 10x technical debt with a shorter fuse.

Both series indexed to Q1, on one scale.

Match generation velocity with test generation velocity. Every model an agent touches gets companion tests, written at the same time, derived from the profile of the actual data rather than from what the transform claims to do. Our definitive guide to test coverage walks through what full coverage means on a real warehouse, and why the count you need runs into the thousands rather than the dozens.

4. Envy, or you bought the graph because everyone had one

Somebody demoed a lineage graph to your steering committee. It glowed. It had 4,000 nodes. You bought it.

Then you added ‘APAC’ to a CASE statement. The graph did not move, because the dependencies did not change. The commission report dropped 30% and you heard about it from sales.

The graph cannot see a change that lives in the values.

Lineage is a blueprint. Tests are the fire alarm panel. Lineage tells you where a fire could spread. Tests tell you which table is burning. Both have a job, and we’ve written about how lineage and data journeys complete each other. But lineage is static analysis, and no engineer ships on static analysis alone.

The fix for the CASE statement is not a bigger graph. It’s a test on the value set: every region in the source appears in the output, and the distinct count of regions matches. Ten seconds to write. It catches the failure the graph cannot see, because the failure lives in the values, not the dependencies.

5. Sloth, or testing is hard so you skipped it

Writing YAML by hand. Mocking test data. Authoring assertions one at a time. Compare that to watching a model produce a 200-line transform in nine seconds.

Sloth here isn’t laziness. It’s friction. Manual test authoring takes three times longer than generating the pipeline did, so it loses every time there’s a deadline.

Under a deadline, that contest has one outcome.

Remove the friction and sloth stops being an option. Profile the database, generate the baseline tests from what the data actually looks like, and spend your time editing the ones you disagree with instead of typing the other 200. Freshness, volume, null rates, distribution shifts, referential gaps, and schema drift are all mechanical. A machine should write them. You should be writing the eight tests that encode business logic nobody can infer from a profile. This is what shifting left and shifting down looks like in practice: catch it where it’s cheap, and automate the part that’s boring.

Sins of the agent loop

6. Vainglory, or letting the agent grade its own homework

You ask Cursor whether the transformation worked. It says yes. You merge.

The same system wrote the transform and the assessment of the transform. That’s a self-report, not a test result. It’s confirmation bias with a token budget.

A self-report is not a test result.

An agent cannot independently evaluate its own logic. It needs an external harness that runs against real rows, in a real database, checking real grain and real constraints. Put that harness at every layer, not just the last one. Test coverage in a medallion architecture covers where the tripwires go across bronze, silver, and gold, and why catching a problem in bronze costs a fraction of catching it in a dashboard.

Independent evaluation also settles arguments. Run one brief through three agent sessions and you get three implementations. Their lineage graphs come back identical. Their test pass rates come back 86%, 94%, and 100%. That number picks the winner and deletes the other two in about four seconds. Nothing else in your stack can do that.

7. Lust, or the lust for speed

Thirty seconds from prompt to production. Past CI, past peer review, past whoever was supposed to check. It feels incredible.

Moving fast without guardrails means you reach the outage sooner. This is the old hero pattern wearing a new hoodie. We’ve argued before that heroic data work is a symptom, not an achievement, and an agent that ships in 30 seconds makes the hero cycle run faster rather than ending it.

Do not slow down. Move the tests inside the loop.

The fix is not slowing down. The fix is moving the tests inside the agent’s loop. An agent that can run real quality tests against real data as part of its own work finds its own failures and fixes them before it opens a pull request. You review a branch that already passed. The Model Context Protocol is what makes this practical: your testing tool becomes something the agent can call, the same way it calls the file system.

Velocity is free. Trust is earned

None of this is an argument for turning off Claude Code. AI is the biggest productivity shift data engineering has seen in a decade, and you’d be foolish to give it up.

The argument is narrower. Code generation speed without automated data quality testing is high-speed technical debt.

When generating code costs nothing, writing code stops being the bottleneck. Verification becomes the bottleneck. If a model writes 4,000 lines of transformation logic in 30 seconds, your testing infrastructure has to profile, test, and score those 4,000 lines in about 15. Otherwise you built a faster path to a production outage.

The golden rule and the four pieces

Four pieces make that work. Architecture first: functional, idempotent designs, so agent retries never duplicate rows or burn compute. Context second: real column profiles, business definitions, and quality metrics fed to the model so it knows what correct means in your data. Coverage third: generated tests that grow at the same rate as your generated code, because hand-written YAML will never keep up. Loops fourth: your testing tools wired into the agent’s workspace over MCP, so the agent catches its own errors before you ever see the branch.

None of the four requires buying anything. They require deciding that verification is engineering work and staffing it that way.

What we built for this

DataKitchen makes the tooling I’ve been describing, and it’s open source.

DataOps Data Quality TestGen is Apache 2.0. Point it at a database and it profiles every column, then writes the tests: integrity, hygiene, freshness, volume, distribution shifts, referential gaps, cardinality surprises, pattern violations. You edit the ones you don’t like. The queries run inside your database, so no data leaves your perimeter and your security review gets shorter. Docker Compose to first quality score takes about 15 minutes. If you want to watch that happen on a public dataset first, here’s three million NYC taxi rides tested in five minutes.

TestGen also ships an MCP server with 96 tools, which is the fourth piece above. Plug it into Claude Code, Claude Desktop, Cursor, or the agent you’re building, and the agent can profile a table, check coverage gaps, run the suite, and read the failures without leaving its loop. The cheat sheet fits on one page. To see it running unattended, read what happened while you slept.

If you’re still deciding whether you need any of this, there are a lot of freaking data quality vendors and we put 55 of them in one table with pricing. If you’d rather have the whole stack built and handed to your team, that’s our AI enablement work.

FAQ