An AI writes a different kind of test than a profiler does. You want both.
What happened when an AI wrote the rules
In August 2026, DataKitchen pointed Claude Code at the January 2024 NYC yellow taxi trips through TestGen’s MCP server. TestGen profiled the data and generated 62 tests from a 300,000-row sample. All 62 passed. The generated minimum-value test for fare_amount set its floor at negative $367, because that was the lowest fare in the sample, so negative fares looked normal.
Then the model read the profile and wrote seven business rules in SQL, nobody dictating them. All seven failed on the sample. On the full 2,964,624 rows, the run had 69 tests, 11 failures, and 2 warnings:
| Test | Written by | Sample (300,000 rows) | Full month (2,964,624 rows) |
|---|---|---|---|
| 62 generated tests | TestGen, from the profile | All passed | 4 minimum-value tests failed |
| Passengers at least 1 | AI | 17,477 failing rows | 171,627 failing rows |
| Payment code in the documented range | AI | 14,252 failing rows | 140,162 failing rows |
| Fare not negative, plus 4 more rules | AI | All failed | All failed, including 37,448 negative fares |
The four generated failures on the full month were real too: records more extreme than anything the sample held. That’s the generated tests doing their job, which is catching change. The AI rules catch what’s wrong today.
The pattern held on regulated data
In September 2026, the same setup ran against 15.5 million CMS Open Payments records. TestGen generated 353 tests on the Massachusetts rows and 349 passed, including on payment dates in the year 2. Sixteen SQL tests, written in a Claude Code session from the federal reporting rule, the 2020 OIG speaker-program alert, the industry codes, and the Massachusetts gift ban, flagged 92 entertainment payments and 91 meals over $500 per person in Massachusetts alone.
The useful detail came next. Moving the 353 generated tests to the national table broke 33 of them, because their thresholds came from the Massachusetts sample. The 16 rule tests didn’t break. Their thresholds came from the law.
Where AI-written tests stop
A model knows what a taxi is because taxis are public knowledge. It doesn’t know what “active” means in your billing system, or which day your finance team closes the month. Those rules live in someone else’s head. The model is still useful there, as a drafter. Show an analyst a rule to correct and she fixes it in seconds. Send her a blank template and you get four rows back, two of which say the data should be accurate.
And never let the agent grade its own homework. That’s the rule behind all seven sins of AI-generated pipelines. The model writes the tests and reads the results. A deterministic tool runs them against real rows and returns the same answer on the same data every time. When an LLM gets root cause wrong, the most common failure is misread evidence, and deterministic results are the correction.
How to try it this afternoon
Install open-source TestGen, turn on its MCP server, and connect the AI client you already use. Ask for a profile and a generated suite first. Then ask the model to read the profile and write tests for the things that can’t be true in your data. Check its rules against the raw table before you trust a single one.