Build an automated data factory
Your team will spend months writing data quality tests by hand. TestGen profiles every column and generates the suite in two steps. It runs on every refresh and re-baselines for each medallion layer. Open source. Built the DataOps way.
Key points
- Writing a data quality test suite by hand takes months. TestGen profiles every column and generates the suite in two steps.
- The 1:10:100 rule: an error caught at ingestion costs a tenth of one caught in production, and a hundredth of one your customer finds.
- Coverage means every table and every column, not the handful someone remembered to write tests for.
- Tests that run on every release turn data engineering into a factory instead of a rescue operation.
Writing tests takes too long. TestGen writes them for you
A 100-table Bronze layer needs about 2,200 tests. Silver needs another 2,200. Gold adds 620 plus business-metric tests. That's 14 months of full-time work, with no meetings, no breaks, no vacation. So most teams write three tests on the dashboard table and ship. TestGen profiles each table column by column, generates freshness, volume, schema, and drift tests in minutes, and re-baselines for each layer's schema. A junior operator can run it. The math stops being against you.
Test every layer of your medallion
Bronze is where bad data lands. Silver is where joins go wrong. Gold is where business-logic drift shows up. Each layer fails differently, and tests baselined on Bronze don't catch a Gold problem. TestGen re-baselines its suite at every layer transition. Add a tripwire task between transforms. When a test fails, the next transform doesn't run. Bad data stops at the layer where it broke, not at the dashboard at 9am.
“So much of what we do involves business questions that are fire drills. Executives want answers as quickly as possible. The infrastructure that we've set up with DataKitchen allows us to mix & match data in new ways so that we can quickly get the answer to a question.”
Get coverage on every table, every column
Test coverage in software is mature. In data, most teams have three tests on the dashboard table and call it done. The rule of thumb: every table gets at least two tests for freshness, volume, and schema. Every column gets at least two tests on its data. Every business metric gets a custom test. Every tool in the pipeline gets a status check. That adds up to thousands of tests across a medallion. TestGen generates them and shows you which tables are covered and which aren't. You see the gaps.
The 1:10:100 cost rule
Catching a data error at the source costs about a dollar a record. Catching it after transformation costs about ten. Letting a customer find it costs about a hundred. The numbers come from George Labovitz and Yu Sang Chang in 1992 and they haven't gotten any cheaper. Shift left: run TestGen at the Bronze ingestion layer where the cost is low. Stop fighting fires at the dashboard where it's high and the VP is on the phone.
The DataOps way to data engineering
Tests are infrastructure. So is the team that runs them. The DataOps Cookbook lays out the methodology, free to read online, with chapters on environments, orchestration, testing, and the org practices that make it stick. The DataOps 101 course is free and you can finish it in an afternoon. The Data Observability and Data Quality certification is free too. Read first, run TestGen second. The 2am Slack threads stop.
Stand up open-source TestGen yourself
Profile your own schema, generate the test suite, and see the quality dashboard in 15 minutes. Free, no vendor lock-in.
Frequently Asked Questions
Common questions about DataOps Software for Data Engineers
What is data engineering?
Data engineering is building and operating the systems that move and transform data for analytics: ingestion, storage, transformation, and the tests that prove each step worked. The work is judged on whether the output is trustworthy and on time, which is why testing belongs in the pipeline rather than in a separate quality function.
How long does it take to write data quality tests by hand?
Months, which is the whole problem. Covering 1,000 columns at an average of 2.5 tests each is over 2,500 tests, and at 30 minutes apiece that is roughly 1,250 hours of uninterrupted work. TestGen profiles every column and generates them instead.
How does automatic test generation work?
It profiles each table and column, analyzes 55 column characteristics, and derives validation tests from what it finds: value ranges, formats, uniqueness, referential integrity, distribution shape, and hygiene issues. The tests come from your actual data rather than from a template.
Do generated tests replace hand-written ones?
No, they clear the way for them. Generation covers the mechanical majority so your time goes to the business rules a profiler cannot infer, like a reconciliation between two feeds or a threshold that only makes sense if you know the domain. Custom conditions and full custom SQL are there for exactly that.
Where do the tests run?
As SQL inside your own warehouse. No rows cross the wire to a vendor, and you pay your existing compute rather than a second service reading every row. For regulated environments that distinction usually decides whether an evaluation continues.
Can tests run in CI/CD?
Yes. Tests can gate a pipeline as a tripwire between transforms, and run against a code change as a regression check before merge. The same test does a different job in each place, which is what makes coverage worth more than any individual check.
What does it cost?
The open-source edition is free under Apache 2.0 with no feature gates or usage limits. Enterprise pricing is flat per user per connection rather than per table, so expanding coverage does not raise the bill.