A vendor demo runs on data the vendor chose, with failures the vendor planted. Your trial should do the same thing with your data and your failures. Otherwise two weeks of green checkmarks tells you the tool installs, and nothing else.
Plant failures you know the answer to.
Seed the failures before you score anything
DataKitchen tested its own tool this way. In New in TestGen: Prove Your Data Migration Worked, we planted 785 changes across three public datasets, and the Table Comparison tests caught 774. The post names what the 11 misses were: binary columns, type changes inside the same general type, and values swapped across a composite key. A trial result that names its misses is worth more than one that reports 100%.
Order matters. Generated tests learn the data they’re shown. In the NYC taxi test, TestGen generated 62 tests from a 300,000-row sample and all of them passed. Generated thresholds calibrate to the data in front of them, so the minimum-fare test set its floor at negative $367. So let each tool profile clean data first. Then plant.
| Failure to plant | How to plant it | What should catch it |
|---|---|---|
| Missing rows | Delete a slice of one day’s load | Volume or reconciliation test |
| Duplicated batch | Load the same file twice | Duplicate or row count test |
| Schema change | Rename one column and retype another | Schema monitor |
| Stale table | Skip one scheduled load | Freshness monitor |
| Impossible values | Set a few amounts negative | A business rule |
| Codes off the list | Insert a status code nobody uses | Accepted values test |
The first row is the one health checks miss. In the same migration test, a 55-test health suite passed with zero failures after 483 orders were left behind. The data looked valid. It wasn’t all there.
Two weeks, roughly
The schedule below is DataKitchen’s suggested plan, not a measured benchmark. Stretch it if your loads run weekly.
| Days | Work | What you record |
|---|---|---|
| 1 to 2 | Install, connect, profile the trial tables | Hours by person, permissions you had to request |
| 3 to 4 | Generate or write the baseline tests | Tests created, hours spent writing them |
| 5 | Add two or three business rules with the analyst who owns the report | Whether the analyst could read and correct them |
| 6 to 10 | Run on every load. Plant one failure a day | Catches, misses, time to notice |
| 11 to 14 | Leave it alone | Alerts that needed no action, thresholds you edited |
Day 5 is the cheap test of whether rules will ever get written. Asking a domain owner to author rules from a blank page fails. Showing them a draft to correct takes seconds. A tool where only an engineer can edit a rule fails this day.
Days 11 to 14 measure maintenance. Count every alert that fired with nothing wrong, and every threshold you edited to quiet one. That count is your noise rate. Noise is how test suites die: somebody mutes the alert, then the channel. The false-positive answer covers what to do with it.
Be honest about what two weeks can’t show. The Guide to DataOps Tests says to let a monitor learn a column’s shape over 90 days. A trial can show you how loud a monitor is. It can’t show you whether it learned your month-end close.
A scoring sheet
The weights below are an illustrative template. Set them to your own priorities before day 1, not after you’ve seen the results.
| Criterion | Weight (illustrative) | Evidence from the trial | Tool A (1 to 5) | Tool B (1 to 5) |
|---|---|---|---|---|
| Planted failures caught | 30% | Catches out of planted, misses named | ||
| Setup hours | 15% | Hours log, days 1 to 4 | ||
| Business rules a non-engineer can read | 15% | Day 5 session | ||
| Noise in the quiet week | 15% | Alerts with no action, thresholds edited | ||
| Can it stop a pipeline | 10% | A failing test exits non-zero in your orchestrator | ||
| Price at your table count | 10% | Quote at today’s tables and at double | ||
| Can you run it yourself | 5% | Self-hosted option, license |
The last row looks small until it isn’t. When GX Cloud shut down on June 1, 2026, its customers got about 30 days to migrate.
For the vendor-level questions to ask before a trial, use the best data quality testing tools answer. For the trial itself, the fastest DataKitchen option is open-source TestGen, which covers one user, one connection, and one project at no cost per the pricing page. Measure it with the same sheet you use on everyone else. The 80% post is our claim about how much it generates. Your planted failures are the check on that claim.