The hard part is the first check, not the hundredth. When we read 849 comments on why data engineers don’t test, one commenter, kenfar, gave the order we’d give: “I’d start with very obvious and simple items: checks for uniqueness, foreign keys, nulls, appropriate dates. Once that’s in place I’d add reconciliation and anomaly detection checks.”
Here’s that order made concrete for a small team.
Twenty starter checks
Apply these to the handful of tables that matter most. The names in the second column are the DataOps TestGen test types that do each job; any tool with the same checks works. Severity follows the three tiers in Your Best Data Quality Rules Live in Someone Else’s Head: hard fail, soft fail, informational.
| # | Check | TestGen test type | Scope | Kind | Severity | Owner |
|---|---|---|---|---|---|---|
| 1 | Table updated on schedule | Table Freshness, or the freshness monitor | Table | General | Hard | Pipeline engineer |
| 2 | Row count in its normal range | Volume monitor or Row Range | Table | General | Soft | Pipeline engineer |
| 3 | Columns added, dropped, or retyped | Schema monitor | Table | General | Soft | Pipeline engineer |
| 4 | Row count never shrinks (append-only tables) | Row Count | Table | General | Soft | Pipeline engineer |
| 5 | No duplicate rows | Duplicate Rows | Table | General | Hard | Pipeline engineer |
| 6 | Primary key is never null | Required Entry | Column | General | Hard | Pipeline engineer |
| 7 | Primary key is never repeated | Unique Values | Column | General | Hard | Pipeline engineer |
| 8 | Foreign keys exist in the dimension | Reference Match | Column | General | Soft | Pipeline engineer |
| 9 | Required fields have no placeholders like “N/A” | Required, No Placeholder | Column | General | Soft | Data steward |
| 10 | Share of missing values hasn’t jumped | Percent Missing | Column | General | Soft | Data steward |
| 11 | Status and category codes are on the list | Value Match | Column | General | Soft | Data steward |
| 12 | Event dates aren’t in the future | Past Dates | Column | General | Soft | Pipeline engineer |
| 13 | Newest record is recent | Recency | Column | General | Soft | Pipeline engineer |
| 14 | Amounts stay inside their observed range | Minimum Value, Maximum Value | Column | General | Soft | Analyst |
| 15 | Identifiers keep their format | Pattern Match | Column | General | Informational | Data steward |
| 16 | Email and ZIP fields are valid | Email Format, Valid US Zip | Column | General | Informational | Data steward |
| 17 | Amounts are never negative | Custom Condition | Column | Business | Hard | Analyst |
| 18 | One rule per key metric, such as shipped ≤ ordered | Custom Condition or Custom Test | Table | Business | Soft | Analyst |
| 19 | Totals by group match the source or last delivery | Aggregate Balance | Table | Business | Soft | Analyst |
| 20 | The headline KPI hasn’t moved more than history allows | Metric Trend | Table | Business | Soft | Metric owner |
Rows 1 through 16 need no business knowledge, and most of them come out of a data profile without anyone typing them. That’s deliberate. The Guide to DataOps Tests puts the standard at two to three tests per table, at least two per column, and one custom test per significant business metric. Rows 17 through 20 are the part a profiler can’t do.
Why row 17 exists when row 14 already checks amounts
Generated range checks learn the data they’re shown. In DataKitchen’s NYC taxi exercise, the generated minimum-value check on fare_amount set its floor at negative $367, because that was the lowest fare in the 300,000-row sample. On the sample it passed, and the business rule “fares are never negative” failed. Against the full month of 2,964,624 rows the generated check failed too, on 22 fares below the floor, because new data didn’t look like old data. Row 14 tells you something changed. Row 17 tells you something is wrong.
Keep the hard fails rare
A hard fail stops the load and pages somebody, so reserve it for failures that would publish something wrong: no load, null keys, duplicate keys, negative money. Make the rest soft or informational. When a check fires every night and nobody acts, demote it rather than muting it. A muted check still looks like coverage.
One owner per check
The owner is whoever can act on a failure. That is usually not the person who wrote the test. Pipeline checks go to whoever is on call. Source-data checks go to whoever can persuade the source owner to fix it. Business rules go to the analyst or steward who knows what “correct” means, and the fastest way to get those rules is to show that person a draft and let them correct it.
Twenty checks on your critical tables won’t make you covered. They will make your next incident the first one you hear about from a test instead of from a VP.