Say a check fails at 6:14am and finance opens the dashboard at 8. You have about two hours to pick one of four moves. If you haven’t decided which, the orchestrator’s default decides for you.
Decide in advance instead, per rule, with three questions.
Three questions, four moves
How bad is a wrong number here? Can you pull it back after someone uses it? And how old can the data be before it’s useless? This is how DataKitchen maps the answers to a move.
| Business impact if wrong | Reversible after use? | Older data acceptable? | Move |
|---|---|---|---|
| High: customers, money, a model, a board deck | No | Yes | Stop. Serve the last good data |
| High | No | No | Stop, and tell the people waiting why it’s late |
| Failure isolated to identifiable rows | Yes, rows can be released later | Yes, for those rows | Quarantine the rows, publish the rest, reconcile the count |
| Cause is timing: late file, lock, timeout | n/a | Yes, for a short wait | Retry, if the load is idempotent. Otherwise stop |
| Low, or the gap is obvious to the reader | Yes | n/a | Publish with a warning to the rule’s owner |
| None. You only want the trend | n/a | n/a | Publish, record the number |
The default when you’re unsure comes from DataKitchen’s efficiency post: halt when the data would be silently wrong, warn when it would be obviously incomplete. A wrong number gets used. A missing one gets questioned.
Stop
A tripwire does the stopping: a test task that fails, so the next step never runs and production keeps serving the last good data. The dashboard answer covers where to put them. The point is reversibility. The Guide to DataOps Tests prefers stopping to alerting because an alert arrives after the data has moved, and a model trained on a column that lost 30 percent of its values can’t be recalled by a notification.
Stopping costs you freshness, and sometimes an SLA. Chip Bloche, DataKitchen’s VP of Data Engineering and Quality Innovation, wrote about breaking a critical SLA when a stakeholder’s referential integrity test refused to publish a refresh with out-of-sync source data. Late, and the data was right. Leadership was grateful once the team explained what the test had caught.
Quarantine
Quarantine works when the bad rows can be named and the rest of the load is fine. The Write-Audit-Publish pattern does it at the batch level: write to a staging location, audit there, and publish only what passes. At the row level, the migration test moved 483 empty orders ($0 subtotal, no line items) to a quarantine table and kept the books balanced: 61,948 source orders equal 61,465 migrated plus 483 quarantined.
The count is what makes it quarantine. Microsoft Fabric’s ON MISMATCH DROP, as DataKitchen’s Microsoft post describes it, deletes the failing rows and writes a number to a log. The gold table looks clean because the evidence left. If nobody reconciles the quarantined count, your totals are silently short, which is the failure you were trying to stop.
Retry
Our rule: retry for timing problems only. A file that hadn’t landed, a lock, a timeout. Bad source values come back identical on the second run.
And retry only an idempotent load. The Seven Deadly Sins post calls out the retry that loads twice: a non-idempotent load turns 1,000 rows into 2,000, and an agent that sees a non-zero exit reruns the command. A FITT pipeline makes rerunning safe by design.
Publish with a warning
The Guide puts most rules that matter here, as soft fails. Its three tiers: hard fails stop the pipeline and should be very few, soft fails let data through and warn the owner, and informational checks record a number for the trend. The starter checks answer assigns a tier to each common check. If a rule warns every night with nothing wrong, it needs fixing, and the false-positive answer covers how to demote it and when to promote one to blocking.
A fifth move exists: fix the data and rerun. Keep a person in it. DataKitchen’s human-gated agent pattern has the agent propose SQL in a ticket, wait for approval, and apply fixes in one transaction to a copy table, so by default the original never changes.