Hiring doesn’t cure firefighting. It adds people to the war room. In one war room DataKitchen has written about, 26 people spent more than eight hours finding the root cause of a blank report.
The fix is to make fewer fires. That takes four steps, and only the last one needs software.
Get your own number first
The survey data is grim. A 2022 Wakefield Research survey sponsored by Monte Carlo, summarized in How Do I Get My Data Team More Efficient?, found data professionals spending 40% of their time checking data quality, with about 13 hours between an incident and its fix. That proves the problem is common. It doesn’t tell you what it costs your team.
So count. Tag every ticket that came from something breaking instead of something planned. Count incidents per week. Then measure detection lag: the time from bad data landing to a person noticing. If the answer is “a day or two, unless it’s the exec dashboard,” you have a detection problem. None of this needs a new process or a timesheet.
Triage every incident the same way
Every incident gets the same four questions, written down in 15 minutes, with no names attached to blame. Here’s a triage record for a hypothetical incident:
| Question | Example answer |
|---|---|
| Who found it? | A finance analyst, Monday 9am |
| Where did it start? | An upstream CRM added a column and changed a type |
| How long was it wrong? | Since Thursday’s load: four days |
| What test would have caught it? | A schema check and a row-count check on the landing table |
The first answer is the one to watch. When a person outside the team finds the problem, your customer is your monitoring system. The ostrich problem post suggests counting how many of last quarter’s incidents someone outside the team found. Few teams like that number.
Hunt the causes that recur
One incident tells you little. The list tells you where to dig. Once a month, read every triage record together and sort by cause. Data Quality Circles applies Pareto here, since about 80% of issues come from 20% of causes, then runs Five Whys on the top few and ends the session with tickets, not observations.
Stopping a cause for good takes three things: a test that would have caught it, an owner for the process that produced it, and a change to whatever kept it hidden. Skip the third and the problem goes quiet without going away. People stop reporting errors when reporting one means owning it.
Put tripwires where the fires start
An alert arrives after the data moved. A tripwire is a test task between two pipeline steps that exits with an error, so the next transform never runs and production keeps serving yesterday’s good data. Put the first ones on the tables your triage records keep naming.
Then decide what each failure does. Halt when the data would be silently wrong, and warn when it would be obviously incomplete. A wrong number gets used. A missing one gets questioned.
Watch the noise. When DataKitchen took over one biotech’s warehouse, it shipped with 150 automated checks and about 20 were still enabled, because the same upstream problems tripped them daily and the team turned off the loudest ones. A tripwire that fires every night trains people to ignore it.
Price it with 1:10:100
The 1:10:100 rule, from George Labovitz and Yu Sang Chang in 1992, says an error caught in development costs about a dollar, in production ten, and in front of a customer a hundred. Most of that hundred is paid in trust. Every cause you remove and every tripwire you add moves the catch from the $100 stage toward the $1 stage.
Tests won’t stop the Salesforce admin from adding a field on a Tuesday. They change when you find out. Monday at 9am from finance becomes Tuesday at 6:14am from a check, and nobody outside the team ever hears about it.