I broke one of our most critical SLAs just last week, and it was the best thing that could have happened.
It was shaping up to be a major embarrassment. One of our key data warehouse refreshes had failed. No new data. No dashboard updates. The refresh was long past its deadline, the project’s key data engineer was on vacation, and I was playing backup. Where was I? At the moment, I was flying home from a data quality conference. This was not good.
Naturally, the airline’s Wi-Fi was terrible, but I managed to get through to a colleague, who knew little about the project. And then we started troubleshooting—one Slack message at a time. I slacked him SQL. He slacked back data. Back and forth we went. An embedded test had failed. Where had our process gone wrong? Everything I could see looked fine. And I was tempted, so tempted, as the clock kept ticking, to disable the test and let it go.
Then it dawned on me that this test wasn’t even ours. It was one of a series of tests that had been added by one of our principal stakeholders in collaboration with his analytics team – esoteric referential integrity checks that had been placed at key points in the build, validating consistency between multiple data sources that fed hourly into our reporting layer.
These tests weren’t easy to define or implement. They came from an understanding of the business rules that we, as data engineers, lacked. And they caught a major problem: the new records we received from one source were completely out of sync with our other data.
Something had gone wrong upstream—very wrong. And because our process was designed to fail fast when integrity was violated, it refused to publish anything at all.
NOTE
From an SLA standpoint, we failed. The data didn’t arrive on time.
But from a trust standpoint? We won.
We trusted stakeholders to define critical business rules that would test for major problems. They trusted us to implement their logic with every refresh. We trusted each other not to blame and shame, but to steadily build out better and better testing to benefit everyone.
Now, together, we had protected leadership from a serious integrity issue. We had prevented a flood of inaccurate reports from reaching decision makers. And when we explained what happened—why the data was late and what we had caught—they weren’t frustrated. They were grateful.
The value of data quality is often invisible. When things go right, no one notices. But this was a moment where things went wrong exactly the way we hoped they would: the system raised a red flag, we caught the issue before it caused damage, and we preserved the integrity of our analytics.
Failing the SLA was the price we paid for trust. And it was worth every second.
The takeaway? If you’re not building comprehensive, automated data quality checks into your production pipelines, you’re one bad refresh away from losing stakeholder confidence and risking devastating business outcomes.
Your data processes shouldn’t just be fast—they should be right. And sometimes, the best outcome is one where the system refuses to run.
FAQ
What are the key points in this blog?
A critical warehouse refresh failed its SLA because an embedded referential integrity test caught new records arriving out of sync with other sources. The test came from a stakeholder’s analytics team and encoded business rules the engineers lacked. The pipeline refused to publish anything, and the stakeholders were grateful rather than frustrated.
Why is failing an SLA sometimes the right outcome?
Because publishing on time and publishing correctly are different obligations. A pipeline designed to fail fast when integrity is violated will miss its deadline rather than release wrong numbers. The cost is a late refresh, and the alternative cost is a flood of inaccurate reports reaching decision makers who then act on them.
Who should write data quality tests?
Not only the data engineers. In this case the tests that mattered came from a principal stakeholder and his analytics team, because they encoded an understanding of the business rules that the engineers did not have. The engineers implemented that logic on every refresh; the stakeholders supplied the rules worth checking.
What does fail fast mean for a data pipeline?
It means the process stops when an integrity check is violated rather than continuing and publishing anyway. In this incident new records from one source were completely out of sync with the other data, and because the build was designed to halt on that condition, nothing inaccurate reached the reporting layer.
How did the stakeholders react to the missed deadline?
They were grateful. Once the team explained why the data was late and what had been caught, the conversation was about a serious integrity issue that had been prevented rather than about a service failure. Trust went up, which is the return on having built those checks in the first place.
