I broke one of our most critical SLAs just last week, and it was the best thing that could have happened.
It was shaping up to be a major embarrassment. One of our key data warehouse refreshes had failed. No new data. No dashboard updates. The refresh was long past its deadline, the project’s key data engineer was on vacation, and I was playing backup. Where was I? At the moment, I was flying home from a data quality conference. This was not good.
Naturally, the airline’s Wi-Fi was terrible, but I managed to get through to a colleague, who knew little about the project. And then we started troubleshooting—one Slack message at a time. I slacked him SQL. He slacked back data. Back and forth we went. An embedded test had failed. Where had our process gone wrong? Everything I could see looked fine. And I was tempted, so tempted, as the clock kept ticking, to disable the test and let it go.
Then it dawned on me that this test wasn’t even ours. It was one of a series of tests that had been added by one of our principal stakeholders in collaboration with his analytics team – esoteric referential integrity checks that had been placed at key points in the build, validating consistency between multiple data sources that fed hourly into our reporting layer.
These tests weren’t easy to define or implement. They came from an understanding of the business rules that we, as data engineers, lacked. And they caught a major problem: the new records we received from one source were completely out of sync with our other data.
Something had gone wrong upstream—very wrong. And because our process was designed to fail fast when integrity was violated, it refused to publish anything at all.
From an SLA standpoint, we failed. The data didn’t arrive on time.
But from a trust standpoint? We won.
We trusted stakeholders to define critical business rules that would test for major problems. They trusted us to implement their logic with every refresh. We trusted each other not to blame and shame, but to steadily build out better and better testing to benefit everyone.
Now, together, we had protected leadership from a serious integrity issue. We had prevented a flood of inaccurate reports from reaching decision makers. And when we explained what happened—why the data was late and what we had caught—they weren’t frustrated. They were grateful.
The value of data quality is often invisible. When things go right, no one notices. But this was a moment where things went wrong exactly the way we hoped they would: the system raised a red flag, we caught the issue before it caused damage, and we preserved the integrity of our analytics.
Failing the SLA was the price we paid for trust. And it was worth every second.
The takeaway? If you’re not building comprehensive, automated data quality checks into your production pipelines, you’re one bad refresh away from losing stakeholder confidence and risking devastating business outcomes.
Your data processes shouldn’t just be fast—they should be right. And sometimes, the best outcome is one where the system refuses to run.
