We’ve all been there. You’re up against a deadline, working tirelessly to get the job done. I know how tempting it can be to take shortcuts when you’re under pressure. But trust me when I say that it’s not worth it.

This is the first post in DataKitchen’s four-part series on DataOps Observability. Observability is a methodology for providing visibility of every journey that data takes from source to customer value across every tool, environment, data store, team, and customer so that problems are detected and addressed immediately. DataKitchen has released the first version of its Observability product, which implements the concepts described in this series.

The Head of data got a call from the CEO of the entire company about a compliance report that was empty, with no data. So, he had to rally 26 different people across his team all-day And what was the problem? A field passed through the pipeline that was blank. Can you imagine how embarrassed he is at the error? How frustrated all those 26 people — most likely the best he has on his team — at having to chase a crappy error? And he has 1000 other pipelines in the same ‘hope it works’ position, just waiting for some customer to find a problem. High risk, indeed.