Say a vendor file lands late. The Airflow DAG that loads it waits, then times out on one task and retries. The warehouse table updates two hours after usual. The Power BI refresh ran on schedule, before any of that finished. Four tools, four green-or-yellow statuses, and a VP looking at yesterday’s number.
Every one of those tools did its own monitoring correctly. That’s the problem. The practice that watches all four at once is DataOps Observability, and the definition lives there.
Every tool watches itself
| Tool | What its own monitoring sees | What it can’t see |
|---|---|---|
| Orchestrator (Airflow, Azure Data Factory) | Task status, retries, run duration | Whether the data the task produced is right |
| Warehouse table monitors | Freshness, row counts, schema | Which job wrote the table, or failed to |
| Lineage graph | Which table feeds which | Whether last night’s run happened |
| BI tool (Power BI, Tableau) | Refresh succeeded or failed | Whether it refreshed on stale tables |
| Infrastructure monitoring (Datadog) | CPU, memory, disk, logs | Whether any of it produced the right number |
Taming the Chaos put the human version of this in 2022: each team knows its small part of the system, and nobody holds the larger picture. So when something breaks, the first hour goes to finding out whose problem it is.
Data and the process are different things to watch
Warehouse-focused data observability watches data at rest. It’s good at that. But a table can be perfectly valid and eight hours stale, and the table monitor can tell you it’s stale without telling you why. The why is in a job that ran late, ran twice, or didn’t run. That’s process, and it lives in a different tool. The freshness and schema drift answer covers the other ways a table monitor misses.
DataKitchen’s five pillars of data journeys break end-to-end coverage into five parts:
- Across the steps: runs, order of operations, schedule.
- Down the stack: error messages, server CPU, disk, run cost.
- Data in place: schema, freshness, volume, and business-domain tests on data at rest.
- Data in use: the results in BI tools, models, and exports.
- Set expectations: compare what happened with what should have, and alert.
Most tools sold as data observability cover pillar three. Infrastructure monitoring covers pillar two. Pillars one, four, and five are where the gaps sit, and data in use is the one customers notice first.
A map isn’t an alarm
Lineage looks like it should solve this. It shows every hop from source to dashboard. But DataKitchen’s lineage comparison puts the limit plainly: if your house is on fire, you don’t go to the blueprints, you listen to the smoke alarm. Lineage says where data should flow. It can’t say whether it flowed last night, or on time.
The hard part is plumbing
The concept is easy. The work is getting every tool to report. The DataOps Observability technical overview is direct about it: the product doesn’t instrument tools by itself. An agent, a direct API post, or a script has to send events. It ships agents for common tools such as Airflow, dbt Core, Databricks, Azure Data Factory, Fivetran, and Power BI. Anything else, including the homegrown Python job somebody wrote in 2019, needs an Event Ingestion API call.
Ownership is the second half. Each tool belongs to a different team, and an end-to-end rule (“the sales dashboard has fresh data by 7am”) belongs to none of them until somebody decides it does.
So start narrow. One deliverable, one journey, every tool on it reporting, and a rule for when it should be done. Data Journey First DataOps argues for exactly that order: observe the production journey first, automate later.