On-Demand Webinar · 25 min

Data Observability Demo Day

DataKitchen's DATAVERSITY Demo Day session from November 2023: why complexity in teams, tools, and environments makes Data Journeys unreliable, and how observing errors across the toolchain and down the stack replaces the game of who is to blame.

Presented by Chris Bergh

What you'll learn 6 points
  • Complexity across teams, tools, and environments is what makes Data Journeys unreliable — hundreds or thousands of journeys with no enterprise-wide visibility and no end-to-end quality control.
  • Errors sit in three layers: data errors, data-and-analytic process errors, and basic IT-level monitoring errors. They appear both across the steps and down the stack.
  • Not being able to correlate those errors quickly is what produces finger-pointing, burdensome problem-hunting, lost productivity, and customers who stop trusting the data.
  • Observability here means events from every journey in one place: TestGen data quality test results, infrastructure logs and metrics, order of operations, tool status, and end-to-end SLA.
  • It attaches to an existing toolchain through agents, so the pipelines themselves do not change.
  • The same end-to-end view applies in development, for regression and impact testing, not only in production.

Slides

17 slides

Questions from this session

What is a Data Journey?

A Data Journey is the whole path data takes from source system to delivered customer value, across every data set, tool, server and pipeline it passes through. An enterprise runs hundreds or thousands of them at once. Treating each one as a single object is what makes end-to-end quality control possible, instead of watching each tool separately and hoping the gaps do not matter.

What kinds of errors does data observability have to catch?

Three layers, and they show up both across the pipeline and down the stack. Resulting data errors: freshness, volume, row count and schema problems in raw and integrated data, then model prediction errors, empty dashboards and short export row counts. Data and analytic process errors: run start and stop failures, schedule errors, order of operations errors and usage issues. Basic IT monitoring errors: log file errors, CPU and disk metrics, and cost issues.

Why is it so hard to find the cause of a data problem?

Because the evidence sits in different places. A business rule failure shows in the warehouse, the run that caused it shows in the orchestrator, and the resource problem behind that shows in infrastructure logs. Without something correlating errors across tools and down the stack, teams fall into finger-pointing and burdensome problem-hunting, productivity goes to the search, and customers stop trusting the data.

What is DataOps Observability?

DataOps Observability is mission control for every Data Journey from data source to customer value. It gives end-to-end visibility across all tools, data and infrastructure, monitors and alerts on the complete toolchain against key metrics, and holds dashboards and historical analytics for the whole estate. The same end-to-end view is used in development for regression and impact testing through a CI/CD process, not only in production.

What is DataOps TestGen?

DataOps TestGen generates and runs data quality tests. It scans and profiles a database to identify bad data, then produces tests from what it found, and the session cites 53 unique data test types combining generated checks with fill-in-the-blank business rule tests. Its results feed DataOps Observability as one of the event streams alongside infrastructure logs and metrics.

Does adding data observability mean changing existing pipelines?

No. Observability is attached through agents that read what the existing toolchain already produces, so the pipelines themselves are not modified and any toolchain is supported. That is what gives rapid time to value: the events arriving in one place include data quality test results, infrastructure logs and metrics, order of operations, tool status and end-to-end SLA.

Where to go next