On-Demand Webinar · 32 min

Beyond Data Observability: Simple, Complex, and Personalized Data Journeys

Anomaly detection on data at rest is not enough. Chris Bergh on why errors appear across the toolchain and down the stack, the four Data Journey design patterns a team has to cover, and what a demanding data consumer needs to see about their own data.

Presented by Chris Bergh

What you'll learn 6 points
  • Schema, freshness, volume, and quality anomalies found by periodically polling data at rest do not stop customer-found errors.
  • Data lineage tells you who to contact about an error. It cannot find an error while the data is in use, inside a report or a model.
  • Errors happen across the steps and down the stack — data, tools, code, workflow, servers, network — and the unhappy customer does not care which layer it came from.
  • The rough split presented: 20% data errors, 30% data-and-analytic process errors, 50% basic IT-level monitoring errors. The mix shifts right over time, from raw data toward the uses of data.
  • Four Data Journey design patterns need covering, not one: an out-of-control database team, an end-to-end data product team, hub-and-spoke enterprise enablement, and demanding data consumers.
  • Personalized observability means a consumer follows their own payload — a chemist's molecule, one high-value file, an insurance card — instead of a one-size-fits-all system status.

Prefer to read it? The written version is in The Need For Personalized Data Journeys for Your Data Consumers.

Slides

47 slides

Questions from this session

Why are the five pillars of data observability not enough?

Schema, freshness, volume, and quality checks run by periodically polling data catch anomalies where the data sits, and they still let customers find errors first. Data lineage identifies who to contact once something is wrong, but it cannot actively find an error in data that is already in use, inside a report, a dashboard, or a model.

What is the difference between data in place and data in use?

Data in place is raw and integrated data sitting in files, S3 buckets, database tables, and star schemas. Data in use is that data inside a predictive model, a dashboard, or a delivery and export step such as reverse ETL. Errors look different in each: freshness, volume, row count, and schema failures in place, and empty dashboards, wrong metrics, model prediction errors, and short export row counts in use.

Where do errors actually happen in data analytics production?

In three layers. Data errors, from schema and freshness failures to a business rule catching sales down 50 percent. Data and analytic process errors, such as run start and stop failures, schedule errors, and order-of-operations errors. Basic IT monitoring errors, such as log file errors, CPU and disk metrics, and cost. Errors also shift right over time, starting in raw data and moving toward the uses of data.

What is a data journey?

A data journey follows a specific set of data across every step from source to customer value and down the whole stack: the data itself, the tools acting on it, the code those tools run, the workflow and schedule, and the servers underneath. Observing across and down is what catches errors that a single-tool view misses.

What are the four data journey design patterns?

A monitoring journey for a team that owns a large, constantly changing database it does not fully control. An end-to-end journey for a data product team that needs to stop the game of blame when customers find a problem. Linked hub and spoke journeys for enterprise data enablement and data mesh teams. A payload journey for demanding data consumers who want the status of their own data.

What is personalized data observability?

Personalized observability tracks one consumer's specific data through the production process instead of sending everyone the same system status. A drug discovery chemist can follow their own molecule's data through the warehouse build and see where it is and when it will arrive. A data engineering team can track individual high-value source files from delivery through ingestion.

Where to go next