On-Demand Webinar · 47 min

Data Journey Is the Missing Piece

The modern data stack is complex, the toolchains are fragmented, and the data changes constantly, so you cannot tell whether what is happening right now is what should be happening. Chris Bergh makes the case that the missing piece is the Data Journey: the thing that connects a data system's expectations to its reality.

Presented by Chris Bergh

What you'll learn 7 points
  • The Data Journey Manifesto opens on a single test: at any time, in a data analytic system, know what should be, what is, and the exact difference between the two. Hope it works is not a strategy, and a customer finding a problem in your data analytics is not an acceptable outcome.
  • A Data Journey is the expectation layer over the paths data takes from source to customer value. It observes rather than runs: it tracks components across the toolchain and down the technology stack, taking in logs, messages, status, and data test results, and it does not execute anything itself.
  • Two manifesto principles cut against the usual assumptions. Your data providers will send you bad data, so protect against it rather than trusting it. And perfect data quality at ingestion is not a cure-all, because many things still go wrong after the data is in.
  • Data preparation itself has become a source of chaos: 5 percent of dbt's customer base has more than 5,000 models or tables, and the same data is used many times across many of them.
  • Data Journeys follow Conway's Law, so the organization's design shows up in the technical design. The relationships between journeys, whether hub and spoke, data mesh, producer and consumer, or streaming and batch, usually live as tribal knowledge rather than as anything active or actionable.
  • Tests have a dual nature, and the majority perform double duty: the same test that monitors production catches a regression in development. That is what makes a Data Journey usable for judging the impact of a change before it ships, instead of relying on manual or static analysis.
  • Paraphrasing Anna Karenina, the manifesto's line is that all happy, error-free Data Journeys are alike, and each unhappy Data Journey is broken in its own unique way.

Slides

46 slides

Questions from this session

What is a Data Journey?

A Data Journey represents the expectations on the paths data takes from source to the insight delivered to a customer. It tracks every level of the stack, from data to servers to software to code, covering components across the toolchain and down the technology stack, and it supplies real-time status and alerts. Its purpose is to tell you whether everything ran on time and without errors, and to name the parts that did not.

How is a Data Journey different from a data pipeline?

A pipeline runs the work. A Data Journey observes it and runs nothing itself. It sits over the pipelines, jobs, and schedulers already in place, holds the expected schedule, durations, dependencies, and quality thresholds, and reports the variance between that expectation and what actually happened.

What is the Data Journey Manifesto?

The Data Journey Manifesto is a statement of principles published at datajourneymanifesto.org, written for teams tired of being blamed for data problems they did not cause. Its principles include knowing the exact difference between what should be and what is, making hope infrequent, treating a customer finding a problem as unacceptable, automating all testing, and treating data production as a factory in the tradition of Toyota, Lean, and Deming.

Why is good data quality at ingestion not enough?

Because most of the path comes after ingestion. Even with perfect initial data quality, the data still moves through ETL, databases, models, dashboards, and exports, and each of those can fail on its own terms. Finding the exact source of a problem across raw data, integrated data, models, reports, servers, software, and code is described as half the battle.

What kinds of data tests does a Data Journey need?

Four kinds are named. Drift and consistency tests are generated automatically from profiling baselines. Business rule tests are parameterized fill-in-the-blank checks that carry domain expertise and double as documentation. Custom SQL and Python tests handle logic too complex to parameterize. Custom API and tool tests cover things like Power BI, Databricks notebooks, and REST endpoints.

Why do Data Journeys need historical data?

Because root cause analysis needs the past. Collecting run history longitudinally is what lets a team analyze, learn, and predict rather than react, and each instance of a journey becomes evidence of whether production errors and missed SLAs are actually going down. The journey also becomes the context that gives each individual event its meaning.

Where to go next