On-Demand Webinar · 53 min

Map and Monitor Your Data Journey

Can you draw a map of every path data takes from source system to production insight? Chris Bergh breaks down a Data Journey, and what it takes to monitor, track, and test the run-time lineage once you have mapped it.

Presented by Chris Bergh

What you'll learn 6 points
  • A data journey is every path data takes from source to the insight delivered to a customer, tracked across all levels of the stack: data, tools, code, and tests. It supplies real-time status and alerts, so a team knows whether everything ran on time and without errors and which specific part did not.
  • At a company in the top 20 by revenue in the United States, the head of data got a call from the CEO about a compliance report that came out empty. He pulled 26 people off their work for a full day, and the cause was one blank field passed through the pipeline. The deploy cycle was six weeks, and a thousand other data journeys sat in the same hope-it-works position.
  • The numbers on data analytic projects: Gartner says 60 percent fail altogether and 87 percent of data science projects never reach production, Eckerson says 79 percent have too many errors, and DataKitchen found 78 percent of data engineers are stressed enough to need a therapist.
  • The model for managing this risk is mission control. NASA and SpaceX run the biggest journey of all by building an interface carrying information about every aspect of the flight, using it to make decisions and communicate to interested parties, storing it for after-the-fact analysis, and alerting automatically.
  • IT hardware monitoring and application performance monitoring are lagging indicators of data problems. APM tools do not check the data, the integrated data, or the reports and models built from it, and they supply no context for the pipelines, jobs, and tools acting on the data.
  • Production data tests fall into five categories: raw data profile qualification and consistency tests, statistical process control, location balance, historical or population balance, and time series anomaly tests. Most tests do double duty as development tests and as production monitoring.

Slides

51 slides

Questions from this session

What is a data journey?

A data journey is the full path data takes from source to the insight a customer receives, including every tool, data set, method, and person along the way. It tracks all levels of the stack, from the data itself to the tools, the code, and the tests, and it supplies real-time status and alerts so a team can tell what ran on time, what failed, and where. Most organizations hold this map as tribal knowledge rather than as something active and actionable.

Why is application performance monitoring not enough for data pipelines?

Disk and CPU metrics are lagging indicators of data problems: by the time infrastructure looks wrong, the bad report has already shipped. APM tools do not check the raw data, the integrated data, or the reports and models created from that data, and they do not give a context that shows the pipelines, jobs, and tools acting on it. They also cannot synthesize production runs and development work into one coherent picture.

What kinds of tests belong in a production data pipeline?

Five categories: raw data profile qualification and consistency tests, statistical process control, location balance, historical or population balance, and time series anomaly tests. Testing happens in four places, on raw data, on integrated data, on tool-modified data, and on tool processing status and timeliness, and it runs both in production against live data and in development against test data. Results are graded as log, warning, or error, and most tests do double duty across development and production.

What questions should a data team be able to answer about its pipelines?

Whether the job finished successfully, whether the dashboard or data set is correct and refreshed with the latest data, whether source files arrived on time and at the right quality, what resources the process consumed, and whether job X ran only after every job in group Y completed. Also how many jobs ran yesterday and how long they took, and which pipeline is troublesome with frequent or intermittent errors. Teams that cannot map the paths data takes cannot answer any of these.

How does DataOps observability support data governance?

A governance program answers what the data is, through a catalog of metadata plus management and search tools, and where it came from, through lineage covering origin, changes, and movement over time. Observability adds the two questions a catalog cannot answer: can I trust the data, answered by test results recording what tests ran against the data and artifacts and what they returned, and is the data fresh, answered by run-time lineage recording when the source and integrated data were last updated.

What are the two steps to adopting DataOps?

Observability first, automation second. Observability reduces risk by finding problems and bottlenecks across the toolchain and giving a team enterprise-wide views of hundreds or thousands of data journeys with monitoring and alerting on both tools and data. Automation follows, adding testing and orchestration that fix the problems so they never happen again, which is where cycle time, error-free days, and team productivity improve.

Where to go next