On-Demand Webinar · 40 min

Data Products and Data Mesh

Chris Bergh on why complexity kills data team productivity, how a data mesh cuts a team's world down to domains it can own, and why a data product is a flow of value rather than a finished project plan.

Presented by Chris Bergh

What you'll learn 6 points
  • Complexity is the thing being fought: many teams and roles, thousands of datasets of different types and shapes, interconnected project plans. Over time it kills productivity.
  • A data mesh cuts the work into decentralised domains, so a team stops boiling the ocean and works on fewer datasets and fewer customers.
  • A domain is more than data. It holds the raw and integrated data, the artifacts built from it, the code that acts on it, the team responsible, and its metadata.
  • Five interfaces make a domain usable by other people: the width (data), the where (location), the what (description), the how (process), and the who (team).
  • Domains compose — one domain's output is another's input — which is what lets a team control its own world and still build on everyone else's work.
  • A data product is judged on value delivered and whether the customer is happy, not on tasks closed against a project plan. Mesh and products are not alternatives; they work together.

Prefer to read it? The written version is in Webinar Summary: Data Mesh and Data Products.

Slides

47 slides

Questions from this session

What is a data mesh?

A data mesh organizes a data team's work into decentralized domains rather than one central platform. Instead of boiling the data ocean, each team focuses on fewer data sets and fewer customers and delivers more insight. In practice it is an organization mesh: full end-to-end ownership of a domain is assigned to one cross-functional team that gets the support it needs, and that team keeps owning the data set rather than rotating from project to project.

What is a data product?

A data product treats all data work as an ongoing, ever-improving flow of value to a customer. Success is not defined by completing the task on a project plan; it is defined by actual usage and by the expanding delivery of insight. Because data products are never done, they are measured on flow rather than milestones: cycle time of deployment, or flow time; success rate of deployment, or flow efficiency; and work done, or flow velocity.

What are the components of a data domain?

Five things: the data itself; the artifacts created from that data such as models, views, reports, and dashboards; the code that acts on the data, including pipelines and toolchains; the team that creates, updates, and runs the domain; and the metadata, meaning catalogs, lineage, test results, and processing history. A domain should be trusted, usable by its customers, discoverable, well-described, secure, and interoperable through URLs and APIs, with one accountable owner a customer can go to for fixes and new requests.

How do domains in a data mesh communicate with each other?

Through five link types. A domain query asks when a domain last updated and whether it succeeded, or asks it to prove data quality with test results. A process linkage hands control and parameters from one domain to the next, an event linkage announces completions, errors, and warnings, and a data linkage covers a shared table such as a common dimension. A development linkage is the ability to re-create another domain in development, read and change its code, and get it to production.

Why does a data mesh need observability?

Each domain changes independently, so something has to confirm the whole composed system is still correct. Every domain is a data journey that tracks data, tools, code, and tests, supplying real-time status and alerts on start times, processing durations, test results, and infrastructure events. The first step in taming complexity is monitoring every tool, team, environment, and customer so problems are detected, localized, and understood immediately.

How do data mesh and data products relate to software engineering practices?

Both are data analytics translations of answers software engineering already found for complexity. DevOps becomes DataOps, domain-driven design becomes data mesh, and software products become data products. The same mapping continues: test-driven development becomes data testing, observability becomes data observability, event-driven architecture becomes streaming and Kappa architecture, APIs become data contracts, and functional programming becomes functional data engineering.

Where to go next