On-Demand Webinar · 1 hr 0 min

DataOps: The Foundation for Your Agile Data Architecture

Chris Bergh walks through four data architecture patterns that promise agility: DataOps, Data Fabric, Data Mesh, and Functional Data Engineering. He covers where each one fits, how they work together, and what a DataOps platform underneath them is actually for. Recorded July 2021; updated August 2026.

Presented by Chris Bergh

What you'll learn 7 points
  • Four architecture patterns are put forward as the routes to agility: DataOps, data fabric, data mesh, and functional data engineering. The session's argument is that they combine in one system rather than competing.
  • Gartner describes a data fabric as a design concept rather than a set of technology components, focused on composability. The caveat given here is that agility is a second-order effect of better tools: the primary driver is people and process following DataOps.
  • The 'AI inside' claims made for data fabric products are compared to levels of autonomous driving. Level 1 keeps your hands on the wheel, level 5 crosses Boston in the snow at night, and AI in the data fabric is at level 1.
  • Data mesh is treated as an organization mesh: full end-to-end ownership of a domain assigned to one cross-functional team of five to nine people, organized around the domain instead of the technology, with domain data treated as a product and other teams as its customers.
  • A data domain has five components: the data, the artifacts built from it such as models, views, reports and dashboards, the code that acts on it such as pipelines and toolchains, the team that creates, updates and runs it, and the metadata including catalogs, lineage, test results and processing history.
  • Functional data engineering means immutable data, pure functions, idempotency, and no side effects. Re-running a task for the same date always produces the same output, so broken data is repaired by rerunning the new code.
  • The commercial pharma example built on all four patterns supports over $10 billion in sales, integrates hundreds of data sets, runs more than 50,000 automated tests, absorbs more than 100 schema and data changes per week, and is staffed by seven data and DataOps engineers.

Slides

57 slides

Questions from this session

What are the four data architecture patterns for agility?

DataOps, data fabric, data mesh, and functional data engineering. DataOps covers the people and process side: development, deployment, monitoring, iteration, collaboration and measurement. Data fabric covers centralized data infrastructure plus virtualization. Data mesh divides ownership into domains held by cross-functional teams. Functional data engineering makes processing immutable, idempotent and reproducible. One system can use all four.

What is a data fabric?

A data fabric is a design concept for composable data infrastructure, not a product category. Gartner's framing is that it lets users build a flexible, scalable architecture that supplies data to human or machine users. In practice it bundles the familiar centralized pieces, ETL, databases, governance, lake, warehouse and stream or batch transformation, plus data virtualization and a semantic layer.

What is in a data mesh domain?

Five things: the data itself, the artifacts built from it such as models, views, reports and dashboards, the code that acts on the data such as pipelines and toolchains, the team that creates, updates and runs the domain, and the metadata including catalogs, lineage, test results and processing history. A domain that hides any of these cannot be composed with other domains.

What is functional data engineering?

Functional data engineering applies functional programming rules to data pipelines: start with immutable data, use pure functions that take data in and give data out, make processing idempotent so a rerun produces the same result, and allow no side effects. The payoffs are reproducibility, which matters legally as well as scientifically, lower complexity, and faster time to value.

How do domains in a data mesh talk to each other?

Through six kinds of link. Domain queries ask when a domain last updated and whether its data passed its tests. Process linkage passes control, and parameters, from one domain to the next. Event linkage announces completion, errors or warnings. Data linkage means a shared table such as a common dimension. Development linkage asks whether a domain can be recreated and modified in development with a path back to production.

What results did the commercial pharma example produce?

The system supports over $10 billion in sales with hundreds of integrated data sets, more than 50,000 automated tests, and more than 100 schema and data changes per week, with very few errors or missed SLAs. It runs with a staff of seven data and DataOps engineers, which is what keeps total yearly hardware, hosting, software and staffing cost low.

Where to go next