On-Demand Webinar · 51 min

How DataOps Enables a Data Fabric

A Data Fabric promises to simplify data management across multi-cloud and on-premise environments. Chris Bergh works through where the analytics agility actually comes from, which comes first (the fabric or DataOps), and how a DataOps platform supports the initiative. Recorded April 2021; updated August 2026.

Presented by Chris Bergh

What you'll learn 6 points
  • Gartner describes data fabric as a design concept rather than a set of technology components, focused on composability so users can build a flexible, agile, scalable architecture that supplies data to humans or machines.
  • A data fabric is mostly the centralized data infrastructure a company already runs: ETL, databases, governance, storage, lake, warehouse, and stream or batch transformation. What is new is an AI component and a data virtualization or semantic layer.
  • The goal of a data fabric is agility, but agility is a second-order effect of better tools. The primary driver is people and process following DataOps.
  • AI inside a data fabric maps to Level 1 of autonomous driving, hands still on the wheel, not Level 5 crossing Boston in the snow at night.
  • The data fabric stops short of the end of the value chain. It covers store, transform, virtualize, and govern, but leaves out models, visualizations, reports, and self-service, so it is more hub than spoke.
  • A canonical data architecture designs only for production and not for the process of changing production, which is a little like designing a mobile phone with a fixed battery. The result is unplanned work, manual deployment, errors, and bureaucracy.

Prefer to read it? The written version is in DataOps Enables Your Data Fabric.

Slides

57 slides

Questions from this session

What is a data fabric?

Gartner frames data fabric as a design concept rather than a set of technology components, built on composability so a flexible, agile, scalable architecture can supply data to human or machine users. In practice a data fabric is most of the centralized data infrastructure an organization already has, meaning ETL, databases, governance, storage, lake, warehouse, and stream or batch transformation, plus an AI component and a data virtualization or semantic layer. Typical toolchain elements include catalogs such as Alation, virtualization such as Denodo or Delphix, pipeline tools such as Informatica, Talend, or Airflow, and databases such as Redshift, S3, or SQL Server.

What is a DataOps process fabric?

A process fabric is a single user experience spanning the whole DataOps process that stitches multiple separate tools into one system, connecting people, processes, and tools rather than only data. It supplies meta orchestration across tools from one pane of glass, automated testing and alerting in both development and production, automated sandbox creation and management, a shared collaboration system across roles and teams, and measurement of delivery speed and quality. It spans environments and data centers, so a production environment in one cloud and a development environment in another sit under the same platform.

Why doesn't a data fabric deliver agility on its own?

Agility is a second-order effect of better tools; the first-order driver is people and process following DataOps. The tools in a data fabric lack the team and environment awareness needed to promote reuse, and they lack a common collaboration system for multiple teams using disparate technologies. The AI marketed as part of a fabric sits at Level 1 on the autonomous-driving scale, keeping hands on the wheel, not the self-driving data the label implies.

What are location balance and historical balance tests?

Location balance tests confirm that data properties match business logic at each stage of processing, so the same measure holds as data moves from store to transform to model to report. Historical balance tests compare current data to previous or expected values, using history as the reference for whether today's values are reasonable. Both are forms of statistical process control: check upper and lower bounds over time, watch for a trend break, and alert.

How should test results be graded in a data pipeline?

Three severity levels. An error stops the line. A warning is something to investigate later. An info result is a list of changes to be aware of. Tests belong inside the pipeline at every step rather than bolted on the side, and they should answer three questions: are the data inputs free from issues, is the business logic still correct, and are the outputs consistent.

What is C/SMOIDTM, and why isn't DevOps CI/CD enough for data?

C/SMOIDTM is the data analytics counterpart to DevOps CI/CD, and the presenter concedes it is the world's worst acronym. It stands for continuous self-service sandboxes, continuous meta orchestration, continuous integration and deployment, and continuous testing and monitoring. DevOps tools such as Jenkins and Azure Pipelines limit their scope to CI/CD and target software development toolchains, so they cannot orchestrate, monitor, and test the two pipelines data work requires: development and production.

Where to go next