On-Demand Webinar · 1 hr 11 min

Everything You Need to Know About DataOps Solutions

Wayne Eckerson, President of Eckerson Group, and Chris Bergh walk through Eckerson's DataOps Processes and Technologies framework: the capabilities a DataOps program needs, and how to compare vendors that rebranded into the category. Recorded September 2021; updated August 2026.

Presented by Chris Bergh

What you'll learn 6 points
  • Wayne Eckerson lists ten symptoms that a team needs DataOps: the data team is flooded with request tickets and burning out, business users do not trust the data, source system changes keep breaking ETL jobs, business users are used to debug data quality issues, analysts recreate existing pipelines with minor variations, SLAs are missed, data scientists wait months for data and compute, the cloud migration is stuck, data fires crowd out predictive analytics, and deploying a single predictive model takes months.
  • Eckerson sorts DataOps vendors into four groups: pureplay DataOps, with DataKitchen as the example, pureplay DataOps+ with DataOps Live, data pipeline platforms with Zaloni, and purpose-built tools with Unravel.
  • DataOps tools are a separate market from the roughly 100 billion dollar data tools market. Data tools focus on the data and the insight: ETL such as Informatica, BI such as Tableau, analytic databases such as Snowflake, data science such as DataRobot, and governance such as Collibra. DataOps tools focus on process and workflow and act on those tools. Some data tool vendors use the word for a DataOps halo, which is marketing rather than product.
  • Gartner found in 2020 that only 22 percent of a data team's time goes to innovation and 78 percent to errors and manual execution. A 2021 DataKitchen and data.world survey found 52 percent of data engineers said errors were a major source of burnout.
  • Intel's enterprise analytics engineering manager Gregorio Martinez describes the practice concretely: reusable design patterns so a pipeline can be built quickly and changed reliably, metadata-driven automation so engineers are not adjusting data models every time a schema changes, lean techniques to find and remove bottlenecks, and more than 1,000 tests in the test automation framework.
  • Teams adopt DataOps in four stages, each with a different person in mind. Production DataOps lowers error rates through automated testing and observability, for the production engineer. Development DataOps raises cycle time, productivity and collaboration, for the DataOps engineer and the data engineers and scientists. Measure DataOps adds process measurement, for the data team director. Transform with DataOps addresses the whole organization, for the chief data officer.

Slides

34 slides

Questions from this session

How do you know your organization needs DataOps?

Look for the symptoms rather than the label. The data team is flooded with tickets and burning out, business users do not trust the data or are being used to debug quality issues, source system changes keep breaking pipelines, analysts rebuild pipelines that already exist with small variations, SLAs are missed, and data scientists wait months for data and computing resources. If deploying one predictive model takes months, that is the same problem.

What is the difference between DataOps tools and data tools?

Data tools work on the data itself: ETL, BI, analytic databases, data science platforms and governance catalogs, a market worth roughly 100 billion dollars. DataOps tools work on the process and the workflow around those tools, integrating with them rather than replacing them, and are measured by business outcomes such as lower production error rates, faster and less risky deploys, and team productivity.

What types of DataOps vendors are there?

Eckerson Group classifies them four ways. Pureplay DataOps vendors do DataOps and nothing else, with DataKitchen as the example. Pureplay DataOps+ vendors add adjacent capability, with DataOps Live as the example. Data pipeline platforms such as Zaloni build DataOps into a broader pipeline product. Purpose-built tools such as Unravel address one slice of the problem.

How is DataOps different from DevOps?

DataOps applies the rigor of software engineering to the development and execution of data pipelines, but the pipeline it manages is different. Software CI/CD does not cover self-service sandboxes, meta-orchestration across many tools, or continuous testing and monitoring of data in production. The two also arrived a decade apart: the first DevOps event was in 2009, the DataOps Manifesto was published in 2017, and the first DataOps event was in 2019.

How does Intel practice DataOps?

Intel builds reusable design patterns so a pipeline can be created quickly and changed without losing consistency, and uses metadata to automate pipelines so engineers are not adjusting data models and transforms on every schema change. It applies lean techniques to measure bottlenecks and then remove them, and runs more than 1,000 tests in its test automation framework, adding more continuously.

Where should a team start with DataOps?

Eckerson splits the starting point in two. Organizationally: educate the team, identify bottlenecks, optimize the processes around them, debrief regularly, and monitor the improvement. Technically: componentize your code, build tests for every code block, treat data as code, and invest in the tooling that supports it, meaning a code repository, continuous integration, test management, orchestration and monitoring.

Where to go next