On-Demand Webinar · 1 hr 1 min

How to Build a Successful Cloud DataOps Program

The DevOps and workflow tools a cloud vendor ships do not add up to a DataOps program. Chris Bergh covers the gaps they leave and the capabilities that close them: meta-orchestration of production and development pipelines, testing across the whole analytic system, self-service data environments, and process measurement. Recorded January 2021; updated August 2026.

Presented by Chris Bergh

What you'll learn 6 points
  • The cloud toolchain problem is a grid, not a list. Six stacks, AWS, Azure, GCP, IBM Cloud Pak for Data, on-premise, and open source, each supply an analytic database, blob store, streaming tool, ETL tool, data science tool, BI tool, and governance catalog, plus a git provider, secret store, Kubernetes cluster, and single sign-on provider.
  • Cloud platforms give data integration without process integration. The tools are powerful, but there is no defined process for using them as one system, which is what a DataOps superstructure supplies.
  • Building DataOps capability from individual cloud vendor tools requires intimate knowledge and management of 7 to 10 largely unintegrated transformation, DevOps, version control, and workflow tools. That makes a system-level view nearly impossible, which is why most data analytic teams do not attempt one.
  • Workflow tools such as Airflow, Control-M, and Azure Data Factory orchestrate a single pipeline but lack three things DataOps needs: integrated production testing and monitoring, connectors across the full chain of data engineering, science, analytics, self-service, governance, and database tools, and meta orchestration, meaning a DAG of DAGs.
  • Infrastructure tools such as Puppet, Ansible, and Terraform stand up hardware but not a data sandbox. They lack test data management, Git and GitOps integration, and wizards that let a data person set up, shut down, and monitor a workplace including test data, environments, version control, and toolchains on demand.
  • The recommendation on building your own is to stop. A few unit tests in development, a manually tended Jenkins deploy, and glue code on Airflow, Talend, or Data Factory do not add up to DataOps, and applying software-developer DevOps tools to data science and analytics projects has repeatedly failed.

Slides

36 slides

Questions from this session

Why do workflow tools like Airflow fail at DataOps?

They orchestrate one pipeline in one tool, and DataOps needs orchestration across a system of tools. Airflow, Control-M, and Azure Data Factory have no integrated production testing and monitoring, no connector set covering the full chain of data engineering, science, analytics, self-service, governance, and database tools, and no meta orchestration, the DAG of DAGs that runs the whole toolchain as one test-informed pipeline.

Why isn't DevOps CI/CD tooling enough for data analytics?

Jenkins and Azure Pipelines limit their scope to continuous integration and deployment, and they target software development tools rather than data toolchains. Data work needs two pipelines orchestrated, monitored, and tested at once, development and production. The data equivalent of CI/CD adds continuous self-service sandboxes, continuous meta orchestration, and continuous testing and monitoring alongside continuous integration and deployment.

What is a DataOps superstructure?

It is the layer that turns a collection of cloud data tools into one system. It provides meta orchestration to design and run a system of tools from a single pane of glass, automated testing and alerting in both production and development, sandbox management that automates environment creation and teardown, a common collaboration system across roles and teams, and measurement of quality, deploys, tests, errors, and SLAs.

Why can't infrastructure automation tools handle data environments?

Puppet, Ansible, and Terraform provision hardware and software but stop short of what a data developer needs to start work. They have no test data management, no integration to Git and GitOps, and no simple way for a data person to set up, shut down, and monitor a workplace on demand, including test data, hardware and software environments, version control, and toolchains that are aware of team organization and process measurement.

What should a DataOps program measure?

Two groups. Production metrics cover the pulse of the current production build, data provider errors and successes, test result history, timings, SLAs, and machine learning model metrics. Team and project productivity metrics cover collaborations, deployment frequency, and test coverage. The missing capability in most stacks is collecting these automatically across every analytics and data science system rather than tool by tool.

Should a data team build its own DataOps platform?

The recommendation from this session is no. Too many companies have tried to reach rapid deployment, low errors, end-to-end collaboration, and measurement by assembling their own, and failed. A few unit tests, a manually tended Jenkins deploy, and glue code on Airflow, Talend, or Data Factory do not achieve DataOps, and a patina of software-developer DevOps tooling does not fit the more complex shape of data science and analytics projects.

Where to go next