On-Demand Webinar · 58 min

Orchestrate Your Development Pipelines for Fast and Fearless Deployment

Part two of DataKitchen's series on orchestrating the three pipelines of DataOps. Chris Bergh on why orchestrating the production pipeline is not enough, and how orchestrating the development pipeline, the one that carries new analytics, lets a team extend and deploy work quickly and without fear. Recorded May 2020; updated August 2026.

Presented by Chris Bergh

What you'll learn 7 points
  • A large European telecom ran a four-stage manual deployment from development to production across data science, visualization, and governance tools. Deploying a change took four months.
  • DataOps runs three orchestrations, not one: the value pipeline in production, the innovation pipeline that moves changes toward production, and an environment pipeline underneath both that creates and manages the environments they run in.
  • Continuous integration and unit tests are not enough for data analytics. Five things have to be added: sandbox management, orchestration of data pipelines and containers with parameterization, a test framework embedded in the pipeline, operational data quality monitoring and alerting, and many-to-many collaboration between development and operations.
  • Most tests do double duty. A test written against fixed data and changing code is a regression test in the innovation pipeline; the same test against fixed code and changing data is a production monitor in the value pipeline.
  • An analytics change needs the full range of test types before it ships: unit, functional, regression, performance, smoke, and end-to-end tests.
  • In a stateful architecture each environment holds state that has to be patched as changes deploy. A functional architecture built on immutable data and pure functions over that data removes that per-environment patching work.
  • Elon Musk's line that the real problem is building the machine that makes the machine, and Deming's finding that 94 percent of causes are common cause, are the argument for fixing the process rather than the individual deliverable.

Slides

65 slides

Questions from this session

What are the three DataOps pipelines?

The value pipeline is production: data flowing through the toolchain to customers. The innovation pipeline is deployment: new code, models, schemas, and visualizations moving from development into production. Underneath both sits the environment pipeline, which creates and manages the development, test, and production environments the other two run in.

Why is CI/CD with unit tests not enough for data analytics?

A DevOps process runs develop, build, test, deploy, and run. A DataOps process adds sandbox management so a developer can get an environment with the right data, software, and libraries; orchestration of data pipelines and containers; a test framework embedded in the pipeline itself; operational data quality monitoring and alerting; and collaboration across many development and operations teams.

What test types should run before deploying a data analytics change?

Six. Unit tests aimed at each component on its own. Functional tests verifying against a spec or user story. Regression tests rerun on every change to prove the rest still works. Performance tests for responsiveness and stability under load. Smoke tests as a quick check that major functions are operational. End-to-end tests proving the pipeline works as a whole.

Why does it take so long to deploy a change to a data pipeline?

The development lifecycle spans four environments, each with different people, tools, code versions, hardware, operating systems, libraries, and data. That gap is usually bridged with manual operations and scripts rather than automation. One large European telecom's four-stage manual deployment took four months from development to production.

What principles make analytic deployment fast and low risk?

Create a repeatable, reliable release process. Automate everything and keep everything under version control. If it hurts, do it more often, so the pain comes forward. Build quality in through automated testing. Done means released to production. Everybody is responsible for the release process. Measure cycle time from deploy to production and improve it.

What is a functional data architecture?

A functional data architecture is built on immutable data and pure functions over that data to produce results. It contrasts with a stateful architecture, where development, system test, pre-production, and production each hold their own state that must be updated or patched during a deployment. Removing that state removes a large source of deployment error and manual work.

Where to go next