On-Demand Webinar · 52 min

Tame DataOps System Complexity with a DataOps Platform (Demo)

Building a DataOps program out of DevOps and workflow tools leaves a team swimming in tools and APIs. Chris Bergh demos DataKitchen DataOps Automation doing the harmonizing instead: meta-orchestration across tools you already own, automated testing and monitoring across the analytic system, self-service environments for development work, automated deployment, and team performance measurement. Recorded February 2021; updated August 2026.

Presented by Chris Bergh

What you'll learn 7 points
  • The demonstration is organized around four complaints a data team makes: I do not want to learn about data quality issues from my customers, I do not want to break production when I deploy my changes, I do not want my team to struggle working together, and I need to show leadership what my team achieved.
  • An American transportation company streams data from its vehicles in both real time and batch through Nifi, Kafka, an ESB, Informatica, Oracle, Redshift, S3, Python notebooks, and Tableau, with different teams in different locations building and running those pipelines. Too many errors, slow deployment, and poor coordination all trace back to that split.
  • A new feature request is handled by creating a separate work area, a DevSprint Kitchen, that carries the whole toolchain: Pentaho transform, Redshift storage, a Jupyter model, and Tableau reporting. Every test runs there before the change merges back to production.
  • Production testing covers five test types: traditional data quality, statistical process control, location balance, historic balance, and business-based tests. They run automatically, across the whole toolchain, with alerts and retained history.
  • Process measurement covers two areas. Production metrics track the current build, data provider errors, test result history, timings, SLAs, and model metrics. Team and project metrics track collaboration, deployment frequency, and test coverage.
  • One orchestration coordinates a Boston home office team on a weekly cadence of changes and a New Jersey self-service team on a daily or hourly cadence, spanning SQL, Python, SSIS, SQL Server, Alteryx, and Tableau.
  • Deming's finding that 94 percent of causes are common cause, and Elon Musk's line about building the machine that makes the machine, are the argument for treating the process as the thing to fix.

Slides

31 slides

Questions from this session

What does a DataOps platform do?

It handles the four processes that surround the analytics work itself: testing and monitoring production across the whole toolchain, deploying changes from development into production without breaking anything, giving multiple teams a shared place to work, and measuring the process so results can be shown to leadership. The tools that transform the data stay in place.

What is a Kitchen in the DataKitchen platform?

A Kitchen is a place to work that carries a whole toolchain, not a single tool. In the demonstration a new customer segmentation request needs a data scientist, Jupyter, a database change through Pentaho, and a new Tableau visualization, so a DevSprint Kitchen is created with all of it. Every test runs there before the work merges back into the production Kitchen.

How do you deploy a change without breaking production?

Do the work in a separate environment that mirrors the production toolchain, then run the full set of tests there before promoting or merging. The tests are the gate rather than a review meeting, which is what moves deployment latency from weeks or months down to hours or minutes.

What should a data team measure?

Two areas. Production metrics: the pulse of the current build, data provider error and success rates, test result history, timings, SLA performance, and machine learning model metrics. Team and project productivity metrics: collaboration, deployment frequency between environments, and test coverage. Analytic teams are rarely analytic about their own internal work.

How does a DataOps platform help teams in different locations work together?

It gives them one orchestration over the tools they each already use. A home office team on a weekly cadence of changes and a local self-service team on a daily or hourly cadence can run SQL, Python, SSIS, SQL Server, Alteryx, and Tableau steps as one process, so neither cadence has to slow down to match the other.

What does "what you do is much less important than how you do it" mean?

What you do is the model, the algorithm, the pipeline, the visualization, the governance, and the data. How you do it is development, deployment, monitoring, iteration, collaboration, and process measurement. Deming found that 94 percent of causes are common cause, which is the case for improving the process rather than looking for a person to blame.

Where to go next