Max Kurton, Editor-in-Chief at Enterprise Management 360, interviewed DataKitchen founder and CEO Chris Bergh about the what, why, and how of DataOps. EM360 published the episode in August 2020 and still hosts the audio; this post went up here that October. What follows is a guide to the ground the conversation covered, and to the position DataKitchen was publishing on each of those questions at the time.
Listen to the episode
The recording lives on EM360’s site: DataKitchen: The What, Why, and How of DataOps. It is their show and their recording, which is why there is no player on this page.
What the conversation covered
Five threads, in the order EM360 lists them:
- What DataOps is, and how it differs from DevOps
- Why a company would want it
- How to build a DataOps team
- What principles DataOps leadership should push for
- What cultural preparation it takes, and why more companies had not tried it
NOTE
The sections below are DataKitchen’s published positions on those five questions in 2020, each linked to the post or paper that carries the argument in full. They are a companion to the recording, not a transcript of it.
DataOps is not DevOps for data
The first distinction, then and now: DataOps borrows from DevOps and then has to handle something DevOps never faced. The data changes independently of the code. A deployment pipeline that works perfectly still ships wrong numbers when an upstream source shifts, so DataOps adds statistical process control from lean manufacturing to the Agile and DevOps inheritance. The full definition and the intellectual heritage sit on their own page.
That difference shows up concretely in what a deployment has to carry. Moving analytics from development into production takes four things, not one: the data you are testing with, the hardware and software stacks that implement the analytics, the code, and all the tests that prove success. A DevOps build-test-deploy handles the code. Why DevOps Tools Fail at DataOps, published two days before this post went up, walks the gaps tool by tool, and the white paper DataOps Is NOT Just DevOps for Data makes the longer case.
The practical consequence is a build-versus-buy question. Assembling DataOps out of DevOps and workflow tools means knowing and managing 7 to 10 of them, and teams that try usually stop at a few unit tests in development, a hand-tended Jenkins deploy, and glue code on Airflow or Talend.
Why: cycle time and error rate have to improve together
The case for DataOps in 2020 rested on two numbers that most approaches trade against each other. A March 2020 Gartner survey, Data Management Struggles to Balance Innovation and Control, put data professionals at 56% of their time on operational execution and 22% on innovation that delivers value. DataKitchen’s 2019 DataOps survey found 30% of respondents reporting more than 11 data errors a month.
Read together, those say a data team is both slow and unreliable, and that the second condition is what produces the first: hours spent finding and fixing errors are hours not spent on new work. DataOps goes after both at once by automating the verification, which is why the choice between quality and cycle time is a fallacy once the tests run themselves.
How to build the team
Our advice on the first hire was specific: a DataOps Engineer. That role implements continuous deployment of analytics, gives data scientists sandboxes they can create on demand, automates the data operations pipeline, and builds the platforms that test and monitor data from ingestion through to the published chart. The skillset spans hybrid and cloud platforms, orchestration, data architecture, data integration and transformation, CI/CD, real-time messaging, and containers.
Improve Business Agility by Hiring a DataOps Engineer sets out why that hire comes first. A company cannot adapt to change faster than it can understand itself and its environment, which makes analytics speed the ceiling on business agility rather than a departmental concern.
Team size cuts the other way from what people expect. A three-person team gets more out of DataOps than a fifty-person team, because there is nobody spare to absorb the unplanned work.
What leadership should push for
Two things: find an executive sponsor, and measure the right pair of numbers.
The sponsor matters because DataOps changes how work gets done, and that provokes resistance. A C-level sponsor gives the team air cover while it builds capability, ties the work to strategic goals, and turns a skunkworks into a funded project. The pitch that fails is efficiency: an executive who hears that data analytics will get more efficient does not see a business objective move. Finding an Executive Sponsor for Your DataOps Initiative argues for translating DataOps into cycle time, agility, and opportunity cost instead.
Then measure two numbers in the first month: cycle time from request to production, and errors that reached a customer. Both should improve at once. If only one moves, you have automated deployment without adding quality control, or added quality control without removing manual steps.
Culture, and why more companies had not tried it
The cultural argument starts with W. Edwards Deming’s claim that 94% of problems are common cause variation: properties of the system rather than of the people inside it. A data team that looks dysfunctional usually has conflicting goals, weak process design, and a request path that stacks delay on delay, which is why what a data team does matters less than how it does it.
As for why more companies had not tried DataOps by 2020, the failure we saw most often was buying tools before changing process, which produces a dashboard nobody acts on. The starting move is smaller than a purchase: add tests to the feeds that break most often, and put the code under version control. A handful of tests is a day of work and it changes the error rate you live with. The DataOps Manifesto sets out the principles in full, and DataOps: The Questions Teams Ask Before They Start collects the questions that come up next.
FAQ
What are the key points in this blog?
EM360 chief editor Max Kurton interviewed DataKitchen founder and CEO Chris Bergh on the what, why, and how of DataOps, and EM360 published the episode in August 2020. The conversation covered the distinction between DataOps and DevOps, the business case, how to staff a DataOps team, what leadership should push for, and the cultural change involved. EM360 hosts the recording.
Where can I listen to the EM360 podcast episode with Chris Bergh?
On EM360’s own site, under the title DataKitchen: The What, Why, and How of DataOps. EM360 hosts the audio in a player on that page, and DataKitchen does not republish it, so there is no player here. The episode went up in August 2020 and is still live.
Who took part in the episode?
Two people. Max Kurton, Editor-in-Chief at Enterprise Management 360, asked the questions. Chris Bergh, founder and CEO of DataKitchen, answered them. Bergh co-wrote the DataOps Cookbook with Gil Benghiat and Eran Strod, and DataKitchen had been publishing on DataOps practice for several years by the time of the recording.
What separates DataOps from DevOps?
Deploying analytics takes four things rather than one: the data used for testing, the hardware and software stacks implementing the analytics, the code, and all the tests that prove success. A DevOps build-test-deploy handles the code. DataOps also has to handle data that changes independently of that code, which is why a pipeline that deploys correctly can still publish wrong numbers.
Who should a company hire first to start DataOps?
A DataOps Engineer. The role implements continuous deployment of analytics, gives data scientists sandboxes they can create on demand, automates the data operations pipeline, and builds the platforms that test and monitor data from ingestion through to the published chart. The skillset spans cloud platforms, orchestration, data architecture, integration and transformation, CI/CD, real-time messaging, and containers.
Why had so few companies adopted DataOps by 2020?
The most common failure was buying tools before changing process, which produces a dashboard nobody acts on. Two other blockers recurred: no executive sponsor to give the work air cover, and a pitch built on efficiency rather than on cycle time and opportunity cost. The starting move is smaller than a purchase: tests on the feeds that break most often, and version control.
