On-Demand Webinar · 1 hr 5 min

A Day in the Life of a DataOps Engineer

DataKitchen DataOps engineers Chip Bloche and Priyanjna Sharma on what the role does day to day: the pipelines they own, the skills it takes, and the point at which a data team needs one. Recorded August 2021; updated August 2026.

Presented by Chip Bloche, Priyanjna Sharma, Chris Bergh

What you'll learn 7 points
  • For a data warehouse team the product is the dataset, and for an analyst it is the analysis. For a DataOps engineer the product is an effective, repeatable process, and the nuggets of code moving through it are fungible.
  • Spending 15 percent of a data team's time on DataOps changes the split from 97 percent development and 3 percent process work to 85 percent and 15 percent. Software teams already run a higher ratio, closer to 77 and 23.
  • A DataOps engineer automates eight things: production orchestration, production data monitoring and testing, self-service environments, development regression and functional tests, test data, deployment, shared components, and process measurement.
  • The automated tests on a DataOps project cover data completeness, data correctness, business rules, location balance, historic balance, and statistical process control.
  • The skill set is a scripting language such as Python or Bash, SQL, git, a DataOps tool, a DevOps configuration tool such as Terraform, Puppet, or Docker and Kubernetes, agile process tools such as Jira, and familiarity with the toolchain the data engineers, scientists, analysts, and governance team already use.
  • A DataOps engineer runs toward error rather than away from it. Every error becomes a new automated test, and many simple tests catch more than a few complex ones.
  • The failure mode is the opposite instinct: meet the spec and only the spec, demand perfection and stifle feedback, resist change, hide risk, and quality-check the end product rather than the process.

Prefer to read it? The written version is in A Day in the Life of a DataOps Engineer.

Slides

40 slides

Questions from this session

What does a DataOps engineer do?

A DataOps engineer owns the assembly line that data engineers, scientists, analysts, and governance staff deliver through, working on the value and innovation pipelines rather than in them. Day to day that means managing environment pipelines and toolchain integration, creating and deploying production pipelines and tests, monitoring execution, reporting on process metrics, and building shared components other teams reuse.

How is a DataOps engineer different from a data engineer?

A data engineer integrates data sets; a DataOps engineer integrates data flows. The distinction shows up in what each one delivers: a data warehouse developer's product is the dataset, an analyst's product is the analysis, and a DataOps engineer's product is an effective, repeatable process measured by speed, flexibility, consistency, and reliability.

What skills does a DataOps engineer need?

A scripting language such as Python or Bash, SQL, source control with git, a DataOps tool, and a DevOps configuration tool such as Terraform, Puppet, or Docker and Kubernetes. Add agile process skills and tools such as Jira, plus enough familiarity with the toolchain the data engineers, scientists, analysts, and governance team use to work across all of them.

How much of a data team's time should go to DataOps?

About 15 percent. Without a DataOps engineer a team spends roughly 97 percent of its effort on data and analytics development and 3 percent on the process that carries it, and the result is high error rates and deployment latency measured in weeks or months. Shifting to 85 and 15 moves deployment latency to hours or minutes.

What tests belong in an automated data pipeline?

Six categories: data completeness, data correctness, business rule tests, location balance, historic balance, and statistical process control. Test every step of the way rather than only the end product, and prefer many simple tests over a few complex ones. Testing what you already know to be true is worthwhile, because that is what breaks silently.

What does it mean to run toward error?

It means treating error as the feedback loop that makes a complex process reliable rather than as something to hide. Each error becomes a new automated test, systems are built expecting people to make mistakes, and people are empowered rather than blamed. Silence is not consent, so a DataOps engineer seeks feedback rather than waiting for complaints.

Where to go next