DataOps is a collection of technical practices, workflows, cultural norms, and architectural patterns that let a data team ship analytics quickly while keeping error rates very low. It applies Agile development, DevOps, and lean manufacturing to data analytics, and it treats data in production as a factory line to be tested and measured rather than a set of jobs to be watched by hand.
Those practices aim at four outcomes:
- Rapid innovation and experimentation delivering new insights to customers with increasing velocity
- Extremely high data quality and very low error rates
- Collaboration across complex arrays of people, technology, and environments
- Clear measurement, monitoring, and transparency of results
The best way to explain DataOps is to review its intellectual heritage, explore the problems it is trying to solve, and describe an example of a DataOps team or organization. Our explanations below start at a very conceptual level, but then quickly proceed into pragmatic and practical terms. We find this is the best way to help data professionals to understand the potential benefits of DataOps.
What is DataOps’ intellectual heritage?
We trace the origins of DataOps to the pioneering work of management consultant W. Edwards Deming, often credited for inspiring the post-World War II Japanese economic miracle. The manufacturing methodologies riding Deming’s coattails are now being widely applied to software development and IT. DataOps further brings these methodologies into the data domain. In a nutshell, DataOps applies Agile development, DevOps and lean manufacturing to data analytics development and operations. Agile is an application of the Theory of Constraints to software development, i.e., smaller lot sizes decrease work-in-progress and increase overall manufacturing system throughput. DevOps is a natural result of applying lean principles (e.g., eliminate waste, continuous improvement, broad focus) to application development and delivery. Lean manufacturing also contributes a relentless focus on quality, using tools such as statistical process control, to data analytics.
What is Agile Development?
For DataOps to be effective, it must manage collaboration and innovation. To this end, DataOps introduces Agile Development into data analytics so that data teams and users work together more efficiently and effectively. In Agile Development, the data team publishes new or updated analytics in short increments called “sprints.” With innovation occurring in rapid intervals, the team can continuously reassess its priorities and more easily adapt to evolving requirements, based on continuous feedback from users. This type of responsiveness is impossible using a Waterfall project management methodology which locks a team into a long development cycle, isolated from users, with one “big-bang” deliverable at the end.
Agile is most effective where requirements evolve faster than a plan can be written, which is the normal condition in data analytics: the question changes, the source changes, or the business reorganizes halfway through the quarter. Short sprints mean the team finds out it built the wrong thing in two weeks rather than in six months. In a DataOps setting, that is what lets a data team respond to a customer request at the speed the customer asked.
What is lean manufacturing and what does it have to do with data analytics?
Lean manufacturing is a methodology, originating in the Japanese manufacturing industry (e.g., Toyota), that focuses on the minimization of waste within a system without sacrificing productivity. Whereas Agile and DevOps relate to analytics development and deployment, data analytics also manages and orchestrates a data pipeline. Data continuously enters on one side of the pipeline, progresses through a series of steps and exits in the form of reports, models and views. The data pipeline is the “operations” side of data analytics. It is helpful to conceptualize the data pipeline as a manufacturing line with an active focus on quality, efficiency, constraints and uptime. To fully embrace this manufacturing mindset, we call this pipeline the “data factory.”
In DataOps, the flow of data through operations is an important area of focus. DataOps orchestrates, monitors and manages the data factory. One particularly powerful lean-manufacturing tool is statistical process control (SPC). SPC measures and monitors data and operational characteristics of the data pipeline, ensuring that statistical variance remains within acceptable ranges. SPC leads to remarkable improvements in efficiency, quality and transparency when applied to data analytics. With SPC in place, the data flowing through the operational system is verified at each step in the data-analytics pipeline. If an anomaly occurs, the data analytics team will be the first to know, through an automated alert.
While the name “DataOps” implies that it borrows most heavily from DevOps, it is all of the methodologies described above — Agile, DevOps, “lean” and statistical process control — that comprise the intellectual heritage of DataOps. Agile governs analytics development, DevOps optimizes code verification, builds and delivery of new analytics and SPC orchestrates, monitors and validates the data factory.
What is DevOps?
DevOps is an approach to software development that accelerates the build lifecycle (formerly known as release engineering) using automation. DevOps focuses on continuous deployment of software by leveraging on-demand IT resources and by automating integration, test, and deployment of code. This merging of software development (“dev”) and IT operations (“ops”) reduces time to deployment, decreases time to market, minimizes defects, and shortens the time required to resolve issues.
Using DevOps, leading companies have been able to reduce their software release cycle time from months to literally seconds. This breakthrough enabled them to grow and lead in fast-paced, emerging markets. Companies like Google, Amazon, and many others now release software many times per day. By improving the quality and cycle time of code releases, DevOps deserves a lot of credit for these companies’ success.
Isn’t DataOps just DevOps for Data?
Nearly everyone makes this assumption when they first hear the term DataOps. While a little semantically misleading, the term “DataOps” does communicate that data analytics can achieve what software development attained with DevOps. That is to say, DataOps can yield an order of magnitude improvement in quality and cycle time when data teams utilize new tools and methodologies. DevOps optimizes the software development pipeline. It is what allows companies like Amazon, Netflix and Google to execute millions of code releases per year. DataOps also accelerates software (new analytics) development but has to simultaneously manage a dynamic manufacturing operation (i.e., data operations). DataOps includes DevOps and other methodologies which apply to the unique challenges of managing an enterprise-critical data operations pipeline. To learn more about the differences between DevOps and DataOps read the white paper, DataOps Is NOT Just DevOps for Data.
| DevOps | DataOps | |
|---|---|---|
| What changes | The code | The code and the data, independently of each other |
| What you ship | An application | Analytics, plus the pipeline that keeps feeding them |
| What breaks production | A bad release | A bad release, or a source system that changed overnight with no release at all |
| What a test checks | The code does what the spec says | The code does what the spec says, and today’s data looks like data |
| Who notices first | Your monitoring | Your customer reading a dashboard, until you add process control |
| Borrowed from | Agile and lean software delivery | Agile, DevOps, lean manufacturing, and statistical process control |
How does DataOps relate to data governance?
Governance decides the rules. DataOps is how the rules get enforced without a person in the loop. Written policy that nobody can check is the normal state of affairs, and it is why governance has the reputation it has: in our survey of 600 data engineers, 69% said their company’s data governance policies make their day-to-day job harder.
The DataOps answer is to write the policy as tests that run with everything else. Access rules, retention windows, PII handling, and lineage requirements become checks in the pipeline, so the governance question changes from “did anyone follow the standard?” to “which run failed the check, and when?” That is the subject of Data Governance as Code.
Data Quality The DataOps Way
Most data quality work is run as a project: someone profiles the warehouse, writes a report, and the report is out of date by the time it is read. DataOps treats quality as a production control instead. Tests run on every table on every run, they are generated rather than hand-written so coverage does not depend on somebody’s afternoon, and a failure stops the pipeline rather than surfacing in a dashboard three days later. The measurement is a by-product: you can say what the error rate was last month, per source, without a separate audit. That is the argument in Data Quality the DataOps Way. The practice itself is laid out in What Is DataOps Data Quality?.
What problem is DataOps trying to solve?
DataOps exerts control over your workflow and processes, eliminating the numerous obstacles that prevent your data organization from achieving high levels of productivity and quality. We call the elapsed time between the proposal of a new idea and the deployment of finished analytics “cycle time.” Many organizations require months of cycle time to deploy 20 lines of SQL. Lengthy cycle times discourage and disappoint users and hinder creativity.
Ideally, data teams work hand-in-hand with their users like a well-oiled machine, fielding new idea proposals, implementing them rapidly and quickly iterating toward higher-quality models and analytics. Unfortunately, our experience is the opposite. Data teams are constantly interrupted by data and analytics errors. A March 2020 Gartner survey found data professionals spend 56% of their time on operational execution and only 22% on innovation that delivers value. Our own survey of 600 data engineers found 50% naming time spent finding and fixing errors as a top cause of burnout, tied with time spent maintaining pipelines and manual processes. Slow and error-prone development disappoints and frustrates data team members and stakeholders. Lengthy analytics cycle time occurs for a variety of reasons:
- Poor teamwork within the data team
- Lack of collaboration between groups within the data organization
- Waiting for IT to disposition or configure system resources
- Waiting for access to data
- Moving slowly and cautiously to avoid poor quality
- Requiring approvals, such as from an Impact Review Board
- Inflexible data architectures
- Process bottlenecks
- Technical debt
- Poor quality creating unplanned work
Obstacles that delay analytics cycle time
Case Study: Meeting the Product Launch Challenge with DataOps
Pharmaceutical giant Celgene improved cycle time by an order of magnitude and can support 12X the number of schema changes and 24X the number of data analysts per data engineer. Celgene encounters very, very few errors or missed SLAs. The team presented these numbers themselves in a DataKitchen webinar.
Watch the Celgene team tell it →
What does a DataOps organization look like?
As we explained above DataOps is not necessarily one thing. To give you an idea of how DataOps works, we’ll describe organizations that use DataKitchen.
Tens or hundreds of data sources are consolidated into a data lake, pass through a complex series of transformations, and are pushed to users through analytics charts and graphs, all under automated orchestration. Automated tests (statistical process controls) validate the data entering the system and the inputs, outputs, and business logic at each transformation step. Status, warning, and failure alerts from all of these process controls to advance to the data team in real-time. Tests also implement a virtual Andon cord to stop a data source in the case of fatal errors. Data errors virtually never enter the data analytics pipeline, and processing errors are caught mid-pipeline before corrupting analytics. Quality and uptime KPPs (key performance parameters) for the data pipeline rise sharply, well above targets. Unplanned work caused by errors falls by more than 99% in the deployments we have run. All of the inefficient manual effort previously devoted to operating, verifying, and fixing the data pipeline is redeployed to higher value-add activities. The data organization stops relying on hope and heroism.
The process and workflow for developing new analytics have been streamlined and operate seamlessly. The target operations environment is abstracted and replicated in virtual workspaces, improving test accuracy, repeatability, and analytics portability. Cycle time shrinks from months to days or hours.
DataOps utilizes process and workflow automation to improve and facilitate communication and coordination within a team and between the groups in the data organization. DataOps restructures data analytics pipelines as services (or microservices) that create a robust, transparent, efficient, repeatable analytics process that unifies all development and operations workflows. It enables teams to work independently, according to the iteration cadence appropriate to their toolchain, and then, with minimal manual steps, brings their work together into a unified whole for delivery to customers.
What is a DataOps Engineer?
DataOps Engineers create and implement the processes that enable successful teamwork within the data organization. They design the orchestrations that enable work to flow seamlessly from development to production. They make sure that environments are aligned and that hardware, software, data, and other resources are available on demand. DataOps Engineers also manage tools that promote reuse — a big multiplier of productivity.
The rise of the DataOps Engineer will completely change what people think of as possible in data analytics. Through tools automation, the DataOps Engineer eliminates data lifecycle bottlenecks, which sap data team productivity. A DataOps Engineer who understands how to automate and streamline data workflows can increase the data team’s productivity by orders of magnitude. A person like that is worth their weight in gold. The role of DataOps Engineer goes by several different names. The skillset includes hybrid and cloud platforms, orchestration, data architecture, data integration, data transformation, CI/CD, real-time messaging, and containers.
How do you prove that DataOps is really adding value?
DataOps will deliver an unprecedented level of transparency into your operations and analytics development. DataOps automated orchestration provides an opportunity to collect and display metrics on all of the activities related to analytics. Common DataOps dashboard metrics relate to team collaboration, error rates, productivity, deployments, tests, and delivery time. We call this the CDO dashboard. The dashboard contains some common metrics. DataOps easily customizes these metrics to meet an organization’s specific needs.
Here’s a short explanation of the CDO Dashboard:
- Team Collaboration — Teamwork is measured by the creation of virtual workspaces, also known as “Kitchens.” Each Kitchen creation corresponds to a new project or sub-project in a team context.
- Error Rates — The graph shows production warnings at a rate of 10 per week falling to virtually zero. This is the positive result of the 100+ tests that are now operating 24x7 checking data, ETL, processing results, and business logic. As the number of tests increases, the data pipeline falls under increasingly robust quality controls.
- Productivity — Team productivity can be measured by the number of tests and analytics created. The rise in “keys” (steps in data pipelines) coupled with the rise in test coverage shows a thriving development team. Also, the number of Kitchen merges at the top right shows the completion of projects or sub-projects. The “Feature to Dev” metric shows new analytics ready for release. “Dev to Prod” merges represent deployments to production (data operations).
- On-time Delivery — Mean deployment cycle time falls sharply, meeting the target service level agreement (SLA).
How do I implement DataOps?
You do not buy DataOps, and you do not have to rebuild your stack to start. It is a set of processes you add to the toolchain you already run, in an order where each step is worth doing on its own if the next one stalls. Our white paper 7 Steps to Implement DataOps works through the sequence:
- Add data and logic tests: every time data flows in and every time work moves through a stage, check it. Start with a handful of tests and add one every time something breaks.
- Use a version control system: everything that produces analytics is code, including SQL, scripts, configuration, notebooks, and the tests themselves.
- Branch and merge: a team member takes a copy, works on it, and merges it back when it passes its tests.
- Use multiple environments: everyone gets their own place to work with real data, so nobody develops in production.
- Reuse and containerize: break the work into components other people can run without asking you how.
- Parameterize your processing: one pipeline that takes arguments beats nine copies that have drifted apart.
- Work without fear and heroism: when the tests and the deployment are automated, shipping stops being an act of courage.
Two things help before you start. The DataOps Maturity Model assessment scores where your practice stands today and returns a report you can hand to a skeptic. DataOps Fundamentals covers the same ground as a free, self-paced course in five modules.
What is a DataOps Platform?
A DataOps platform is the software that runs the practices above. It creates the environments, runs the tests, orchestrates the pipeline, and reports what happened. Some organizations assemble one from parts they already own. Adopting a platform built for it is faster, because the integration work between all those parts is where homegrown efforts stall.
DataKitchen DataOps Automation is ours. It brings analytics planning, development, and operations into one framework, and it pulls your existing tools into automated orchestrations instead of replacing them. Four capabilities do the work:
- Safe, synchronized workspaces: production and development environments stay aligned, so a deployment does not fail on a difference between them. Access control secures each workspace. Starting a project means spinning up a sandbox in minutes with test data, validation tests, tools, and a password vault already in it, rather than waiting months for IT.
- Automated deployment: new analytics pass their validation tests and move from development to production in a few clicks. The verification tests do the job your impact review board was doing.
- Orchestration, testing, and monitoring: tests sit throughout the pipeline, checking work in progress and looking for anomalies as data arrives from hundreds or thousands of sources. When a test fails, the response matches the severity: a warning, an alert, or suspension of the data source. Dashboards summarize test results, quality, and productivity, so you can show what the DataOps effort has changed.
- Collaboration: the workspaces give analytics a defined path from person to person into production. Sandboxes carry reusable components, source control tracks the artifacts, and everyone reads the same view of the development and operations pipelines.
DataKitchen is not the only option. There are good tools for pipeline orchestration, automated testing, production and quality alerts, deployment automation, sandbox creation, and model deployment, and most of them interoperate with what we build. What matters is that something in your stack owns each of those four capabilities. If nothing does, the work lands on a person, and that person is the constraint.
Every tool vendor claims to do DataOps. Has the term lost its meaning?
Since DataOps awareness spiked in 2018, marketers have started to hijack the term and bend it toward whatever technology they are selling. Many of these tools can contribute to DataOps. No tool delivers DataOps by itself. An excellent place to get a synthesized, publicly vetted view of DataOps is Wikipedia. You can also read and sign the DataOps Manifesto. Our recent blog post, A Guide to Understanding DataOps Solutions, helps cut through all the marketing spin.
It’s important to remember that DataOps is a combination of methodologies and tools. Stay focused on the goals: improving data and analytics quality, reducing the cycle time of creating new analytics, and increasing the productivity of the data organization by orders of magnitude. You can’t go wrong serving these aims.
Where does AI fit into DataOps?
Two ways, and they pull in opposite directions.
AI writes pipeline code fast, which raises the value of everything DataOps already asks for. An agent needs somewhere to work that is not production, and it needs tests to tell it whether the transform it just rewrote is correct. Teams with environments and test coverage get real work out of coding agents. Teams without them get plausible SQL and no way to grade it. That gap is what our AI enablement work addresses.
The other direction is the risk. A language model answers a question about your data with the same confidence whether the tables underneath it are verified or badly broken. It cannot know the eligibility file truncated last night. The more of your analytics an AI writes or explains, the more the trust rests on the tests underneath it rather than on the model.
What is Data Journey First DataOps?
Data Journey First DataOps is your first step in making DataOps happen in your organization. The idea is to first focus on understanding and observing the journey that data takes through your production environment — from ingestion to processing to delivering actionable insights. This monitoring process identifies data errors, tool problems, and timing issues, enabling a quick win for your DataOps implementation by driving immediate improvements. Lowering production errors increases the reliability of your data and gives your team more time to focus on automation. And all this has to happen in days, not months. There are only five pillars that teams need to get value quickly, with little work, and without significantly changing what they already have working in production.
For more information on Data Journey ideas and background, see:
- Data Journey Manifesto
- Why the Data Journey Manifesto?
- Five Pillars of Data Journeys
- Webinar: Data Journey — The Missing Piece
What is DataOps Observability?
DataOps Observability monitors, tests, alerts, and analyzes your data estate in real-time. It provides a view of every data journey from data source to customer value, from any team development environment into production, across every tool, team, environment, and customer so that problems are detected, localized, and understood immediately.
DataOps Observability makes your Data Journeys come alive!
Should I be skeptical of the hype around DataOps?
Probably, but DataOps is based upon a solid foundation that includes Agile development, DevOps, lean manufacturing and statistical process controls. These mature methodologies have added value in enterprises and businesses for decades.
Where can I learn more about DataOps?
You’ve come to the right place! A great place to start is to read The DataOps Cookbook. Other useful resources include:
- [White Paper] DataOps is NOT Just DevOps for Data
- [White Paper] The Seven Steps to Implement DataOps
- [Blog] For Data Team Success, What You Do is Less Important than How You Do It
- [Blog] A Guide to Understanding DataOps Solutions
- [Webinar] Why Do DataOps?
- [Book] Recipes for DataOps Success
- [Book] The DataOps Way to Data Quality
- [Pillar page] What Is DataOps Process Observability?
- [Pillar page] What Is DataOps Data Quality?
- [Explainers] The What is… hub
- [Assessment] DataOps Maturity Model
- [Free course] DataOps Fundamentals Certification
- [Manifesto] Data Journey Manifesto
- [Blog] Data Journey First DataOps
- [Latest Link Collection] DataKitchen Resource Guide To Data Journeys & Data Observability & DataOps