Breaking Through the Noise
A Guide to Understanding DataOps Solutions
DataOps is the hot topic on every data professional’s lips these days, and we expect to hear much more about DataOps in 2020. This is not surprising given that DataOps holds true potential for enabling enterprise data teams to generate significant business value from their data. Companies that implement DataOps find that they are able to reduce cycle times from weeks (or months) to one day, virtually eliminate data errors, and dramatically improve the productivity of data engineers and analysts.
As a result, vendors that market DataOps capabilities have grown in pace with the popularity of the practice. To date, we count over 70 companies in the DataOps ecosystem. However, the rush to rebrand existing products as related to DataOps has created some marketplace confusion. Because it is such a new category, both overly narrow and overly broad definitions of DataOps abound. As a result, it is easy to get overwhelmed when trying to evaluate different solutions and determine whether they will help you achieve your DataOps goals.
So, What is DataOps Anyway?
In short, DataOps is a set of technical practices, cultural norms, and architectures that enable:
-
Rapid experimentation and innovation for the fastest delivery of new insights to customers;
-
Low error rates;
-
Collaboration across complex sets of people, technology, and environments;
-
Clear measurement and monitoring of results.
Similarly, Gartner defines DataOps as, “a collaborative data management practice focused on improving the communication, integration, and automation of data flows between data managers and data consumers across an organization.” Like its DevOps cousin, key elements of DataOps include increased deployment frequency, automated testing and monitoring, version control, and collaboration.
This sounds great and you are ready to get started, but the next big question is how can your organization best achieve this transformation? How can you sift through all the marketing speak and find the solutions that will truly help you?
Understanding DataOps Solutions
DataOps addresses a broad set of workflow processes, including analytics creation and your end-to-end data operations pipeline. In general, it’s not a single tool you can purchase and forget. Fundamentally, any DataOps solution should improve your ability to orchestrate data pipelines, automate testing and monitoring, and speed new feature deployment – while continuing to choose the right tool for the right part of the job.
To be certain, many companies that are marketing their products as DataOps solutions play a critical role in the ecosystem. However, it is important to understand exactly what role they play. If you purchase a fancy new ETL tool, will you suddenly realize all the benefits of DataOps? Probably not.
When evaluating DataOps solutions, consider the following ways that companies are marketing their capabilities.
The Data Toolchain – Many tools being marketed today as DataOps solutions are simply independent components of the data toolchain that collect, store, transform, visualize, and govern the data running through the pipeline. Although all of these technologies play an important role in the value pipeline, they do not ensure that each step in the data pipeline is executed and coordinated as a single, integrated, and accurate process or help people and teams better collaborate. Remember that a DataOps process automates the orchestration and testing of these tools across the pipeline. In fact, in a true DataOps environment, it does not matter which data tools you use. Your team can continue to use the ETL or analytics tools they like best or add new tools at any time. Typically, components of toolchain are being marketed as DataOps solutions in two different ways.
-
DataOps Rebranding – One of the reasons that the concept of DataOps has become so muddied is because some companies are rebranding the actual concept of DataOps to fit with what their product does. For example, DataOps has been rebranded as ETL (e.g., Hitachi Vantara, Attunity), streaming ETL (e.g., StreamSets, Lenses.io), or data virtualization (e.g., Delphix).
-
The Halo Effect – Because DataOps is a hot marketing term it is not surprising that many data companies are using this concept in their marketing to generate interest. The companies doing “halo effect” marketing are using the correct definition of DataOps. However, if you read closely, the message is generally that, “DataOps is great, but use our tool first.” Some examples of this type of marketing are IBM’s marketing of its Cloud Pak for Data, Trifacta for end-user data prep, and Qlik for data analytics.
Data Process Tools – Data process and automation tools are being correctly marketed as important components of a DataOps solution. You’ll need some combination of these tools if you decide to implement DataOps yourself. Many popular DevOps tools can also be used.
-
Orchestration of end-to-end multi-tool, multi-environment pipelines can be facilitated by tools like Apache Airflow or Saagie.
-
Automated Testing and Monitoring at every step in production and development pipelines is important to catch and address errors before they reach the business user. iCEDQ is a leading testing and monitoring platform.
-
Environment and Deployment technologies allow teams to spin-up self-service work environments and innovate without breaking production. New features can be deployed with the push of a button. There are a host of tools built for this purpose, including well-known open-source tools such as Git (version control), Docker (containerization), and Jenkins (CI/CD).
All-in-One DataOps Platforms – Building a DataOps environment is challenging and requires a true organizational transformation and commitment of time and resources. Even the best-equipped organizations can encounter obstacles trying to bring it all together. DataKitchen offers the first end-to-end platform that can serve as a foundation for your DataOps initiative. It seamlessly automates and manages workflows related to both data operations and new analytics development, using the tools you already have. In fact, the DataKitchen platform can interoperate with any of the data toolchain and process tools mentioned above. The platform fosters collaboration by providing a single view of the entire pipeline. Version control and environment management enable work to move seamlessly from person to person or team to team. The platform also provides useful metrics that show whether your DataOps initiative is adding value.
DataOps, when implemented correctly, holds exciting promise for data teams to be able to reclaim control of their data pipelines and deliver value instantly without errors. It is easy to get confused by all the marketing noise, but remember that DataOps, at its core, is a collaborative process that orchestrates data pipelines, automates testing and monitoring, and speeds new feature deployment. Whether you use an all-in-one tool like DataKitchen or build it yourself, the right combination of tools, processes, and people are critical to make DataOps a success.
To learn more about how a DataOps Platform can help your data organization develop analytics at lightning speed and eliminate errors, contact us at www.datakitchen.io.
FAQ
What are the key points in this blog?
DataOps is a set of technical practices, cultural norms, and architectures, not a single product. A 2019 census counted more than 70 companies in the DataOps ecosystem, and much of that marketing rebrands an existing category or borrows the term to sell a toolchain component. Judge a solution on whether it orchestrates pipelines, automates testing and monitoring, and speeds deployment while letting you keep the tools you already use.
What is DataOps?
DataOps is a set of technical practices, cultural norms, and architectures that enable rapid experimentation and innovation, low error rates, collaboration across complex sets of people, technology, and environments, and clear measurement and monitoring of results. Gartner describes it as a collaborative data management practice focused on improving the communication, integration, and automation of data flows between data managers and data consumers.
Is DataOps a single tool you can buy?
No. DataOps addresses a broad set of workflow processes, from analytics creation through end-to-end data operations, so no purchase delivers it on its own. Buying a new ETL tool will not produce the benefits. What a DataOps solution should do is improve your ability to orchestrate pipelines, automate testing and monitoring, and speed deployment while you keep choosing the right tool for each job.
How do vendors rebrand existing products as DataOps?
Two ways. Rebranding redefines DataOps to match what a product already does, so ETL, streaming ETL, or data virtualization gets presented as DataOps itself. The halo effect uses the correct definition but concludes that you should buy the vendor’s tool first. Both patterns describe toolchain components that collect, store, transform, visualize, or govern data rather than orchestrating and testing them.
Do you have to replace your existing data tools to adopt DataOps?
No. In a true DataOps environment it does not matter which data tools you use — your team can keep the ETL or analytics tools it likes best and add new ones at any time. DataOps sits above the toolchain, orchestrating and testing the steps those tools perform, and a platform built for it interoperates with toolchain and process tools alike.
What should you look for when evaluating a DataOps solution?
Orchestration of end-to-end, multi-tool, multi-environment pipelines; automated testing and monitoring at every step of production and development; and environment and deployment management that lets teams spin up self-service workspaces without breaking production. If you build it yourself, that means combining tools such as Apache Airflow, a testing platform, Git, Docker, and Jenkins.
