As recently as eighteen months ago, we would mention “DataOps” and get blank stares.
As recently as eighteen months ago, we would mention “DataOps” and get blank stares. Today, analyst firms Gartner, Eckerson, 451, and Forrester publish extensively on DataOps and interest in DataOps is soaring. There are thousands of articles per year written about DataOps in the media – one of our own DataOps blog posts has been viewed 40,000+ times. DataKitchen began speaking and writing about DataOps principles long ago, even before the industry termed it DataOps. We can only marvel at where we started and the momentum that DataOps achieved in 2019.
In many enterprises across the globe, data analytics is broken. Access to data is restricted, impeding new development. There are no checks or controls on data, so errors frequently corrupt charts and graphs. Technical debt causes data scientists to spend 75% of their time on manual steps and unplanned work. Managers rely on heroism – employees who work weekends – to compensate for inadequate processes. Small changes to analytics take weeks or months to make it into operations. The rate of failed data projects is so high that the percentage of organizations that consider themselves “data-driven” is actually falling in surveys.
Proponents of DataOps see ways to overcome these challenges. We envision a world of “data democracy” – where data flows openly into data organizations and feeds warehouses for analysts and business users. We imagine architectures that test and statistically control data at every stage of the data operations pipeline, so errors are trapped and remediated before they reach analytics. We see a new approach to analytics where secure development sandboxes with tools, data and resources are spun up instantly, and new analytics are tested and released in rapid increments via automated DevOps orchestration. The robust processes and quality controls inherent in DataOps mean less technical debt, less unplanned work and less employee heroism on weekends.
DataKitchen is far from the only company promoting DataOps as a better way to do analytics. The DataOps ecosystem expanded to 70+ vendors and OSS projects in 2019. To further illustrate the marked rise in DataOps, we’ve compiled a list of metrics that show just how much DataOps-related activity has grown and developed over the past year:
- 7,000+ signatures to the DataOps Manifesto
- Searches on the term “DataOps” growing over 170% year over year
- DataOps inquiries at analyst firms are up over 200% since last year
- Three industry conferences devoted to DataOps and tracks on DataOps at several other conferences
- Noted DevOps author Gene Kim including a chapter on DataOps in his latest book “The Unicorn Project.”
- Quantity of media articles referencing DataOps doubled in a year
Not only is the ecosystem creating articles, blogs, videos and presentations. DataOps is now the subject of several excellent books published by notable industry players and thought leaders:
- DataOps Cookbook by Christopher Bergh, Gil Benghiat, and Eran Strod (FREE)
- DataOps: The Authoritative Edition by John Schmidt, Kirit Basu
- Creating a Data-Driven Enterprise with DataOps; Inside Facebook, Uber, Linkedin, Twitter and eBay by Ashish Thusoo; Joydeep Sen Sarma
- DataOps A Complete Guide by Gerardus Blokdyk
- Practical DataOps: Delivering Agile Data Science at Scale Paperback by Harvinder Atwal
When we entered 2019, Gartner had just placed DataOps on the coveted innovation phase of the Hype Cycle for Data Management. Afterward, there was a flood of DataOps interest and ideas exchanged. Looking ahead, we see even greater opportunities to promote DataOps efficiency. Data organizations that suffer from long cycle time and poor quality can apply DataOps methods to become more responsive to user and market requirements. We anticipate spending the upcoming year doing more speaking and writing about DataOps, and, in 2020, we hope to see even more case studies from enterprises that are applying DataOps in their data operations. We look forward to continuing the conversation.
