DataOps Soars in 2019

by | Dec 30, 2019 | Blog

As recently as eighteen months ago, we would mention “DataOps” and get blank stares. 

As recently as eighteen months ago, we would mention “DataOps” and get blank stares. Today, analyst firms GartnerEckerson451, and Forrester publish extensively on DataOps and interest in DataOps is soaring. There are thousands of articles per year written about DataOps in the media – one of our own DataOps blog posts has been viewed 40,000+ times. DataKitchen began speaking and writing about DataOps principles long ago, even before the industry termed it DataOps. We can only marvel at where we started and the momentum that DataOps achieved in 2019.

In many enterprises across the globe, data analytics is broken. Access to data is restricted, impeding new development. There are no checks or controls on data, so errors frequently corrupt charts and graphs. Technical debt causes data scientists to spend 75% of their time on manual steps and unplanned work. Managers rely on heroism – employees who work weekends – to compensate for inadequate processes. Small changes to analytics take weeks or months to make it into operations. The rate of failed data projects is so high that the percentage of organizations that consider themselves “data-driven” is actually falling in surveys.

Proponents of DataOps see ways to overcome these challenges. We envision a world of “data democracy” – where data flows openly into data organizations and feeds warehouses for analysts and business users. We imagine architectures that test and statistically control data at every stage of the data operations pipeline, so errors are trapped and remediated before they reach analytics. We see a new approach to analytics where secure development sandboxes with tools, data and resources are spun up instantly, and new analytics are tested and released in rapid increments via automated DevOps orchestration. The robust processes and quality controls inherent in DataOps mean less technical debt, less unplanned work and less employee heroism on weekends.

DataKitchen is far from the only company promoting DataOps as a better way to do analytics. The DataOps ecosystem expanded to 70+ vendors and OSS projects in 2019. To further illustrate the marked rise in DataOps, we’ve compiled a list of metrics that show just how much DataOps-related activity has grown and developed over the past year:

  • 7,000+ signatures to the DataOps Manifesto
  • Searches on the term “DataOps” growing over 170% year over year
  • DataOps inquiries at analyst firms are up over 200% since last year
  • Three industry conferences devoted to DataOps and tracks on DataOps at several other conferences
  • Noted DevOps author Gene Kim including a chapter on DataOps in his latest book “The Unicorn Project.
  • Quantity of media articles referencing DataOps doubled in a year

Not only is the ecosystem creating articles, blogs, videos and presentations. DataOps is now the subject of several excellent books published by notable industry players and thought leaders:

  • DataOps Cookbook by Christopher Bergh, Gil Benghiat, and Eran Strod (FREE)
  • DataOps: The Authoritative Edition by John Schmidt, Kirit Basu
  • Creating a Data-Driven Enterprise with DataOps; Inside Facebook, Uber, Linkedin, Twitter and eBay by Ashish Thusoo; Joydeep Sen Sarma
  • DataOps A Complete Guide by Gerardus Blokdyk
  • Practical DataOps: Delivering Agile Data Science at Scale Paperback by Harvinder Atwal

When we entered 2019, Gartner had just placed DataOps on the coveted innovation phase of the Hype Cycle for Data Management. Afterward, there was a flood of DataOps interest and ideas exchanged. Looking ahead, we see even greater opportunities to promote DataOps efficiency. Data organizations that suffer from long cycle time and poor quality can apply DataOps methods to become more responsive to user and market requirements. We anticipate spending the upcoming year doing more speaking and writing about DataOps, and, in 2020, we hope to see even more case studies from enterprises that are applying DataOps in their data operations. We look forward to continuing the conversation.

Sign-Up for our Newsletter

Get the latest straight into your inbox

Monitor every Data Journey in an enterprise, from source to customer value, in development and production.

Simple, Fast Data Quality Test Generation and Execution. Your Data Journey starts with verifying that you can trust your data.

Orchestrate and automate your data toolchain to deliver insight with few errors and a high rate of change.

recipes for dataops success

DataKitchen Consulting Services


Assessments

Identify obstacles to remove and opportunities to grow

DataOps Consulting, Coaching, and Transformation

Deliver faster and eliminate errors

DataOps Training

Educate, align, and mobilize

Commercial Pharma Agile Data Warehouse

Get trusted data and fast changes from your warehouse

 

dataops-cookbook-download

DataOps Learning and Background Resources


DataOps Journey FAQ
DataOps Observability basics
Data Journey Manifesto
Why it matters!
DataOps FAQ
All the basics of DataOps
DataOps 101 Training
Get certified in DataOps
Maturity Model Assessment
Assess your DataOps Readiness
DataOps Manifesto
Thirty thousand signatures can't be wrong!

 

DataKitchen Basics


About DataKitchen

All the basics on DataKitchen

DataKitchen Team

Who we are; Why we are the DataOps experts

Careers

Come join us!

Contact

How to connect with DataKitchen

 

DataKitchen News


Newsroom

Hear the latest from DataKitchen

Events

See DataKitchen live!

Partners

See how partners are using our Products