Similar to how DevOps resulted in a transformative improvement in software development and Lean has resulted in improvements in manufacturing, DataOps aims to create an agile data culture in organizations.
Currently, data teams at most organizations have high errors in data management operations every month, due to factors like incorrect data, broken reports, late delivery, and customer complaints. Most companies are also slow in creating new development environments as well as deploying data analytic pipeline changes to production.
Estabrooks mentioned that hero mentality that we see in some teams is actually career-ending and is bad for individual team members and also for the team as a whole. Instead, he recommends teams to create repeatable processes for quality and predictable database builds and deployments. He referred to the book The Phoenix Project for best practices in DevOps area.
A DataOps mindset change is needed in order to power agile data culture; this includes transition from manual operations to automated operations, a tool-centric approach to a code-centric one, as well as integrating quality into product features from earlier phases in the data management lifecycle.
Estabrooks discussed a seven-step process for diverse teams in organizations (data analysts, data scientists, and data engineers) to realize the DataOps transformation and deliver business value quickly and with high quality. The steps include the following:
He described the functional and physical perspectives of DataOps architecture, which include a DataOps platform consisting of separate components for storage, metadata, authentication, secrets management, and metrics.
Estabrooks concluded the presentation by listing the goals of DataOps architecture, which include updating and publishing changes to analytics within a short time without disrupting operations, discovery of data errors before they reach published analytics, and creating and publishing schema changes as frequently as every day.