Data scientists work tirelessly to build and train a model then face the daunting challenge of deploying it into production. The model itself is only a fraction of the overall ML system. Moving a model from development into operations involves provisioning infrastructure, installing and configuring software, preparing data, and testing both code and data. With manual processes, these steps can require several months of effort. Integration with existing operations is also challenging. Gartner recently reported that the complexity of solution integration with existing infrastructure was named as a top barrier to AI implementations.
Software applications are typically code-driven. Developers deploy software and then they may not touch it again until there’s a requirement to fix a bug or add a new feature. Not true for ML systems. Model performance is affected by many factors, including changes in data, users, ML systems, performance requirements or the external environment. Model updates are code, data and environment-driven. Once a model is in production, it needs to be actively monitored for any degradation in performance. Without automation, it can take weeks to detect that a model’s accuracy has deteriorated and several additional weeks to retrain the model with the newest data or remediate issues affecting model performance.
Figure 1: Model performance decays over time due to changing data, code, users, system environment and other external factors. Machine learning models must be monitored in production and retrained or redeveloped periodically. (Source: Forrester)
Imagine creating, deploying and managing ten models. Or hundreds. Businesses may wish to turn to data science models to gain competitive advantage, but the onerous workflow processes related to model development, deployment and maintenance often get in the way. In the coming years, ML systems will serve as a major differentiator for companies. Markets will be led by the most agile organizations, who can train, deploy, monitor and update machine-learning systems in the shortest amount of time. Agile companies will be adept at putting new ideas into action and quickly altering course when business conditions change.
ML Agility
ModelOps, also known as MLOps, is an application of DataOps principles and automation to machine learning systems. ModelOps turns data science workflows into robust, repeatable processes executed in minimal time and with virtually zero errors. When you use DataKitchen DataOps Automation to implement ModelOps, your end-to-end machine-learning lifecycle is optimized for agility and quality. A DataKitchen ModelOps workflow includes:
- Self-Service Sandboxes – An environment instantiated (and deleted) on-demand that includes everything a data scientist needs to create, train, test and deploy machine-learning models: complete toolchain, security vault, prepackaged data sets, role-based access control for teams, integration with workflow management, and a Kitchen workspace that ties it all together as a unified environment.
- Continuous Deployment – Building, integrating, testing and deploying ML models and ML systems as an automated software-controlled process that can be executed on-demand and complete in minutes or hours.
- Meta-Orchestration – a hierarchy of orchestrations. Automated execution of complex, multi-step data pipelines that span data centers and varied toolchains.
- Automated Testing – Testing in all phases of orchestration from data ingestion to transformation to model execution to delivery of polished visualizations. DataKitchen DataOps Automation offers a single test environment encompassing heterogeneous toolchains.
- Continuous Monitoring – Repeated evaluation of model accuracy and efficacy in production with automated orchestrations or alerts when results meet statistical criteria.
- Continuous Governance – Governance workflows as repeatable, verifiable orchestrations implemented using governance-as-code automation.
ModelOps as a Competitive Advantage
Many data science professionals believe that models by themselves can provide competitive differentiation. Machine learning and AI will offer ways for companies to gain competitive advantage. What many fail to understand is that the organizational workflows related to ML model creation, deployment, monitoring and management will serve as an even greater point of competitive differentiation. Companies interested in machine learning and AI will hire smart data scientists, but not every company will employ them productively. The enterprises that implement ModelOps will be capable of iterating and updating models with far greater agility. Model agility translates into business agility. Data teams that embrace ModelOps with automated platforms like DataKitchen DataOps Automation will be better able to develop agile business processes and emerge as market leaders.
FAQ
What are the key points in this blog?
Deploying a model is a systems problem, not a modeling problem. The trained model is a fraction of the work; the rest is provisioning infrastructure, installing and configuring software, preparing data, and testing both code and data. Model updates are driven by code, data, and environment together, so accuracy has to be watched continuously. ModelOps, also known as MLOps, applies DataOps principles and automation to that whole lifecycle.
What is ModelOps?
ModelOps, also known as MLOps, is the application of DataOps principles and automation to machine learning systems. It turns data science work — provisioning environments, deploying, testing, monitoring, retraining — into repeatable processes rather than one-off manual projects. The goal is to run the machine learning lifecycle the way a disciplined software team runs continuous delivery, with tests and monitoring built into every step.
Why do machine learning models need monitoring after deployment?
Because model accuracy decays. Changes in data, users, the machine learning system itself, performance requirements, or the outside environment all move a model away from the behavior it was trained for — a decay pattern shown in a Forrester chart cited in this September 2020 post. Without automated monitoring, weeks can pass before anyone notices, and weeks more before a retrained model is live.
What makes moving a model into production so slow?
The work around the model. Provisioning infrastructure, installing and configuring software, preparing data, and testing both code and data all have to happen before a model serves anything, and done by hand those steps can stretch to months. Integrating the result with existing operations adds more. Automating those steps is what shortens the path, not a better algorithm.
What does a ModelOps workflow include?
Six parts: self-service sandboxes that create a full toolchain, data, and access control on demand; continuous deployment of models and the systems around them; meta-orchestration across toolchains and data centers; automated testing from ingestion through transformation, model execution, and delivery; continuous monitoring of accuracy in production; and continuous governance run as repeatable, verifiable orchestrations rather than manual review.
How is ModelOps different from DevOps for software?
A software release is code-driven: developers deploy, then leave the code alone until a bug or a new feature brings them back. A model is code, data, and environment-driven, so it can degrade while nothing about it changes. ModelOps therefore adds continuous accuracy monitoring and a retraining path to the usual build, test, and deploy loop that DataOps Automation runs.
