At DataKitchen, we are believers in delivering value. We work with our customers to find a first project that can drive real benefits and meet their critical business needs. Customers use our DataKitchen technology and their experience to address a focused business problem.
Recently we’ve been working with customers in various industries: transportation, telecommunication, and consumer goods to jump-start their DataOps journey. Their business users often have no concept of what it takes to design and deploy robust data analytics. The gap between expectations and execution is one of the main obstacles keeping these analytics teams from succeeding. Managers may ask for a simple change to a report or model or a new dataset. They don’t expect it to take weeks or months.
These teams are trying to answer two simple questions. First, how can their team collaborate to reduce the cycle time to create and deploy new data analytics (data, models, transformation, visualizations, etc.) without introducing errors? And second, where to start this process? We’ve written about how to apply the ‘Theory of Constraints’ to choosing your first DataOps win. The answer relates to finding and eliminating the bottlenecks that slow down analytics development.
What follows are examples of different types of bottlenecks, why they were selected first, and the benefits of resolving those bottlenecks with DataKitchen.
Enabling Rapid Deployment to Production: From Months to Days
A telecom company needs to increase the rate at which new data features are delivered into production in the EDW. In one past example, it took four months to complete the development cycle from new ideas into production (i.e., to move code from development to production).
We worked to identify outcomes that will define success. Those include:
- Automate the manual tests, which alone will bring about major improvements in cycle time.
- Add new activities into the integration and test environment that will dramatically decrease the time it takes to find and fix issues, thus speeding features into production.
- Rapid addition of features and deployment into production in order to reduce the time from months to days (or faster).
Part of their challenge is that their current process involves a four-stage manual deployment from development to production (see below). This manual process introduces complexity, slowness, and errors (see diagram below).
This customer has many tools, including Data Science, visualization and governance tools, in their analytics toolchain. They chose to focus on changes to their core data warehouse as the first bottleneck to address with DataOps. Their other teams have similar challenges, but a focused adoption by the data warehouse team was the biggest bottleneck that offered a significant short-term business benefit if addressed.
So how to do you make sure that when you move something from a Dev Environment through each of the other separate environments, when the business needs it, that everything still works? The answer is automated testing. This company, like many today had very little automated testing in place. Almost all testing to prove that new code works (in this case they are using SQL-based data transformation on Oracle DB) was done by hand. The DataKitchen Recipe below describes how they created dozens of automated tests in the DataKitchen platform that prove that everything works as they moved the new SQL code from one environment to another.
In some ways, the bottleneck is not just technical in nature. Quickly moving code from development into production can be scary – with sometimes painful and costly business implications. What if we make a mistake? Will we get yelled at by the business? Will the business make a critical (and wrong!) decision based on erroneous data? Ensuring new feature deployment success requires both a platform like DataKitchen and an effective approach to writing tests. As part of working with the customer we spend time educating them on how to write great tests as well as the core principles of DataOps.
Reducing Errors in A Multi-Technology Toolchain through Improved Orchestration and Collaboration
A transportation company has challenges, both real-time and in batch, managing the workflow orchestration of data streaming from their vehicles into actionable insight for their employees. Like many companies, they do not have just one data architecture, they have several – batch, streaming, big data, small data, on-premises, cloud, and prescriptive and predictive models – all working together. Plus, they have different teams managing the creation and the operation of these pipelines in different locations. Whew! What that means is that the disparate teams need to develop a common outcome (report, dashboard, model, etc.) together by addressing these business challenges:
- Data Pipelines = complexity – tools, platforms, teams
- Complexity = delays and risk – in value and innovation
- Slow delivery of data products (Data Science)
- Too much manual intervention/testing (Business Reporting)
- Slower delivery than desired
But to meet those challenges, they need to work on their current data operations, technologies and delivery:
- Orchestration: Running pipelines of various technologies at the right time
- Iteration: Rapidly creating, iterating and deploying data science and data engineering pipelines and their data products (reports, dashboards, models) – Full Pipeline CI-CD
- Quality: Detecting issues and errors in complex, multi-tool data pipelines
- Hybrid: Enabling on-premise and cloud approaches and an evolving technology landscape
- Collaboration: Enabling multiple teams to collaborate more effectively to deliver data products to end-users faster with higher quality. Also enabling transparency of all operations in a complex pipeline.
So, what bottleneck did they focus on first? Orchestration of the toolchain for low error execution and enhanced collaboration. They started with a variety of technologies that the company currently uses. These include: On-Prem
- Oracle
- SQL Server
- Informatica
- Apache Nifi
- Hadoop/HDFS
- PySpark Cloud
- Redshift
- S3
- Tableau online
- Mlfow
They then created a single DataKitchen Recipe, which provides a framework to detect issues in streaming data across a multi-technology toolchain, runs tests that ensure the quality of streaming data, and alerts users if the condition is not met. That Recipe uses all their existing tools to transform data into business insight. Other important considerations for the team were that the data was validated to be ‘fit for purpose’ so that the assumptions made while transforming data remain true. They also focused upon Recipe CI/CD so that once a change passes its tests across any of the technologies, it is deployed to the pipeline in less than five minutes.
Freeing Data Scientists’ Time Through Automation of Machine Learning Deployment and Production
A consumer product team recently completed a machine learning (ML) use case combining web scraped data with Oracle service cloud data. It works great; the business users love the early versions of the product, but:
- The various pipeline components are run by hand in an inconsistent manner, and the orchestration is not automated
- The movement of code from sandbox to development is manual
- There are few tests to ensure data quality
- The web scraping component frequently fails, and there is insufficient notification and restart capabilities
DataKitchen and the company have identified another bottleneck that first needs to be addressed. Talented (and expensive) Data Scientists can create the first version of an idea but have no interest in running it on a day to day basis. How can that ML model and all its associated data transformation, code, and UI be put into operation following a DataOps Approach? To address this bottleneck the team created a Recipe in DataKitchen that provides the ability to orchestrate the components developed in the ML use case. That Recipe has the ability to run data tests and detect errors in the data in the production environment. It also has the ability to detect operational errors (e.g., the web scraping issues), alert users to those errors, and permit re-start of the processing of the data pipeline (Recipe). See diagram below.
Furthermore, to enable a rapid development cycle and enhance the Recipe to add the Oracle Service Cloud data, the team needs to develop this new pipeline capability in a Development Kitchen and deploy quickly to a Production Kitchen. See the diagram below.
Where is your bottleneck? Where can you start doing DataOps today?
DataOps applies lean manufacturing management methods to data analytics. One leading method, the Theory of Constraints, focuses on identifying and alleviating bottlenecks. Data analytics can apply this method to address the constraints that prevent the data analytics organization from achieving its peak levels of productivity. Bottlenecks lengthen the cycle time of developing new analytics and prevent the team from responding quickly to requests for new analytics. If these bottlenecks can be improved or eliminated, the team can move faster, developing, and deploying with a high level of quality in record time. If you have multiple bottlenecks, you can’t address them all at once. As these examples have shown, there are multiple ways to provide immediate, clear value for doing DataOps and working with DataKitchen!
FAQ
What are the key points in this blog?
DataOps adoption starts with one focused project that solves a business problem someone already has. The selection rule comes from the Theory of Constraints: find the bottleneck that lengthens analytics cycle time and clear that one first. Examples include a four-month manual deployment cycle shortened by automated testing, a multi-technology toolchain fixed through orchestration, and a machine learning pipeline that data scientists were running by hand.
How do you choose your first DataOps project?
Pick the bottleneck that costs the most and can be cleared quickly. Look for the stage where work piles up: manual testing, a multi-stage deployment, a pipeline that someone runs by hand. A focused first project in a single team beats a broad rollout, because it delivers a visible business benefit fast enough to earn support for the next one.
What is a bottleneck in analytics development?
A bottleneck is the step that limits how fast a data team can deliver new analytics. DataOps borrows the Theory of Constraints from lean manufacturing to find it: the constraint sets the pace of the whole system, so improving anything else changes nothing. Common ones are manual testing, hand-run pipelines, and deployment processes that need several people to sign off.
Why does a small change take months to reach production?
Because the path from development to production is usually manual. One telecom team ran a four-stage manual deployment and needed four months to move new code into production, a process that introduced complexity, slowness, and errors. The other half of the problem is fear: releasing quickly is frightening when a wrong number could send the business to a bad decision.
How does automated testing speed up analytics delivery?
Automated tests prove that a change still works before it reaches production, which removes the manual verification step and the hesitation that comes with it. One team replaced hand testing of SQL transformations with dozens of automated tests that run as code moves between environments. Writing good tests is a skill, so the principles of DataOps matter as much as the platform.
What comes after the first DataOps project succeeds?
The next bottleneck, usually in an adjacent team. Organizations rarely have only one constraint, and they cannot address them all at once, so the sequence matters more than the ambition. Teams that want DataOps skill spread across the whole data organization rather than concentrated in one group often build a center of excellence or a DataOps Dojo where staff learn on real projects.
