The real problem in data analytics is that teams need to
deliver insight to their customers without error
put new ideas into production rapidly
minimize their ‘insight manufacturing’ expenses
… all at the same time
As former leaders of data teams, we understand that data and analytics have three main problems: delivering insight to customers without error and rapidly putting new ideas into production while minimizing expenses. These three objectives are interconnected and essential to the success of any data team. Achieving these objectives requires a well-coordinated effort from the team with a clear understanding of the goals and objectives. The hard part is doing them all at once. That is the single biggest challenge.
Delivering insight to customers without error is critical to the success of any data team. Your customers rely on the insights provided by the team to make critical business decisions, and any mistake can have significant consequences. The team must ensure that the data they are working with is clean and accurate and that the analysis created from it is rigorous and reliable. Data Observability must be a top priority, and the team must have robust processes to ensure the data is correct and free from corruption. Teams need to check more than just the raw data – they need to check the integrated data and the products of any tools using the data (e.g., BI tools like Tableau, PowerBI, etc.). Like in product manufacturing, finding and reducing errors are the keys to data and analytics manufacturing success. Manufacturing production errors refer to mistakes or defects that occur during the manufacturing process. These errors include incorrect materials, machine malfunctions, or human errors. These errors can significantly impact the final data product’s quality, reliability, and timeliness. The day-to-day production of data analytics is also a manufacturing process. DataOps Observability is an essential practice in modern data-driven organizations that ensures real-time insights into the manufacturing process. DataOps Observability refers to real-time monitoring, detecting, and diagnosing of data status, data pipelines, data tools, and other systems. Without this essential capability, data teams will spend their time searching for problems and “fighting fires” and will not have the time to build and deploy new, valuable data analytics products. Products like DataKitchen’s DataOps Observability product can enable these insights rapidly across any technology stack or cloud.
When data teams have the time to build new analytics products, rapidly putting new ideas into production is also a crucial objective for any data team. The world of business is constantly evolving, and the team must be able to respond quickly to changing circumstances. The team must be agile and flexible, able to pivot quickly and adapt to new challenges. This requires a culture of innovation, experimentation, and willingness to take risks and try new approaches. The team must be willing to fail fast and learn from their mistakes, iterating quickly to refine their technique. It is not enough to just deploy to production quickly; teams need to lower the risk of deployment failure. Nothing is as costly as deploying changes that break valuable, working analytics. Let’s assume you have hired an intelligent 22-year-old fresh out of college. Top-tier software teams benchmark those junior engineers’ by their ability to get a minor bug fix into production in their first week of work. Why? It’s not because they trust the newbies. It is because the software team has built a system of regression, functional, and other impact tests that automatically give feedback to that junior engineer about the success (or not) of the change on the system. Data teams need to have the same benchmark capability. Products like DataKitchen’s DataOps Automation engine enable these guard rails across any technology stack or cloud.
Minimizing ‘insight manufacturing’ expenses is the third objective and often contradicts the first two objectives. The team must find ways to reduce errors and deploy rapidly without incurring unnecessary costs. This requires a deep understanding of the cost drivers for data analytics and a willingness to find creative solutions to reduce costs. The highest cost for any data analytics team is the people on that team. You can’t keep hiring more people whenever your team is tasked with answering another business question! How do you get your team to be more productive? Productivity does not come through the fingers of each data engineer or scientist; it comes from building a system around those engineers that allows them to run production with minimal errors and move new things into production quickly, with low risk, so they can focus on making their customers successful. As Gartner states: “teams guided by DataOps practices and tools will be ten times more productive than teams that do not use DataOps.”
The team leader must provide clear direction and expectations to enable this, ensuring everyone is working towards the same goals. The team leader must also foster a culture of trust and respect, empowering team members to take ownership of their work and encouraging collaboration and knowledge sharing. Measuring these goals is very important to success. As the adage goes, You can’t improve what you can’t measure. The team must be data-driven, agile, and innovative to achieve these objectives, and being able to track improvement is critical. Products like DataKitchen’s DataOps Automation provide valuable insights into the team- and system-wide performance.
Balancing against that internal perspective on the team is the challenge of dealing with customer expectations. Those expectations may seem unreasonable, but with empathy toward your business customer, you can succeed within their expectations. Your customers naturally have unrealistic demands, often believing the data they need should be delivered safely and consistently like a utility that never shuts down., Next, we will discuss these customer expectations and explore ways to manage them effectively.
Data customers think your team must:
deliver like a utility where everything is always safe and available.
create new insights in an hour because it can’t be that hard.
be magicians who quickly get things done
… and will not be convinced otherwise
Data customers often expect data analytics to be as reliable as a utility. They expect everything to be always safe and available, without downtime or interruptions. Are you happy when your home electricity goes off? No, you expect always to work. It is the same for your data analytics customers. However, producing data analytics is a complex process that involves dealing with large amounts of often unstructured and messy data and a brittle mix of data tools and technologies. You will never be able to set realistic expectations with customers to manage these expectations against that backdrop. No amount of educating customers about the data analytics process and the potential challenges that may arise will dissuade them from that opinion. They will always think: “I can get a box from Amazon the next day, so why can’t our internal reports be ready every day at 9 am?”
Another common expectation is that new insights can be delivered very, very rapidly. They ask many questions about the data, which requires changes to the manufacturing process, and are looking for answers quickly. Often the data is part of a complex decision process they are juggling. Can I make this change? Will it work? Does it have ROI? Can I sell it internally? To them, the fact that data analytics is a complex process that requires time and effort to produce is immaterial. And you know rushing the process can lead to inaccurate results and may damage the credibility of the data analytics team. No amount of educating customers about the data analytics process and the time required to produce accurate results will change their minds.
Finally, customers often view data analytics as nothing more than magic. They expect data analytics teams to be able to solve complex problems and produce results with little effort quickly. This expectation can be challenging to manage. Why? Because your busy, stressed business customer does not have the time or effort to care. They have many challenges that your team may not be able to see. They may be unable to understand our data, code, and analytics skills. No amount of educating customers about the data analytics process and the skills required to produce meaningful insights will change their minds. The fact that they see your data team as a set of magicians is both a compliment and a curse.
So what to do about these disconnects with your customers? How can your team meet these expectations while managing a complex process effectively? Data analytics teams should be transparent about their methods and communicate their findings in an understandable way to non-technical stakeholders. Additionally, data analytics teams should highlight the effort and expertise required to produce accurate results and demonstrate the value that data analytics can bring to the organization. Empathy is a two-way street – try to see the world from your customer’s perspective rather than the tired “they just don’t get it” attitude. And, of course, data teams should use DataOps automation and observability tools to get a leg up.
Delivering insight to customers without error while rapidly putting new ideas into production and minimizing expenses – simultaneously – is THE problem in data analytics. To achieve these objectives, the data team must be data-driven, agile, and innovative. Does this sound impossible? Don’t despair! The principles of DataOps enabled via DataOps Observability and Automation are keys to unlocking this challenge. DataKitchen’s scalable software accelerates your path to resolving this complicated ‘union of opposites.’
