The Challenge
A large pharmaceutical Business Analytics (BA) team struggled to provide timely analytical insight to its business customers. The company invested significant effort into managing lists of potential prescribers for certain drugs and treatments. However, the BA team spent most of its time overcoming error-prone data and managing fragile and unreliable analytics pipelines.
Customers and market forces drive deadlines and timeframes for analytics deliverables regardless of the level of effort required. It may take six weeks to add a new schema, but the VP may say she needs it for this Friday’s strategy summit. The BA team was under constant pressure to deliver analytics on demand.
The BA team often did not receive a data platform tailored to their needs. IT gave the business analysts a bunch of raw tables, flat files and other data. What they received from IT and other data sources were, in reality, poor-quality data in a format that required manual customization. Out of hundreds of tables, business analysts had to pull together the ten tables that mattered to a particular question. Tools like Alteryx helped a little, but business analysts still performed an enormous amount of manual work. When the project scope and schedule were fixed, managers had to add more staff to keep up with the workload.
If the IT or data engineering team did not respond with an enabling data platform in the required time frame, the business analyst performed the necessary data work themselves. This ad hoc data engineering work often meant coping with numerous data tables and diverse data sets using Alteryx, SQL, Excel or similar tools. Business analysts were living under constant pressure to deliver. They spent 80% of their time performing tasks that were necessary but far removed from the insight generation and innovation that their business customers needed.
The dynamic nature of data leads to complexity, including significant quality control (QC) challenges. Testing and validating analytics took as long or longer than creating the analytics. QC is extraordinarily time-consuming unless it is automated. Business analysts needed the flexibility that comes with control of data and orchestrations. Management of the process can’t be performed “off the grid” without control, enforcement and consistency.
The Solution
To meet the demand for continuous, high-quality insight, the BA team implemented a DataOps “process hub.” This allows the Data Analysts to take control of the workflows that operate on and around data. A process hub ensures that the processes and workflows that create the enterprise data platform are just as important as the data itself. Some data analysts use long checklists to execute steps that integrate, clean and transform data. A process hub automates, synthesizes and synchronizes these steps, so they do not have to be executed and managed manually.
Workflows that add value to data are critical elements of intellectual property that must be retained and controlled. A DataOps process hub instantiates all processes and workflows (i.e., tribal knowledge) related to data and analytics into code. With a process hub, workflows are curated, managed and continuously improved. A robust and vibrant process hub is a clear sign of data organizational maturity.
The process hub manages significant complexity because data flows in many directions between platforms, tools, people and functions. As data keeps changing, the process hub enforces quality and the functional order of operations. Valid data depends on a robust, repeatable process. The process hub catches and manages data and analytics errors. Without the process hub, errors cause outages. There is no “single version of the truth” without a “single version of the process.”
This large enterprise has many products and brands with overlapping marketing campaigns. The process hub leverages the work product of several operational systems depicted on the right side of figure 1: a master data management (MDM) system, opt-out management from active marketing campaigns, a CRM and a marketing campaign management platform. These systems interface to the process hub through automated orchestrations. If data is sequestered in access-controlled data islands, the process hub can enable access. Operational systems may be configured with live orchestrated feeds flowing into a data lake under the control of business analysts and other self-service users.
Figure 1: A DataOps Process Hub
The process hub provides shared services and resources that feed the needs of the enterprise, divisions, product teams and specific projects while also providing feedback to operational systems. On the left side of figure 1, we see analytics produced for the various business development stakeholders. Despite the complexity, mission-critical analytics must be delivered error-free under intense deadline pressure.
The business analysts creating analytics use the process hub to calculate metrics, segment/filter lists, perform predictive modeling, “what if” analysis and other experimentation. Requirements continually change. Data is not static. Data workflows are not like IT infrastructure that is set up and then simply maintained. New analytics are added, and existing analytics are refined. The process hub resynchronizes master data, updates demographic information and reapplies suppressions, opt-outs and other attributes.
The Results
With the DataOps process hub in place, the BA team is significantly more productive. The process hub lowers the cost per additional data and analytics question. It centralizes the process so that it can be generalized and then applied to specific analytics questions in a fraction of the time it would otherwise require. An automated, governed, and reusable process decreases unplanned work and reduces the cycle time of change. It also minimizes the need for outside consultants who tend to rely upon heroism and tribal knowledge. A process hub is the key to creating an open, secure platform that enables those with access to efficiently update and skillfully manage the processes used to create data and analytics.
Bottom line, the process hub enabled the company to deliver higher quality analytics at a faster rate with fewer business analysts. They experienced:
- Delivery of new analytics in one-third the time;
- Data errors reduced by 80%;
- 5X increase in the number of sales and marketing users supported per data analyst; and
- Visualizations updated per week increased from 50 to 1500
The process hub helped this pharmaceutical company achieve its strategic goals by improving analytics quality, responsiveness, and efficiency. The process hub supports improved processes, automation of tools, and agile development of new analytics. With a process hub, the analytics team delivered value to users in record time, accelerating and magnifying their impact on top-line growth.
For more on process hubs, please see our post titled “DataOps For Business Analytics Teams.”
FAQ
What are the key points in this blog?
A large pharmaceutical Business Analytics team replaced manual data preparation with a DataOps process hub that turns integration, cleaning and transformation workflows into code. The team reported delivering new analytics in one-third the time, data errors reduced by 80%, a 5X increase in the number of sales and marketing users supported per data analyst, and visualizations updated per week rising from 50 to 1500.
What is a DataOps process hub?
A DataOps process hub instantiates the processes and workflows related to data and analytics — the checklists and tribal knowledge an analyst would otherwise run by hand — into code that is curated, managed and continuously improved. It automates, synthesizes and synchronizes the steps that integrate, clean and transform data, and it enforces the functional order of operations as the underlying data changes.
What results did the pharmaceutical business analytics team report?
Four numbers: new analytics delivered in one-third the time, data errors reduced by 80%, a 5X increase in the number of sales and marketing users supported per data analyst, and visualizations updated per week rising from 50 to 1500. The stated bottom line was higher quality analytics delivered at a faster rate with fewer analysts.
Why do business analysts end up doing their own data engineering?
Because the data platform they need does not arrive in time. IT hands over hundreds of raw tables and flat files, so analysts pull together the ten tables that matter to a particular question and do the data work themselves in Alteryx, SQL or Excel. At this company, business analysts spent 80% of their time on that work rather than on generating insight.
Which systems does a DataOps process hub connect to?
In this example the hub takes automated orchestrated feeds from a master data management system, opt-out management for active marketing campaigns, a CRM and a marketing campaign management platform, and publishes analytics back to business development stakeholders. Where data sits in access-controlled islands, the hub can enable access through live orchestrated feeds into a data lake the analysts control.
How does a process hub reduce data errors and rework?
By automating quality control, which is otherwise as slow as building the analytics — testing and validating took as long or longer than creating them. The hub catches and manages errors before they cause outages, and it resynchronizes master data, updates demographic information and reapplies suppressions and opt-outs. That cadence matters in commercial pharma, where prescriber lists and campaign data change constantly.
