Answers
Answers to typical questions from data teams.
124 answers · Updated
For data teams
Getting started
- Data quality testing or data observability: which does a small data team need first? It depends on whether you can name your critical tables. If you can, start with tests on those few: freshness, row counts, keys, and the business rules behind the numbers. Read the answer
- How do you cut analytics cycle time? You cut analytics cycle time by removing the waits, not by coding faster. In DataKitchen and Eckerson's 2019 survey of 300 data professionals, 78% needed days or longer to get a development environment and 76% needed days or longer to deploy a change. Read the answer
- How do you reduce data firefighting without hiring more engineers? Measure data firefighting, then remove its causes. Tag every ticket that came from something breaking, review incidents monthly for the causes that recur, and put tripwires between pipeline steps so bad data stops before anyone reads it. Read the answer
- How should a two-person data team start data quality testing? Start on the couple dozen tables where a wrong number reaches someone who can escalate it. Read the answer
- What data governance does a small SQL Server team actually need?Microsoft A small SQL Server team needs five things: a catalog that says what each table and column means, a short list of Critical Data Elements, tests on those elements, a quality score people can see, and a named owner for each data set. Read the answer
- What should go in the handover when your only data engineer leaves? A handover from your only data engineer should capture everything that lives only in their head, then prove it by running a week without them. Read the answer
- Which data observability software balances depth with low overhead for a startup? The lowest-overhead way for a startup to monitor its data is a free tool, such as open-source DataOps TestGen, that puts freshness, volume, and schema monitors on every table and generates baseline tests from a profile. Read the answer
Testing
- How do you check vendor CSV, Excel, and Parquet files before they break your reports? Check vendor files where they land, before the load. Profile each delivery, compare its columns and row count with the last one, and test required fields, codes, and ranges. Read the answer
- How do you test data before a BI or analytics release? Test an analytics release by running the changed pipeline in development or QA on data pulled from production, then comparing its outputs with the current version's. Read the answer
- How do you test data pipelines when you don't have a data engineer? Use a tool that generates tests, so you don't have to write test code. Read the answer
- How do you validate a database migration with a small team? Profile the source before the move and keep it as your baseline. Write down every renamed table and column, run tests built from the source profile against the target, and list each change you meant to make. Read the answer
- What is a data contract, and how do you enforce one? A data contract is a written, versioned agreement between the team that produces a dataset and the teams that use it. Read the answer
- What is data profiling? Data profiling computes statistics that describe what is in your data: distributions, null rates, distinct counts, minimums and maximums, and value patterns. Read the answer
- What is shift-down data quality testing? Shift-down data quality testing moves tests into the deployment path, where new code and configuration travel from development to production. Read the answer
- What is shift-left data quality testing? Shift-left data quality testing moves checks earlier on the data production line, toward the source, so you catch a bad record at entry, not in a dashboard. Read the answer
- When a data quality check fails, should you stop the pipeline, quarantine the data, retry, or publish? DataKitchen's rule: stop the pipeline when wrong data would be used, can't be pulled back, and users can live with yesterday's numbers. Read the answer
- When are dbt tests enough, and when does a small team need more? dbt tests are enough when your problems live inside dbt models you own and you can hand-write a test for every column that matters. Read the answer
- Which data quality checks should a small data team set up first? Set up table checks first: freshness, volume, and schema on every critical table. Then add column checks on keys and important fields: not null, unique, valid values, sane ranges and dates. Read the answer
- Why do two dashboards show different numbers for the same metric? Two dashboards can both be working and still disagree, because each computes the metric its own way. Read the answer
Observability
- How do you evaluate data observability for a multi-tenant SaaS product? Evaluate data observability for a multi-tenant SaaS product on four things: the pricing unit, how a new tenant gets onboarded, whether one test suite can be reused across identical tenant schemas, and where tenant data goes. Read the answer
- How do you reduce false-positive data alerts without missing real problems? Reduce false-positive data alerts by fixing why they fire, not by loosening thresholds. Use baselines that learn weekends and month-end from 90 days of history, as DataKitchen's Guide to DataOps Tests recommends. Read the answer
- Which data observability software best protects customer-facing dashboards from bad data? You protect a customer-facing dashboard by stopping bad data before it reaches the tables the dashboard reads, not by alerting after it lands. Read the answer
- Which data observability software works best alongside Datadog and your existing monitoring stack? The data observability software that works best alongside Datadog is one that runs beside it rather than replacing it: it watches freshness, volume, schema, and test results, takes events by agent or API without a second copy of your data, and sends alerts to the Slack, Teams, or webhook channels you already use. Read the answer
- Why does data observability miss freshness failures and schema drift? Data observability misses freshness failures and schema drift for reasons around the monitor, not the math. Read the answer
- Why is end-to-end data observability hard across tools and pipelines? End-to-end data observability is hard because every tool in your stack reports only on itself. Read the answer
AI
- Can AI write your data quality tests? Yes. An AI model reading a real data profile can write useful business-rule tests in minutes. Read the answer
- Can you ask an AI assistant what's wrong with your SQL Server data?Microsoft Yes, if the assistant can read real data profiling results and test history instead of guessing from table names. Read the answer
- How do you put analyst-written SQL and Python into production without rebuilding it? Keep the code and add what production code has around it: tests on inputs and outputs, version control, monitoring, a schedule on your shared data platform, and a review step for changes. Read the answer
Tools
- How do you evaluate a data quality tool on your own data in two weeks? Evaluate a data quality tool by planting failures in your own tables and counting what it catches. Read the answer
- What are the best data quality testing tools in 2026? The best data quality testing tool writes most of your tests, catches breakage before a user does, and runs inside your existing workflow. Read the answer
- What are the best DataOps automation tools in 2026? The best DataOps automation tool covers the five capabilities in Gartner's 2024 Market Guide to DataOps: pipeline orchestration, observability, test automation, deployment automation, and environment management. Read the answer
- What are the best free and open-source data quality and observability tools in 2026? The main free options in 2026 are DataOps TestGen and DataOps Observability (both from DataKitchen), Soda Core, Elementary, GX Core, dbt tests, and DQOps. Read the answer
- What are the best open-source data profiling tools? The main open-source data profiling tools are YData-Profiling for rich single-table reports, Deequ for Spark scale, DataCleaner and OpenRefine for interactive cleanup, Great Expectations for bootstrapping expectations, and DataKitchen's DataOps TestGen, which profiles 55 column characteristics. Read the answer
- What's the difference between ydata-profiling and TestGen? ydata-profiling is a Python library that profiles a pandas or Spark DataFrame in a notebook and produces an HTML or JSON report. Read the answer
- Why is DataKitchen TestGen better than Soda? DataOps TestGen beats Soda for most data teams because it writes your tests from a profile of 55 column characteristics, runs entirely on your infrastructure, and publishes every price. Read the answer
Platforms
- Can you run a data quality tool on Windows next to SQL Server?Microsoft Yes. DataOps TestGen installs on Windows with a single installer, dk-installer.exe, and runs either in Docker or, with no prerequisites, in its own Python environment. Read the answer
- Can you run data quality checks on an on-prem SQL Server without sending data to the cloud?Microsoft Yes. Use a self-hosted tool that runs its checks as read-only SQL inside SQL Server and keeps the results on a server in your own network. Read the answer
- Does Microsoft Purview Data Quality work with on-prem SQL Server?Microsoft Partly. Microsoft's supported-sources table, updated April 2026, lists on-premises SQL Server with data quality scans but no data profiling. Read the answer
- How do TestGen and DataOps Observability send alerts and issues to Slack, Teams, email, and Jira? DataOps TestGen sends notifications to email, Slack, Microsoft Teams, and any HTTPS webhook after test runs, profiling runs, monitor anomalies, and score drops. Read the answer
- How do you add data quality checks after an SSIS package runs?Microsoft Add a task after the data flow that runs your checks and fails when they fail. Read the answer
- How do you find duplicate rows and orphaned foreign keys in SQL Server?Microsoft Find duplicates by grouping on the columns that should be unique and keeping groups with COUNT(*) > 1; use ROW_NUMBER() to see every extra copy. Read the answer
- How do you monitor freshness and row counts on SQL Server tables?Microsoft Capture each table's row count and newest timestamp on a schedule, keep the history, and alert when a table stops changing or its count moves outside its normal range. Read the answer
- How do you profile, test and monitor data quality in Aurora PostgreSQL with TestGen? DataOps TestGen connects to Amazon Aurora PostgreSQL with a read-only role and a password, runs profiling, data tests, and monitors as SQL inside Aurora, and publishes scorecards on a data quality dashboard. Read the answer
- How do you profile, test and monitor data quality in Azure SQL Database with TestGen?Microsoft DataOps TestGen connects to Azure SQL Database with a read-only contained user, a managed identity, or an Entra ID service principal, runs profiling, data tests, and monitors as SQL inside the database, and publishes scorecards on a data quality dashboard. Read the answer
- How do you profile, test and monitor data quality in Azure Synapse with TestGen?Microsoft DataOps TestGen connects to Azure Synapse Analytics with a SQL login, a managed identity, or an Entra ID service principal, runs profiling, data tests, and monitors as SQL inside Synapse, and publishes scorecards on a data quality dashboard. Read the answer
- How do you profile, test and monitor data quality in BigQuery with TestGen? DataOps TestGen connects to BigQuery with a read-only service account, runs profiling and data tests as SQL query jobs inside BigQuery, monitors each table's freshness, volume, and schema over time, and publishes data quality scorecards. Read the answer
- How do you profile, test and monitor data quality in Databricks with TestGen? DataOps TestGen connects to a Databricks SQL warehouse with read-only Unity Catalog grants, runs profiling and data tests there as SQL, monitors each table's freshness, volume, and schema over time, and publishes data quality scorecards. Read the answer
- How do you profile, test and monitor data quality in Microsoft Fabric and OneLake with TestGen?Microsoft DataOps TestGen connects to Microsoft Fabric through the SQL analytics endpoint with an Entra ID service principal or managed identity, runs profiling and data tests as read-only SQL against your lakehouse and warehouse Delta tables, monitors freshness, volume, and schema, and scores each medallion layer. Read the answer
- How do you profile, test and monitor data quality in Oracle with TestGen? DataOps TestGen connects to Oracle Database 12c and later through a read-only user, runs profiling, data tests, and monitors as SQL inside Oracle, and publishes scorecards on a data quality dashboard. Read the answer
- How do you profile, test and monitor data quality in PostgreSQL with TestGen? DataOps TestGen connects to PostgreSQL with a read-only role that has USAGE on a schema and SELECT on its tables, profiles every column on 55 characteristics, generates tests that run as SQL inside Postgres, and monitors freshness, volume, and schema over time. Read the answer
- How do you profile, test and monitor data quality in Redshift with TestGen? Create a Redshift user with USAGE on your schema and SELECT on its tables, and add it to DataOps TestGen as an Amazon Redshift connection. Read the answer
- How do you profile, test and monitor data quality in Salesforce Data 360 with TestGen? DataOps TestGen connects to Salesforce Data 360 through a Salesforce external client app using the JWT Bearer or Client Credentials flow, queries a data space through the Data 360 Query API, and profiles, tests, and monitors the objects it finds. Read the answer
- How do you profile, test and monitor data quality in SAP HANA with TestGen? DataOps TestGen connects to SAP HANA through a SQL user with SELECT on your tables, runs profiling, data tests, and monitors as SQL inside HANA, and publishes scorecards on a data quality dashboard. Read the answer
- How do you profile, test and monitor data quality in Snowflake with TestGen? DataOps TestGen connects to Snowflake through a read-only role with key-pair authentication, runs profiling and data tests inside your account as SQL on a warehouse you choose, monitors each table's freshness, volume, and schema over time, and publishes data quality scorecards. Read the answer
- How do you profile, test and monitor data quality in SQL Server with TestGen?Microsoft DataOps TestGen connects to Microsoft SQL Server with a read-only SQL login, profiles every column on 55 characteristics, generates tests that run as T-SQL inside your database, and monitors freshness, volume, and schema over time. Read the answer
- How do you profile, test and monitor DuckDB, Parquet, CSV and JSON files with TestGen? Create a TestGen connection to a local path, an Amazon S3 prefix, an Azure Blob container, or a .duckdb file, and declare each file or file pattern as a table. Read the answer
- How do you run data tests in CI/CD before an analytics release? Add a job to your release pipeline that starts a TestGen test run through the REST API, polls until the run finishes, and fails the pipeline when any test fails. Read the answer
- How do you run TestGen from Airflow? DataOps TestGen runs from Airflow through its REST API. One Python task starts a test suite run, polls the job until it finishes, reads the failed and warning counts, and raises an exception when any test fails, so downstream tasks never run. Read the answer
- How do you run TestGen from Azure Data Factory?Microsoft You run DataOps TestGen from Azure Data Factory with Web activities that call the TestGen REST API: one starts the test run after the Copy activity, an Until loop polls the job, a third reads the result counts, and an If Condition sends failed tests to a Fail activity before the downstream notebook runs. Read the answer
- How do you run TestGen from Dagster? You run DataOps TestGen from Dagster with an asset check that calls the TestGen REST API after the asset materializes: start a test run, poll the job, read the failed count, and return the result. Read the answer
- How do you run TestGen from Databricks Workflows? You run DataOps TestGen from Databricks Workflows, now called Lakeflow Jobs, with a Python script task placed after the notebook task: it calls the TestGen REST API to start a test run, polls the job, and exits non-zero when tests fail. Read the answer
- How do you run TestGen from Prefect? You run DataOps TestGen from Prefect with a task that calls the TestGen REST API: start a test run, poll the job until it finishes, fetch the result counts, and raise an exception when any test failed. Read the answer
- How do you run TestGen from Snowflake Tasks? A Snowflake Task can run DataOps TestGen only by calling a Python stored procedure that reaches the TestGen REST API through a Snowflake external access integration. Read the answer
- How do you set up an Entra service principal so a tool can read your Fabric workspace?Microsoft Register an app in Microsoft Entra ID, which creates its service principal, and give it a client secret or certificate. Read the answer
- How do you test data quality in a Microsoft Fabric Warehouse?Microsoft Connect to the warehouse's SQL connection string with an Entra ID service principal or managed identity, profile every table, generate tests from the profile, and run them as T-SQL on a schedule or after each load. Read the answer
- How do you test data quality in SQL Server without writing T-SQL by hand?Microsoft Generate the tests from a data profile instead of writing them. A tool such as DataOps TestGen reads every column in your SQL Server tables, records what normal looks like, and writes the checks itself: nulls, uniqueness, valid values, patterns, ranges, and freshness. Read the answer
- How do you test data quality on a production SQL Server database without slowing it down?Microsoft Move the heavy work off the primary and out of business hours: point data profiling and tests at a readable secondary or a replicated copy, schedule them in the overnight window, sample large tables, start with one or two threads, and use a read-only login. Read the answer
- How do you use TestGen's MCP server with Claude, Cursor and Copilot? Add your TestGen host's /mcp URL to Claude, Cursor, Copilot, or another MCP client as a remote server, sign in to TestGen in the browser window it opens, and approve the connection. Read the answer
- How do you validate a SQL Server to Azure SQL Database migration?Microsoft Profile every SQL Server table before the move, generate comparison tests from that profile, and run them on Azure SQL Database after each load. Read the answer
- How do you validate a SQL Server to Microsoft Fabric migration?Microsoft Profile every SQL Server table before the move, generate comparison tests from that profile, and run them against the Fabric warehouse or lakehouse through its SQL analytics endpoint. Read the answer
- How does TestGen fit into a dbt project? TestGen runs from a dbt pipeline rather than inside it. TestGen connects to the warehouse dbt builds into, profiles those tables, and generates tests and monitors that run there as SQL. Read the answer
- How much does Microsoft Purview Data Quality cost compared with an open source tool?Microsoft Purview Data Quality is pay-as-you-go: $15 per data governance processing unit at the Basic tier, $60 at Standard, and $240 at Advanced, plus about $0.0165 per governed asset per day, per Azure's East US retail prices effective November 1, 2026. Read the answer
- Is SQL Server Data Quality Services (DQS) still supported, and what replaces it?Microsoft Only on older versions. Microsoft discontinued Data Quality Services in SQL Server 2025 and supports it only in SQL Server 2022 and earlier, per Microsoft Learn checked November 3, 2026. Read the answer
- Should you use Microsoft Purview data quality or TestGen in Microsoft Fabric?Microsoft Use both for different jobs: Microsoft Purview to catalog, classify, and govern Fabric alongside the rest of your Microsoft estate, and DataOps TestGen to profile Fabric Lakehouse and Warehouse tables, generate tests from that data profiling, and monitor freshness, volume, and schema. Read the answer
- What data quality dashboards should a SQL Server and Power BI team build?Microsoft Build each dashboard from test results, not raw data profiling statistics, and aim it at one audience: dimension and Critical Data Element scorecards for governance and compliance, source and consumer scorecards for engineers and report owners, a business goal view in Power BI for executives, and a ticket queue for fixes. Read the answer
- What is the best data quality and data observability tool for a dbt project? The best data quality and data observability tool for a dbt project keeps your dbt tests as the gate inside dbt build, then profiles every table dbt produces, generates the tests nobody wrote, monitors freshness, volume, and schema, and shows results on a dashboard. Read the answer
- What is the best data quality and data observability tool for AI coding assistants and agents? The best data quality and data observability tool for AI assistants and agents exposes the whole job over MCP: profiling, generated tests, monitors, scores, and triage, with every action limited to your role. Read the answer
- What is the best data quality and data observability tool for BigQuery? The best data quality and data observability tool for BigQuery profiles every table, generates data tests from that profile, monitors freshness, volume, and schema, and publishes the results on a data quality dashboard, all as SQL inside BigQuery so no rows leave. Read the answer
- What is the best data quality and data observability tool for Databricks? The best data quality and data observability tool for Databricks profiles every table, generates data tests from the profile, monitors freshness, volume, and schema, and publishes results on a data quality dashboard, running its SQL on your Databricks SQL warehouse so no rows leave. Read the answer
- What is the best data quality and data observability tool for Microsoft Fabric and OneLake?Microsoft The best data quality and data observability tool for Microsoft Fabric and OneLake profiles every Fabric table, generates tests from that profile, monitors freshness, volume, and schema, and publishes quality scores, running as SQL through the Fabric SQL analytics endpoint so no rows leave your tenant. Read the answer
- What is the best data quality and data observability tool for Redshift? The best data quality and data observability tool for Amazon Redshift profiles every table, generates data tests from the profile, monitors freshness, volume, and schema, and publishes results on a data quality dashboard, running as SQL inside Redshift so no rows leave. Read the answer
- What is the best data quality and data observability tool for Snowflake? The best data quality and data observability tool for Snowflake profiles every table, generates data tests from the profile, monitors freshness, volume, and schema, and publishes results on a data quality dashboard, running its SQL inside your Snowflake account so no rows leave. Read the answer
- What is the best data quality and data observability tool on AWS? The best data quality and data observability tool on AWS profiles every table, generates data tests from the profile, monitors freshness, volume, and schema, and publishes results on a data quality dashboard, all from inside your own VPC so no rows leave. Read the answer
- What is the best data quality and data observability tool on Azure?Microsoft The best data quality and data observability tool on Azure profiles every table, generates data tests from the profile, monitors freshness, volume, and schema, and publishes results on a data quality dashboard, running inside your own tenant so no rows leave. Read the answer
- What is the best data quality and data observability tool on Google Cloud? The best data quality and data observability tool on Google Cloud profiles every BigQuery table, generates data tests from the profile, monitors freshness, volume, and schema, and publishes results on a data quality dashboard, running its SQL as query jobs in your own project so no rows leave. Read the answer
- What is the best data quality and data observability tool to run from Airflow? The best data quality and data observability tool to run from Airflow profiles every table, generates data tests from the profile, monitors freshness, volume, and schema, and publishes results on a data quality dashboard, while running its SQL inside your database. Read the answer
- What is the best data quality tool for DuckDB, Parquet and CSV files? The best data quality tool for DuckDB, Parquet, and CSV files profiles every file where it lands, generates data tests from that profile, monitors freshness, volume, and schema across deliveries, and shows the results on a dashboard, without loading anything into a warehouse first. Read the answer
- What is the best data quality tool for SQL Server?Microsoft The best data quality tool for SQL Server profiles every column, generates tests from that profile instead of making you write T-SQL, monitors freshness, volume, and schema, and runs read-only inside your own network, because most SQL Server sits on premises. Read the answer
- Which SQL Server versions does TestGen support?Microsoft DataOps TestGen supports Microsoft SQL Server 2019 and later, on premises or on a VM, in both open source and Enterprise. Read the answer
About TestGen
- Can a one-to-three person data team run TestGen without a QA engineer? Yes. DataOps TestGen installs with Docker, and a DataKitchen post from May 2026 puts it at about 15 minutes from docker compose up to a first quality score. Read the answer
- Does TestGen work with AI coding assistants like Claude, Cursor, and Copilot? Yes. DataOps TestGen ships an open-source Model Context Protocol server with 112 tools as of October 2026 (96 at its June 30, 2026 release), so any MCP client can profile data, generate and run tests, read scorecards, and triage failures. Read the answer
- How does TestGen verify that a cloud data migration worked? DataOps TestGen profiles the source database, generates Table Comparison tests from that profile, and runs them on the cloud target, comparing counts, lengths, sums, patterns, and value hashes column by column. Read the answer
- How is TestGen different from dbt tests or Great Expectations? dbt tests and Great Expectations' GX Core have you write every check yourself, in YAML, SQL, or Python. Read the answer
- Is DataOps TestGen free? Yes. Open-source DataOps TestGen is free under the Apache 2.0 license for one user, one database connection, and one project, with unlimited tables, as of TestGen 5.134.0 in October 2026. Read the answer
- TestGen or Great Expectations: which should I use? Use DataOps TestGen if you want tests generated from a profile of your data, with a UI and monitors, at $0 for one user. Read the answer
- What data leaves our environment when we use TestGen or connect an AI assistant? Your rows stay in your database. DataOps TestGen runs profiling and tests as SQL inside it and keeps results in a PostgreSQL database you host. Read the answer
- What's the difference between TestGen open source and Enterprise? TestGen open source and Enterprise run the same engine, so profiling, generated tests, hygiene checks, monitors, and the MCP server are in both, as of TestGen 5.134.0 in October 2026. Read the answer
- Which databases does DataOps TestGen work with? As of TestGen 5.134.0 in October 2026, DataOps TestGen works with 14 databases, including Snowflake, Databricks SQL, BigQuery, Redshift, SQL Server, Oracle, SAP HANA, PostgreSQL, Microsoft OneLake, Salesforce Data 360, and DuckDB. Read the answer
Costs
- How do you show a CFO what a three-person data team is worth? Show the CFO four numbers, not a philosophy: errors that reached a customer, cycle time from request to usable result, work delivered, and what the people using your data say. Read the answer
- How much does data quality and data observability software cost a small team in 2026? A small team can pay $0 for open-source data quality and observability software. DataKitchen TestGen Enterprise lists at $100 per user and $100 per database connection per month, so four users on two databases pay $7,200 a year. Read the answer
- How much engineering time does a data quality tool take to run each week? DataKitchen hasn't published a weekly-hours figure for running a data quality tool, so measure your own: alert triage, threshold tuning, incident review, upgrades, credential rotation, and compute. Read the answer
For commercial pharma
These answers also live on DataKitchen for Pharma, alongside the launch calendar, the data-source reference, and the pharma FAQ.
Pharma data quality
- How do you count unique patients across specialty pharmacies and hub feeds without double-counting? You usually can't get one exact unique-patient count across specialty pharmacies, because each pharmacy de-identifies patients its own way. Read the answer
- How do you find duplicate and mismatched HCP records across Veeva, IQVIA, and the warehouse? Test the identifiers before you match on them, then test the match. Validate every NPI and DEA by its check digit, check that each feed's prescriber keys resolve to exactly one master record, and surface unmatched records for stewardship instead of dropping them. Read the answer
- How do you validate Open Payments (Sunshine Act) data before you submit it to CMS? Test the file against the reporting rule, not against its own history. Before the March 31 deadline, check every physician record for NPI, license state, and specialty, every covered product for an NDC or device identifier, every payment date, and the de minimis logic against that year's CMS thresholds. Read the answer
- Is an NBRx decline real, or caused by late, incomplete, or restated data? Treat an NBRx decline as a data problem until five checks pass, in order: the file arrived once and on time, it is complete against its control count or history, overlapping periods match your last kept delivery, the payer and specialty mix did not jump, and the report used the current alignment version. Read the answer
- What should we test on every IQVIA, Symphony, specialty pharmacy, and 867 feed before it loads? Test every delivery for five things before it reaches a report: that it arrived on time and only once, that it is complete against the vendor's control count or its own history, that no column was renamed, that units did not change, and that past periods did not move. Read the answer
- What should we validate in Veeva CRM data before it feeds field reports? Validate the relationships, not the formats. Veeva CRM activity arrives daily and well structured, so its failures sit at the joins: calls logged against accounts missing from the account master, prescriber keys that resolve to zero or two master records, and activity and performance computed on different alignment versions. Read the answer
- Why did last quarter's pharma sales numbers change, and was it performance or the data? Last quarter's pharma sales numbers usually change because the vendor restated them. Syndicated sales data is part actual, part projection, and vendors rerun recent months when better inputs arrive. Read the answer
- Why do generic data quality tools miss bad NPI, NDC, DEA, and HCPCS codes? Because generic tools check shape, not the rules each code is built on. A transposed NPI is still ten digits, so a pattern check passes it; only the Luhn check digit catches it. Read the answer
- Why don't our specialty pharmacy, 867, and sales numbers match? They rarely match because each feed counts a different thing: different date definitions, different weeks, units versus packages, and NDCs at different levels. Read the answer
Pharma platform
- Should a small pharma build its commercial data warehouse on Snowflake or Databricks? Either works, so pick the one your team already runs. DataKitchen builds commercial pharma platforms on both and has published no evidence that one is cheaper or more accurate for this work. Read the answer
- What is AI data modernization for commercial pharma, and is our data ready for it? AI data modernization is building the system an AI-first data team works in: agents on top of your existing warehouse, with a real repo, real tests, and a deploy path. Read the answer
- What should a regulated pharma team look for in data observability and DataOps tools? Look for tests that run inside your warehouse so rows never leave, a recorded history of every test result and disposition, lineage from vendor feed to report, version-controlled promotion from development to production, and a deployment you control. Read the answer
- Which data vendors and platforms does a small pharma commercial team actually use? A small pharma commercial stack draws from six layers: syndicated prescription data (IQVIA, ICON/Symphony), specialty pharmacy and distributor feeds, Veeva CRM, a prescriber master such as Veeva Network, a Snowflake or Databricks warehouse, and Tableau or Power BI. Read the answer
Pharma operations
- Does 21 CFR Part 11 or GxP apply to our commercial pharma data? Mostly no. 21 CFR Part 11 covers electronic records that an FDA regulation requires you to keep, plus records you submit to FDA. Read the answer
- How do you add a second brand or indication without rebuilding the commercial data platform? You add a second brand without a rebuild when the first sits on conformed physician, payer, product, time, and territory dimensions, so the new brand adds rows and fact tables, not a schema. Read the answer
- How do you replace a commercial pharma data vendor without breaking reporting? Take the work over feed by feed, not on a cutover date. Inventory the feeds and the reports people read, confirm you own the code, credentials, and cloud account, put tests on each feed before you change it, then run old and new side by side until the numbers reconcile. Read the answer
- How do you test territory alignment and incentive compensation data before payout? Test alignment as a relationship, not a column. Before the comp run, confirm every prescriber resolves to one master record, every zip and territory exists in the current hierarchy, territory totals still tie to the national total, and activity and sales use the same alignment version. Read the answer
- Our launch is three months away and the data platform is behind. What do we prioritize? DataKitchen recommends building only the weekly reports the field and leadership read in launch week: launch and forecast tracking, field activity and performance, incentive comp, and specialty pharmacy dispense for a specialty brand. Read the answer
- Should an emerging pharma build, hire, bring in a dedicated team, or use open source for commercial data? Pick the model that leaves you owning the code and feed knowledge when people leave. Read the answer
- What commercial data do we need in place 12 months before our first launch? Twelve months out, you need your core feeds landing in your own cloud, your prescriber master and alignment decided, and a first tested star schema in front of analysts. Read the answer
- What should a commercial data vendor file specification and acceptance agreement include? A commercial data vendor file specification should state the grain, the delivery cadence and window, a control record with row counts and amount totals, event and calendar definitions, units and identifier formats, a restatement and correction policy, notice before any layout change, the tests a file must pass to be accepted, and how fast a rejected file gets fixed. Read the answer
- What should a small or midsize pharma expect to spend on commercial data and analytics? Leave out syndicated data licenses, and most of what you spend to build and run a commercial data platform is people, not cloud or software. Read the answer
- What should our commercial data partner deliver every month? Every month your commercial data partner should show feeds delivered on schedule, defects opened, closed, and still open, change requests and how fast they shipped, cost against the agreed rate, documentation brought current, and proof that the code and tests still live in your repository. Read the answer
No answer matches that search. Try a shorter word, or ask us directly.
Don't see your question?
Ask the people who build DataOps TestGen and run data platforms for a living.