The weekly status meeting tells you what the team did. The monthly scorecard tells you whether it’s working. Most vendor reports confuse the two and send a page of activity.
The cadence underneath
DataKitchen’s commercial pharma engagements run on three rhythms: a daily scrum and ticket triage in Jira, a weekly status with your analytics lead on work completed, in progress, and planned, and a quarterly business review with leadership on timelines and metrics for bugs, tests, and datasets. Documentation lives in a shared wiki. Time is tracked and reported.
The monthly scorecard sits between the weekly and the quarterly. It’s for the person who signs the invoice.
A sample scorecard
| Measure | Where the number comes from | Illustrative month |
|---|---|---|
| Delivery reliability: feeds landed on schedule | Freshness tests on every feed | 37 of 38 refreshes on time; one specialty pharmacy file a day late |
| Defects that reached a person: opened, closed, still open | Jira, tagged by feed | 2 opened, 3 closed, 1 open |
| Test failures caught before publishing | Test run history | 14, all stopped in staging |
| Change requests: shipped, median days to ship | Jira | 9 shipped, median 3 days |
| New datasets added | Git and the data dictionary | 1, documented on arrival |
| Cost: rate and hours | Invoice and time tracking | Flat rate unchanged; hours down 6% |
| Documentation current | Generated data dictionary vs datasets in production | All production datasets documented |
| Ownership | Your cloud account and your repository | Code, tests, and credentials all in your accounts |
Every value in the third column is illustrative, made up to show the shape of the report. None is a DataKitchen customer result.
The split between defects and test failures is the one to insist on. A test failure is the system working: a bad file stopped before anyone saw it. A defect is a wrong number that reached a person. A good month has plenty of the first and few of the second.
What good looks like, from real operations
Celgene is the reference point DataKitchen publishes. A commercial insights leader there reduced errors to about one per quarter and held it for years. The Otezla launch ran seven data engineers and 10 to 12 analysts over hundreds of datasets with very few errors or missed deadlines.
The counter-example is just as useful. At one rare-disease biotech, the prior vendors’ issue log recorded 30 issues from May 2024 and 43 in 2025, seven still open on transfer day. Nobody needed a scorecard to know it was going badly. They needed one to see it six months earlier.
Process metrics come from the platform
Don’t accept numbers somebody typed into a slide. DataOps Automation tracks test coverage, error rates, deployment cycle time, and features shipped, which is what a quarterly review runs on. Test results come from the test history. Change turnaround comes from Jira. Ownership comes from your own cloud console.
If a partner can’t produce a measure from a system, that’s a finding.
Ownership, every month
This line looks redundant until the month it isn’t. The warehouse sits in your cloud account. Every transform, test, and schema lives in your repository from the first sprint. The data quality software underneath is Apache 2.0. Handover terms decide whether a platform is an asset or a dependency, and checking them monthly is how a transfer stays a normal day.
Watch the hours, too. That biotech’s run team kept one person on automation and improvement, and monthly hours fell while the second indication’s work was added. Falling hours while scope grows means work is being deleted. Hours cut by working faster come back the week a new feed arrives.
If you’re replacing a partner rather than grading one, take it over feed by feed.