What is the best data quality tool for DuckDB, Parquet and CSV files?

The best data quality tool for DuckDB, Parquet, and CSV files profiles every file where it lands, generates data tests from that profile, monitors freshness, volume, and schema across deliveries, and shows the results on a dashboard, without loading anything into a warehouse first. DuckDB itself has no data quality checks, so the tool supplies them.

By Chris Bergh Published October 21, 2026

Key points

DuckDB has no built-in data quality tests; its SUMMARIZE command returns per-column statistics such as min, max, approximate distinct count, quartiles, and null percentage, with no tests, history, or alerts.
Since TestGen 5.134.0, released October 14, 2026, DataOps TestGen profiles CSV, Parquet, and JSON files and DuckDB database files in place, on a local path, in Amazon S3, or in Azure Blob Storage, through an embedded DuckDB 1.5.5 engine.
Great Expectations' GX Core, Soda Core, Pandera, and dbt with the dbt-duckdb adapter can all check file data, but in each one you write the checks yourself; TestGen generates tests from 34 test types out of a profile.
As of October 2026, Soda Core is licensed under the Elastic License 2.0, while GX Core and dbt-duckdb are Apache 2.0 and Pandera is MIT.
Open-source TestGen is free for one user, one connection, and one project; TestGen Enterprise is $100 per user per month plus $100 per database connection per month.

This page compares tools. For the delivery-by-delivery procedure, see how to check vendor files. For TestGen setup, see the TestGen files walkthrough.

The usual buyer here is a small team checking vendor files before the warehouse load. The file lands, a loader accepts it, and the problem shows up in a report a week later.

What DuckDB gives you natively

DuckDB has no data quality tests. Its SUMMARIZE command returns min, max, approximate distinct count, average, standard deviation, quartiles, count, and null percentage per column (checked October 2026). That’s a good first look. It keeps no history, runs no tests, and alerts nobody.

Tools worth considering

Vendor claims are from each vendor’s own docs and pricing pages, checked October 21, 2026.

Tool	Profiling	Generated data tests	Observability monitors	Data quality dashboards	Reads files where they sit	Open source	Price basis
DataOps TestGen	55 column characteristics	Yes, 34 test types	Freshness, volume, schema	Scorecards	Local path, S3, Azure Blob	Apache 2.0	Free for one connection; Enterprise per user and connection
GX Core	Yes	No, you write each Expectation	Rules you write	No	Local, S3, Azure Blob, GCS, through pandas or Spark	Apache 2.0	Free
Soda Core	In Soda Cloud	No, you write contracts	Checks you write	In Soda Cloud	A `.duckdb` or Parquet file	Elastic License 2.0	Free, Team $750 a month, Enterprise custom
Pandera	Schema inference for pandas	Inferred schema for pandas	No	No	No, validates a DataFrame you load	MIT	Free
dbt with dbt-duckdb	No	No, four built-in generic tests you attach	Source freshness	No	Yes, sources point at files, including S3	Apache 2.0	Free

The GX Core profiling and generated-test entries come from DataKitchen’s GX OSS comparison, reviewed June 11, 2026.

Who should pick what

Pick Pandera or GX Core if your files already pass through Python and you want checks in code review. Pandera is the lightest for a pandas or Polars pipeline. Pick dbt-duckdb if you transform the files in dbt anyway. Pick Soda if you want contracts in YAML and a hosted UI, and you accept a source-available license. If partners send Excel, CsvPath Framework lists Excel validation; TestGen needs a CSV first.

Pick TestGen if nobody has time to write the checks. It profiles the file, flags PII and hygiene issues, generates the tests, and keeps history across deliveries, so a renamed column shows up in the next profile. Failed tests are the evidence for your data contract conversation with the vendor. DataKitchen’s NYC taxi run loaded one 48 MB Parquet file into Postgres before TestGen saw a row. TestGen 5.134.0 removed that step.

The trade-off: file runs use TestGen’s own memory and disk, one run at a time by default. Size the machine to your largest file.

The core profiling, testing, and monitoring engine is free and open source. Enterprise adds team access, security, Commercial Pharma tests, source-data controls for AI, and support. Open source covers one user, one connection, one project, and unlimited tables. Enterprise is $100 per user per month plus $100 per database connection per month. For the database list, see which databases TestGen supports.

Documentation: Connect your Data, Access Requirements, Manage Table Groups, Size TestGen for DuckDB and File Data Sources

Frequently asked questions

Short answers to the follow-up questions people ask next.

Can I use TestGen alongside DuckDB's own SQL and SUMMARIZE?

Yes. TestGen opens a .duckdb file read-only, so you can keep querying the same file in DuckDB and running SUMMARIZE for a quick look. TestGen can't open the file while another process holds it open for writing, because DuckDB allows one writer at a time. SUMMARIZE gives you one snapshot of column statistics; TestGen keeps profile history, generates tests, and runs monitors across every delivery.

What does a file data quality tool cost for a small team?

It can cost nothing. Open-source TestGen is free for one user, one connection, and one project, with unlimited tables, and a file connection counts as that one connection. GX Core, Pandera, and dbt-duckdb are free libraries. Soda Cloud listed a Free plan at $0 and a Team plan at $750 a month in October 2026. TestGen Enterprise is $100 per user per month plus $100 per database connection per month.

Does any data leave my environment?

No. TestGen reads your files inside its embedded DuckDB engine, on infrastructure you run, and stores computed results and metadata, such as statistics, test outcomes, and monitor history, in its own database. Profiling keeps each column's most frequent values and minimum and maximum text values for display, and a large run can spill a temporary copy to TestGen's disk until the run ends. TestGen never modifies your files.

Does TestGen read Excel files?

No. As of TestGen 5.134.0 in October 2026, TestGen reads CSV, Parquet, and JSON files and DuckDB database files, not Excel workbooks. Save each sheet as CSV first. If your partners only send Excel and you want the tool to read it directly, CsvPath Framework lists Excel validation, with rules you configure yourself.

Which file format should I ask a vendor for?

Ask for Parquet. Parquet stores its own column types, so TestGen reads them exactly. CSV and JSON carry values without declared types, so TestGen infers each type from the values, and a ZIP code column of all digits comes back as a number. If the vendor can only send CSV, check the inferred types in the table preview before you profile.

Try it on your own data

DataOps TestGen is free and open source. Profile a table and generate its tests this afternoon.

Install Open Source TestGen Request a Demo