Vendor files break reports in a predictable way. The file arrives, a loader accepts it, three transforms carry it forward, and the problem surfaces in a report a week later. By then the vendor has sent four more files.
So check the file before the load, at the place it lands.
What TestGen reads, and what it doesn’t
DataOps TestGen 5.134.0, released October 14, 2026 in both the open-source and Enterprise editions, profiles, tests, and monitors files with no database in front of them. An embedded DuckDB 1.5.5 engine reads them inside TestGen.
| Format | Supported directly | Notes |
|---|---|---|
| CSV | Yes | Column types inferred from values |
| Parquet | Yes | Types come from the file |
| JSON | Yes | Column types inferred from values |
| DuckDB database file | Yes | Opened read-only |
| Excel (.xlsx) | No | Save as CSV first |
| Avro, ORC | Not as files | Testable since October 2025 when Redshift Spectrum or Snowflake external tables expose them |
Files can sit on a local path, in Amazon S3, or in Azure Blob Storage. Before 5.134.0, a file had to sit behind a database, loaded into it or exposed as an external table. The NYC taxi test started from one 48 MB Parquet file that went into Postgres before TestGen saw a row.
A worked setup for daily vendor drops
Say a vendor drops claims_YYYYMMDD.csv into an S3 prefix every morning.
- Create an Amazon S3 connection with a base URI and read-only access (
s3:ListBucketands3:GetObject). - Declare the table as a pattern.
claims/*.csvmakes every delivery rows of one table. TestGen reads the union of their columns, so a field the vendor adds next Tuesday appears in the next profile, at the file. - Preview before profiling, and confirm the inferred types. A text column of all-digit ZIP codes comes back as a number, and a JSON file declared as CSV gives you nonsense columns, not an error.
- Profile and generate tests. You get the same profile, hygiene issues, generated tests, and volume and schema monitors a warehouse table gets.
- Add tests for the vendor’s promises: required fields present, codes on the agreed list, dates in range, row count near the usual delivery. Those are the terms of a data contract, and a failing test is the evidence you send back.
Limits worth knowing before you start
File runs use the memory and disk of the machine running TestGen, sized to the largest table you read. Only one runs at a time by default; raise TG_MAX_CONCURRENT_IN_PROCESS_RUNS once you’ve given TestGen the memory. Most installs run in Docker, and a local path is resolved inside the container, so mount the folder read-only first and enter the path the container sees.
Excel is the gap. If your vendor sends workbooks, you need a step that saves each sheet as CSV before TestGen sees it, and that step is one more thing to test. Ask for CSV or Parquet in the contract. Parquet keeps the types.
And keep the credentials read-only. TestGen never writes to your files, but a custom test runs SQL you write, and TestGen can’t stop that SQL from writing if the credentials allow it.