You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/04/09 14:14:26 UTC
[PR] fix: generate integer keys instead of floats in TPC-DS data [datafusion-benchmarks]
Dandandan opened a new pull request, #31:
URL: https://github.com/apache/datafusion-benchmarks/pull/31
## Summary
- Fix `tpcdsgen.py` to cast DataFrame to the correct PyArrow target schema before writing Parquet, ensuring nullable integer columns (surrogate keys, quantities) are stored as `int32` instead of `double`/`float64`
- Fix trailing pipe detection to work with both old and new dsdgen versions (v4.0.0 no longer adds a trailing `|` as field terminator)
- Regenerate all SF1 parquet data with dsdgen v4.0.0 and zstd compression
## Background
The previous data had ~100 columns across 15 tables stored as `double` that should be `int32` (e.g. `ss_sold_date_sk`, `cs_bill_customer_sk`, `ss_quantity`). This happened because DataFusion's CSV reader promotes nullable `int32` columns to `float64` when there are null values. The fix collects the result into a PyArrow table and casts to the declared schema before writing.
## Test plan
- [x] Verified all 24 generated parquet files have zero `double` columns
- [x] Verified null counts match between old and new data
- [x] All files under GitHub's 100MB file size limit
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: generate integer keys instead of floats in TPC-DS data [datafusion-benchmarks]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #31:
URL: https://github.com/apache/datafusion-benchmarks/pull/31#issuecomment-4214933273
@comphead FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: generate integer keys instead of floats in TPC-DS data [datafusion-benchmarks]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #31:
URL: https://github.com/apache/datafusion-benchmarks/pull/31#issuecomment-4363857481
I just ran into the same issue 😅 Would be great to see this merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: generate integer keys instead of floats in TPC-DS data [datafusion-benchmarks]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #31:
URL: https://github.com/apache/datafusion-benchmarks/pull/31
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org