You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "yadavay-amzn (via GitHub)" <gi...@apache.org> on 2026/05/15 08:20:03 UTC
[PR] Parquet: Enforce row group size limit with compression [iceberg]
yadavay-amzn opened a new pull request, #16347:
URL: https://github.com/apache/iceberg/pull/16347
Fixes #16325.
## Problem
When using GZIP or ZSTD compression, the row group size check in `ParquetWriter` uses `writeStore.getBufferedSize()` which reports compressed bytes after page flushes. Since compressed size is significantly smaller than the configured `targetRowGroupSize`, the threshold is never reached and row groups grow unbounded.
## Fix
Track uncompressed bytes by measuring the `getBufferedSize()` delta before and after each `model.write()` call (before `endRecord()` triggers page flush and compression). Use this accumulated uncompressed size in `checkSize()` instead of the post-compression buffered size. Reset on row group flush.
## Testing
Added `testRowGroupSizeEnforcedWithCompression` in `TestParquet` -- writes 500 records of ~1KB each with GZIP compression and a 64KB row group target. Asserts multiple row groups are created.
- **Without fix**: all 500 records end up in 1 row group (compressed size never hits threshold)
- **With fix**: multiple row groups created respecting the 64KB target
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Enforce row group size limit with compression [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on PR #16347:
URL: https://github.com/apache/iceberg/pull/16347#issuecomment-4726783032
@yadavay-amzn the fix has been merged and the issue has been closed. Feel free to close this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Enforce row group size limit with compression [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16347:
URL: https://github.com/apache/iceberg/pull/16347#issuecomment-4703667371
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Enforce row group size limit with compression [iceberg]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #16347:
URL: https://github.com/apache/iceberg/pull/16347#issuecomment-4462004732
@nssalian Thanks for the heads up. I looked at #16327 -- the key difference is that your PR makes the fix opt-in via a new config flag (default `false`), while this PR fixes the behavior by default.
My reasoning: the current behavior is a bug (row group size limit is silently ignored with compression). Users who set `write.parquet.row-group-size-bytes` expect it to be enforced regardless of codec. Making the fix opt-in means most users will never discover the config and continue hitting unbounded row groups.
Happy to discuss the trade-off -- if maintainers prefer the opt-in approach for backward compatibility, I can close this. But I think the correct default is to enforce the configured limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Enforce row group size limit with compression [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on PR #16347:
URL: https://github.com/apache/iceberg/pull/16347#issuecomment-4461574575
#16327 solves this, @yadavay-amzn please close this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet: Enforce row group size limit with compression [iceberg]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn closed pull request #16347: Parquet: Enforce row group size limit with compression
URL: https://github.com/apache/iceberg/pull/16347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org