You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kszucs (via GitHub)" <gi...@apache.org> on 2026/07/06 16:23:35 UTC

[PR] Add Parquet content-defined chunking (CDC) writer support [iceberg-python]

kszucs opened a new pull request, #3608:
URL: https://github.com/apache/iceberg-python/pull/3608

   # Rationale for this change
   
   PyArrow's `ParquetWriter` has natively supported content-defined chunking (CDC) since 21.0.0, producing stable page boundaries across appends (useful for content-addressable storage / dedup). PyIceberg's write path already funnels through a single kwargs builder (`_get_parquet_writer_kwargs`), so this wires CDC through as `write.parquet.content-defined-chunking.*` table properties, mirroring the property names and defaults iceberg-rust already uses for cross-engine consistency. A `pyarrow>=21.0.0` version guard raises a clear `ImportError` if CDC is requested on an older PyArrow (extracted into a shared `_require_pyarrow_version` helper, reused by the existing Azure-filesystem version guard).
   
   ## Are these changes tested?
   
   Yes: unit tests for `_get_parquet_writer_kwargs` (disabled by default, enabled with defaults, enabled with custom values, unsupported PyArrow version) and an integration-style test that writes a table with CDC enabled end-to-end and reads it back.
   
   ## Are there any user-facing changes?
   
   Yes: four new table properties (`write.parquet.content-defined-chunking.enabled`, `.min-chunk-size`, `.max-chunk-size`, `.norm-level`), documented in `configuration.md`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] fead: add Parquet content-defined chunking writer support [iceberg-python]

Posted by "kszucs (via GitHub)" <gi...@apache.org>.
kszucs commented on PR #3608:
URL: https://github.com/apache/iceberg-python/pull/3608#issuecomment-4925170717

   @anxkhn could you please take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] fead: add Parquet content-defined chunking writer support [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3608: fead: add Parquet content-defined chunking writer support
URL: https://github.com/apache/iceberg-python/pull/3608


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] fead: add Parquet content-defined chunking writer support [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3608:
URL: https://github.com/apache/iceberg-python/pull/3608#issuecomment-5349579974

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] fead: add Parquet content-defined chunking writer support [iceberg-python]

Posted by "kszucs (via GitHub)" <gi...@apache.org>.
kszucs commented on PR #3608:
URL: https://github.com/apache/iceberg-python/pull/3608#issuecomment-5020037678

   @anxkhn @ebyhr may I request a review? Sorry for pinging directly, but the request review functionality is not available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] fead: add Parquet content-defined chunking writer support [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3608:
URL: https://github.com/apache/iceberg-python/pull/3608#issuecomment-5433237325

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] fead: add Parquet content-defined chunking writer support [iceberg-python]

Posted by "kszucs (via GitHub)" <gi...@apache.org>.
kszucs commented on PR #3608:
URL: https://github.com/apache/iceberg-python/pull/3608#issuecomment-5491323915

   Reopened as #3889 (rebased on current main) — GitHub wouldn't let this one be reopened after the branch was updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org