You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "wingkitlee0 (via GitHub)" <gi...@apache.org> on 2025/09/02 14:32:41 UTC
[I] [Python][Doc] use_content_defined_chunking not shown correctly for ParquetWriter / write_table [arrow]
wingkitlee0 opened a new issue, #47477:
URL: https://github.com/apache/arrow/issues/47477
### Describe the bug, including details regarding any error messages, version, and platform.
Was reading about [`Content-Defined Chunking`](https://huggingface.co/blog/parquet-cdc) and came across
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html etc
`use_content_defined_chunking` was documented in the docstring, but was not part of the APIs on the top of the page.
Also, it will be useful to have a note about the consequence of setting `use_content_defined_chunking=True` on systems that do not support CDC.
(final personal thought: should this filesystem-specific option to be set somewhere else? for now this is only relevant for Hugging face now?)
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Docs] use_content_defined_chunking not shown correctly for ParquetWriter / write_table [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on issue #47477:
URL: https://github.com/apache/arrow/issues/47477#issuecomment-5716496853
This issue got resolved by #47498 as well.
The issue's side question:
> Also, it will be useful to have a note about the consequence of setting use_content_defined_chunking=True on systems that do not support CDC.
was answered by @kszucs https://github.com/apache/arrow/pull/47498#issuecomment-5574070842
> essentially there are no meaningful effects on non-CAS storage systems, the data pages won't be fixed sized but mostly that's it.
For now no new notes in the docs for that, but welcome to open new follow-up requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][Docs] use_content_defined_chunking not shown correctly for ParquetWriter / write_table [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja closed issue #47477: [Python][Docs] use_content_defined_chunking not shown correctly for ParquetWriter / write_table
URL: https://github.com/apache/arrow/issues/47477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org