You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2024/11/22 23:47:54 UTC
[I] Allow disabling the writing of Parquet Offset Index [arrow-rs]
etseidl opened a new issue, #6778:
URL: https://github.com/apache/arrow-rs/issues/6778
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
As of now, when writing Parquet files, the [offset index](https://github.com/apache/parquet-format/blob/master/PageIndex.md) structures are populated and written regardless of whether statistics and column indexes are written. It is unclear if this behavior is intended or not.
**Describe the solution you'd like**
Add a writer option to disable the collection and writing of the offset index. Of course the offset index is required if the column index is written, so this option would probably only be useful when not writing column indexes (i.e. when the statistics level is `None` or `Chunk`).
**Describe alternatives you've considered**
Alternatively, the writing of the offset index could be disabled whenever the column index is disabled (i.e. when the stats level is not `Page`). This solution assumes the current behavior is not intentional.
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow disabling the writing of Parquet Offset Index [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6778:
URL: https://github.com/apache/arrow-rs/issues/6778#issuecomment-2495140615
> I think parquet-rs should then not write offset indexes if page statistics are not enabled.
There are valid use-cases where the offset index is beneficial, but the page statistics might not be desired. For example, if using some external index or statistics. The offset index is critical to being able to efficiently perform pushdown.
I think we can add an option to disable offset index generation, but we should make this an explicit option and stick a big warning that it may severely degrade read performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow disabling the writing of Parquet Offset Index [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #6778: Allow disabling the writing of Parquet Offset Index
URL: https://github.com/apache/arrow-rs/issues/6778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow disabling the writing of Parquet Offset Index [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6778:
URL: https://github.com/apache/arrow-rs/issues/6778#issuecomment-2549319142
`label_issue.py` automatically added labels {'parquet'} from #6797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow disabling the writing of Parquet Offset Index [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #6778:
URL: https://github.com/apache/arrow-rs/issues/6778#issuecomment-2495157389
Thanks @tustvold, sounds reasonable. I'll start on adding an option next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow disabling the writing of Parquet Offset Index [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #6778:
URL: https://github.com/apache/arrow-rs/issues/6778#issuecomment-2495136003
After some git dumpster diving, it seems like at one point the offset indexes were not written if the column indexes were not valid. https://github.com/apache/arrow-rs/blame/fba19b0142daed54c181cdb8f634f29cf7d37f8d/parquet/src/column/writer/mod.rs#L503-L510
Writing of the page indexes was decoupled in #4567. Since this is a special case where page indexes are desired, but the column index cannot be written due to all NaNs, it seems the original intent was to not write the offset index. I think parquet-rs should then not write offset indexes if page statistics are not enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow disabling the writing of Parquet Offset Index [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #6778:
URL: https://github.com/apache/arrow-rs/issues/6778#issuecomment-2495136853
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org