You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2025/10/08 12:54:43 UTC
[I] Make compression adaptive with V2 data pages [parquet-java]
pitrou opened a new issue, #3344:
URL: https://github.com/apache/parquet-java/issues/3344
### Describe the enhancement requested
When writing a V2 data page, it seems that compression is always unconditionally enabled even when compression doesn't actually yield any benefits:
https://github.com/apache/parquet-java/blob/0fea3e1e22fffb0a25193e3efb9a5d090899458a/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java#L305-L311
It would be relatively easy to use a hardcoded threshold above which compression is disabled, which makes reading faster.
### Component(s)
Core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Make compression adaptive with V2 data pages [parquet-java]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #3344:
URL: https://github.com/apache/parquet-java/issues/3344#issuecomment-3431666491
Yes, and I agree that adding a tunable parameter would be even better. In Arrow C++ IPC we have:
https://github.com/apache/arrow/blob/03896451c69658105e857ae7103e5081bbaa9bd6/cpp/src/arrow/ipc/options.h#L72-L85
It could have a conservative default such as `0.02`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Make compression adaptive with V2 data pages [parquet-java]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #3344:
URL: https://github.com/apache/parquet-java/issues/3344#issuecomment-3432038854
A config is ok to me. It also makes sense in https://github.com/apache/arrow/blob/03896451c69658105e857ae7103e5081bbaa9bd6/cpp/src/parquet/column_writer.cc#L1051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Make compression adaptive with V2 data pages [parquet-java]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #3344:
URL: https://github.com/apache/parquet-java/issues/3344#issuecomment-3431630734
In case it helps for reference, @mapleFU made a similar change in the arrow-rs / Rust parquet reader a few weeks ago:
- https://github.com/apache/arrow-rs/pull/8257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org