You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/10/08 12:14:41 UTC
[I] Remove unused `parquet::format` crate [arrow-rs]
alamb opened a new issue, #8572:
URL: https://github.com/apache/arrow-rs/issues/8572
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- part of https://github.com/apache/arrow-rs/issues/5853
The `parquet::format` module contains auto generated code for parsing data in the parquet.thrift format
After we merge https://github.com/apache/arrow-rs/pull/8530 from @etseidl these structures will no longer be needed and their presence is ust confusing
I tested it out a bit locally and the only place the thrift structures are now used is in the `parquet-format` tool and some benchmarks
<img width="1276" height="471" alt="Image" src="https://github.com/user-attachments/assets/95a38522-6682-48a6-84b9-4a7e4468fa35" />
**Describe the solution you'd like**
Delete the `parquet::format` module (`parquet/src/format.rs`), and update the tools to use the new thrift structures
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Remove unused `parquet::format` crate [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8572:
URL: https://github.com/apache/arrow-rs/issues/8572#issuecomment-3401647692
> A question that came up is do we want to keep parquet-layout. There are other tools that display page information in more detail.
I would personally suggest not maintaining the tool.
Maybe we can just leave a comment in the binary with a link to the other tools to use instead? If someone feels strongly they can work on adding an appropriate API for page header extraction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Remove unused `parquet::format` crate [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8572:
URL: https://github.com/apache/arrow-rs/issues/8572#issuecomment-3403700101
Note to self to revisit this for `59.0.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Remove unused `parquet::format` crate [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8572:
URL: https://github.com/apache/arrow-rs/issues/8572#issuecomment-3398415783
Thanks to @lichuang we're halfway to completing this.
A question that came up is do we want to keep `parquet-layout`. There are other tools that display page information in more detail. If we want to keep it, then we'll need to add something to the metadata reader to extract page header info. Or we could try to leverage the page encoding stats and offset index if they're available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Remove unused `parquet::format` crate [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8572:
URL: https://github.com/apache/arrow-rs/issues/8572#issuecomment-4482531165
Completed in #9962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Remove unused `parquet::format` crate [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8572:
URL: https://github.com/apache/arrow-rs/issues/8572#issuecomment-3398453256
I also wonder if we should deprecate `format` before removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Remove unused `parquet::format` crate [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl closed issue #8572: Remove unused `parquet::format` crate
URL: https://github.com/apache/arrow-rs/issues/8572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org