You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AdamGS (via GitHub)" <gi...@apache.org> on 2023/11/27 11:32:42 UTC
[I] Parquet: read parquet metadata with page index in async and with size hints [arrow-rs]
AdamGS opened a new issue, #5129:
URL: https://github.com/apache/arrow-rs/issues/5129
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
We have multiple cases where we want to just load parquet file metadata with the page index, we already have metadata and file sizes stored externally but there doesn't seem to be a way to read that data without using `object_store`.
`fetch_parquet_metadata` doesn't load the page index and but does use hints, and `ArrowReaderMetadata` doesn't use hints and forces us to jump through hoops and have an additional implementation of `AsyncFileReader`.
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
I think that adding an argument to `fetch_parquet_metadata` is easier than stabilizing `MetadataLoader`, and will work with many other readers that can read byte ranges.
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
We currently just maintain a minor fork with `MetadataLoader` and `MetadataFetch` made public.
Will be glad to do any and all work required here (either adding an argument to `fetch_parquet_metadata`, helping stabilize `MetadataLoader` or anything else) , but I want to make sure I'm doing something that's actually valuable and not just spamming the maintainers here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: read parquet metadata with page index in async and with size hints [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5129:
URL: https://github.com/apache/arrow-rs/issues/5129#issuecomment-1878523118
`label_issue.py` automatically added labels {'parquet'} from #5137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: read parquet metadata with page index in async and with size hints [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5129: Parquet: read parquet metadata with page index in async and with size hints
URL: https://github.com/apache/arrow-rs/issues/5129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: read parquet metadata with page index in async and with size hints [arrow-rs]
Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on issue #5129:
URL: https://github.com/apache/arrow-rs/issues/5129#issuecomment-1829752707
Awesome! I'll hopefully have a PR ready today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parquet: read parquet metadata with page index in async and with size hints [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5129:
URL: https://github.com/apache/arrow-rs/issues/5129#issuecomment-1829649223
I don't see a compelling reason not to make MetadataLoader public and would be happy to review a PR doing so, the API seems unlikely to change in the near future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org