You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "double-free (via GitHub)" <gi...@apache.org> on 2024/08/18 14:39:51 UTC
[PR] Yy/read pruned parquet [arrow-rs]
double-free opened a new pull request, #6269:
URL: https://github.com/apache/arrow-rs/pull/6269
# Which issue does this PR close?
Closes #6268 .
# Rationale for this change
See details in the issue description.
# What changes are included in this PR?
Support reading selected fields from a parquet file.
# Are there any user-facing changes?
No API changes, but document needs to be updated:
> Derive flat, simple RecordReader implementations. Works by parsing a struct tagged with #[derive(ParquetRecordReader)] and emitting the correct writing code for each field of the struct. Column readers are generated in the order they are defined.
Column readers are generated by name instead of index now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "double-free (via GitHub)" <gi...@apache.org>.
double-free commented on code in PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#discussion_r1735030673
##########
parquet_derive_test/src/lib.rs:
##########
@@ -70,12 +70,12 @@ struct APartiallyCompleteRecord {
// If these fields are guaranteed to be valid
// we can load this struct into APartiallyCompleteRecord
#[derive(PartialEq, ParquetRecordWriter, Debug)]
-struct APartiallyOptionalRecord {
+struct AnOptionalRecord {
Review Comment:
yes, I will restore the struct's name, but the field names must be changed. This is because the fact that now the columns are matched with field name (not index, as previous), so the constraint is:
> the decoded struct must have consistent field name with the column name in parquet
Given that the struct are written by `APartiallyCompleteRecord`, `maybe_i16` should be renamed back to `i16`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#discussion_r1734498878
##########
parquet_derive_test/src/lib.rs:
##########
@@ -70,12 +70,12 @@ struct APartiallyCompleteRecord {
// If these fields are guaranteed to be valid
// we can load this struct into APartiallyCompleteRecord
#[derive(PartialEq, ParquetRecordWriter, Debug)]
-struct APartiallyOptionalRecord {
+struct AnOptionalRecord {
Review Comment:
Why did you rename this structure and the fields? It seems like the original name `APartiallyOptionalRecord` is more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "double-free (via GitHub)" <gi...@apache.org>.
double-free commented on code in PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#discussion_r1735030673
##########
parquet_derive_test/src/lib.rs:
##########
@@ -70,12 +70,12 @@ struct APartiallyCompleteRecord {
// If these fields are guaranteed to be valid
// we can load this struct into APartiallyCompleteRecord
#[derive(PartialEq, ParquetRecordWriter, Debug)]
-struct APartiallyOptionalRecord {
+struct AnOptionalRecord {
Review Comment:
yes, I will restore the struct's name, though given the fact that now the columns are matched with field name (not index, as previous), so the constraint is:
> the decoded struct must have consistent field name with the column name in parquet
Given that the struct are written by `APartiallyCompleteRecord`, `maybe_i16` should be renamed back to `i16`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: Match fields by name, support reading selected fields rather than all [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#issuecomment-2322889047
I have updated the comments and merged up from main. I plan to merge this PR once the CI has completed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "double-free (via GitHub)" <gi...@apache.org>.
double-free commented on PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#issuecomment-2315883658
> Thank you @double-free -- the explanation and code look good to me. I am sorry for the delay in reviewing
>
> My only question is related to renaming of the `struct` in the tests -- could you double check that and respond to my question?
>
> Thanks again
Hi, I have replied to those questions. Please review again.
Btw, can you point me where to update the document of `parquet_derive`? If we choose to merge the PR, this statement will be outdated:
> Column readers are generated in the order they are defined.
Column readers are generated according to names instead of indices now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#discussion_r1734498878
##########
parquet_derive_test/src/lib.rs:
##########
@@ -70,12 +70,12 @@ struct APartiallyCompleteRecord {
// If these fields are guaranteed to be valid
// we can load this struct into APartiallyCompleteRecord
#[derive(PartialEq, ParquetRecordWriter, Debug)]
-struct APartiallyOptionalRecord {
+struct AnOptionalRecord {
Review Comment:
Why did you rename this structure? It seems like the original name `APartiallyOptionalRecord` is more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: Match fields by name, support reading selected fields rather than all [arrow-rs]
Posted by "double-free (via GitHub)" <gi...@apache.org>.
double-free commented on PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#issuecomment-2322896202
> Thanks again @double-free
No problem, I really appreciate your time, and will continue to contribute once I find reasonable improvements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#discussion_r1734502786
##########
parquet_derive/src/lib.rs:
##########
@@ -206,6 +205,12 @@ pub fn parquet_record_reader(input: proc_macro::TokenStream) -> proc_macro::Toke
let mut row_group_reader = row_group_reader;
+ // key: parquet file column name, value: column index
+ let mut name_to_index = std::collections::HashMap::new();
Review Comment:
I am not sure this strategy will work for nested structures. However, given all the tests pass we likely don't have coverage for such structures so I am not sure the existing code would work for it either.
Thus I think this is fine for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#discussion_r1734498878
##########
parquet_derive_test/src/lib.rs:
##########
@@ -70,12 +70,12 @@ struct APartiallyCompleteRecord {
// If these fields are guaranteed to be valid
// we can load this struct into APartiallyCompleteRecord
#[derive(PartialEq, ParquetRecordWriter, Debug)]
-struct APartiallyOptionalRecord {
+struct AnOptionalRecord {
Review Comment:
Why did you rename this structure and the fields? It seems like the original name `APartiallyOptionalRecord` is more accurate and the names of the fields are more specific
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "double-free (via GitHub)" <gi...@apache.org>.
double-free commented on code in PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#discussion_r1735024495
##########
parquet_derive/src/lib.rs:
##########
@@ -206,6 +205,12 @@ pub fn parquet_record_reader(input: proc_macro::TokenStream) -> proc_macro::Toke
let mut row_group_reader = row_group_reader;
+ // key: parquet file column name, value: column index
+ let mut name_to_index = std::collections::HashMap::new();
Review Comment:
Yes, agree this is fine for now. As said in the document:
> This does not generate readers or writers for arbitrarily nested structures.
This solution is reasonable under this constraint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: Match fields by name, support reading selected fields rather than all [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#issuecomment-2322892749
Thanks again @double-free
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: support reading selected fields from parquet file [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269#issuecomment-2322887953
> Btw, can you point me where to update the document of `parquet_derive`? If we choose to merge the PR, this statement will be outdated:
>
> > Column readers are generated in the order they are defined.
It appears from https://github.com/search?q=repo%3Aapache%2Farrow-rs%20%22order%20they%20are%22&type=code
this is controlled by https://github.com/apache/arrow-rs/blob/69e5e5f059783a4b98bd8dacb7e1487a6ae7e77b/parquet_derive/src/lib.rs#L149
Since I am working to prepare a release now, I will update the docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet_derive: Match fields by name, support reading selected fields rather than all [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6269:
URL: https://github.com/apache/arrow-rs/pull/6269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org