[I] parquet_derive: support reading selected columns from parquet file [arrow-rs]

["double-free (via GitHub)" <gi...@apache.org>]

double-free opened a new issue, #6268:
URL: https://github.com/apache/arrow-rs/issues/6268

   
   # Feature Description
   
   I'm effectively using `parquet_derive` in my project, and I found that there are two inconvenient constraints:
   
   1. The `ParquetRecordReader` enforces the struct to organize fields exactly in the **same order** in the parquet file.
   2. The `ParquetRecordReader` enforces the struct to parse **all fields** in the parquet file.
   
   As describe in its document:
   
   > Derive flat, simple RecordReader implementations. Works by parsing a struct tagged with #[derive(ParquetRecordReader)] and emitting the correct writing code for each field of the struct. Column readers are generated in the order they are defined.
   
   In my use cases (and I believe these are common requests), user should be able to read pruned parquet file, and they should have the freedom to re-organize fields' ordering in decoded struct.
   
   # My Solution
   
   I introduced a `HashMap` to map field name to its index. Of course, it assumes field name is unique, and this is always true since the current `parquet_derive` macro is applied to a flat struct without nesting.
   
   # Pros and Cons
   
   Obviously removing those two constraints makes `parquet_derive` a more handy tool.
   
   But it has some implied changes:
   
   - previously, since the `ParquetRecordReader` relies only on the index of fields, it allows that a field is named as `abc` to implicitly rename itself to `bcd` in the encoded struct. After this change, user must guarantee that the field name in `ParquetRecordReader` to exist in parquet columns.
     - I think it is more intuitive and more natural to constrain the "field name" rather than "index", if we use `ParquetRecordReader` to derive a decoder macro.
   - allowing reading partial parquet file may improve the performance for some users, but introducing a `HashMap` in the parser may slowdown the function a bit.
     - when the `num_records` in a single parsing call is large enough, the cost of `HashMap` lookup is negligible.
   
   Both implied changes seem to have a more positive impact than negative impact. Please review if this is a reasonable feature request. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] parquet_derive: support reading selected columns from parquet file [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #6268:
URL: https://github.com/apache/arrow-rs/issues/6268#issuecomment-2322898974

   `label_issue.py` automatically added labels {'parquet-derive'} from #6269


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] parquet_derive: support reading selected columns from parquet file [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb closed issue #6268: parquet_derive: support reading selected columns from parquet file
URL: https://github.com/apache/arrow-rs/issues/6268


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org