[I] Make cloning `ParquetMetaData` less expensive [arrow-rs]

["etseidl (via GitHub)" <gi...@apache.org>]

etseidl opened a new issue, #11328:
URL: https://github.com/apache/arrow-rs/issues/11328

   ### Is your feature request related to a problem or challenge?
   
   This is related to #7582 and grew out of a discussion in https://github.com/apache/datafusion/issues/24288. To summarize, given a desire to cache parts of the parquet metadata to allow selective population and reuse, somewhere along the line one encounters a need to clone the `ParquetMetaData` (for instance if the metadata is wrapped in an `Arc` and is subsequently shared). This clone can be very expensive as it entails copying the dense vector of `RowGroupMetaData`, which is essentially a `num_row_groups X num_columns` matrix of `ColumnChunkMetaData`s.
   
   
   ### Describe the solution you'd like
   
   Wrap the `file_metadata` and `row_groups` members of the `ParquetMetaData` in `Arc` to allow cheap cloning. This leads to few problems because those fields are generally read once and not mutated. The one production exception to this appears to be in `SerializedFileWriter::new_with_options` where a newly created `ParquetMetaData` is converted to a builder, the row groups extracted and filtered via a predicate, and the `ParquetMetaData` reassembled.
   
   ### Describe alternatives you've considered
   
   Careful management of the ParquetMetaData lifetime may allow clones to be avoided. 
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Make cloning `ParquetMetaData` less expensive [arrow-rs]

["etseidl (via GitHub) via github" <gi...@arrow.apache.org>]

etseidl closed issue #11328: Make cloning `ParquetMetaData` less expensive
URL: https://github.com/apache/arrow-rs/issues/11328


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org