Re: [I] [Rust] [DataFusion] Improve parallelism when reading Parquet files [arrow]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #26916:
URL: https://github.com/apache/arrow/issues/26916#issuecomment-2085672258

   This came up on discord recently so moving some info 
   
   DataFusion supports reading the same parquet file using multiple threads
   
   Here is the low level arrow API to do it from @tustvold 
   
   > You can read row groups in parallel by creating separate readers with different row group selections
   
   https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/type.ParquetRecordBatchReaderBuilder.html#method.new_with_metadata
   
   This is used by by datafusion automatically. The function that does the parallelization is 
   1. https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.repartitioned 
   2. implemented for https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/parquet/struct.ParquetExec.html
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org