[I] Enable multi-threaded reads of struct-list stored data in parquet files [arrow]

["dougbrn (via GitHub)" <gi...@apache.org>]

dougbrn opened a new issue, #48636:
URL: https://github.com/apache/arrow/issues/48636

   ### Describe the enhancement requested
   
   We recently discovered that nested data structures within parquet files, such as struct of lists, do not benefit from multi-threading enabled by default in pyarrow's parquet reader. However, if these are instead represented by a top-level data structure list a set of list fields, then the multi-threading works as expected. It would be nice, if possible, to enable multi-threading within nested structures that contain multiple fields. Here's a few code snippets/screenshots for context and reproducibility:
   
   File Generation:
   
   ```
   # Code block to generate needed parquet files
   from nested_pandas.datasets import generate_data
   
   # Generate a parquet dataset with struct-list format
   nf = generate_data(100,2000, seed=1)[["nested"]]
   nf.to_parquet("nested_parquet.parquet")
   
   # Generate a parquet dataset with list-array format
   nf["nested"].to_lists().to_parquet("list_parquet.parquet")
   ```
   
   Versioning & Storage Context
   ```
   import pyarrow as pa
   pa.__version__
   > '22.0.0'
   
   # struct of lists storage as read by pyarrow
   pa.parquet.read_table("nested_parquet.parquet").field("nested")
   > pyarrow.Field<nested: struct<t: list<element: double>, flux: list<element: double>, band: list<element: string>>>
   
   # list storage as read by pyarrow
   pa.parquet.read_table("list_parquet.parquet").field("t")
   > pyarrow.Field<t: list<element: double>>
   ```
   
   Single-Thread Timings:
   
   <img width="616" height="165" alt="Image" src="https://github.com/user-attachments/assets/42cf6292-26d5-4c9e-8bb2-255af3f98b08" />
   
   Multi-Thread Timings:
   
   <img width="618" height="166" alt="Image" src="https://github.com/user-attachments/assets/4c3229df-2870-4550-bfcb-f661682bc50c" />
   
   We see that multi-threading improves the read speed for list-arrays, but not for struct-list formatted data.
   
   ### Component(s)
   
   Parquet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Enable multi-threaded reads of struct-list stored data in parquet files [arrow]

["OmBiradar (via GitHub)" <gi...@apache.org>]

OmBiradar commented on issue #48636:
URL: https://github.com/apache/arrow/issues/48636#issuecomment-4679445276

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org