You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pmarks (via GitHub)" <gi...@apache.org> on 2026/02/09 14:32:16 UTC
[I] Parallel Parquet Reading [arrow-rs]
pmarks opened a new issue, #9381:
URL: https://github.com/apache/arrow-rs/issues/9381
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I want to make many parallel data fetch requests to the underlying object store when fetching data with many small row groups.
This is relevant for few-column queries parquet files with modest-sized row groups using high-latency object storage like S3 and R2.
Do people think this is problem worth solving? Any suggestions on what a good API would look like? I’m going to take crack at making something work, just to explore the space but would appreciate any input.
**Describe the solution you'd like**
At a super high level the ideal interface would be ParquetRecordBatchStream or similar, but where I can configure the number of parallel read requests to generate.
**Describe alternatives you've considered**
I don't have any good ideas for how to get IO parallelism with the current types. The sequential nature of row group processing is fairly deeply baked into the state-machine architecture.
There are some related issues that touch on this, but the capability of having IO for multiple row groups in flight at the same time still appears to be unsupported:
https://github.com/apache/arrow-rs/issues/5522
https://github.com/apache/datafusion/pull/18391
https://github.com/apache/arrow-rs/issues/7983
https://github.com/apache/arrow-rs/issues/5141
https://github.com/apache/arrow-rs/pull/6907
**Additional context**
For example, I have a parquet file where I need to make ~1k reads of 250kB to read a particular column. If we assume that the per-request latency of the object store is 70ms (as observed for R2 in various benchmarks) and we get 25MB/s of throughput, then making serial requests will take 1k * 70ms + 1k * 250kB/(25MB/s) = 70s (latency) + 10s (data transfer). S3 and R2 scale to many parallel GET requests, letting us hide much of the per-request latency, if we can parallelize the requests. In a browser I can make 6 parallel requests, so we’d expect the total time to come down to ~ 70s/6 + 10s = 21s for my particular use case of in-browser parquet viz.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parallel Parquet Reading [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9381:
URL: https://github.com/apache/arrow-rs/issues/9381#issuecomment-4307491968
`label_issue.py` automatically added labels {'parquet'} from #9396
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parallel Parquet Reading [arrow-rs]
Posted by "pmarks (via GitHub)" <gi...@apache.org>.
pmarks commented on issue #9381:
URL: https://github.com/apache/arrow-rs/issues/9381#issuecomment-3884531197
I would be very happy to review! Also if you can give me a pointer to some existing code using the pattern, I could take a crack at making an example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parallel Parquet Reading [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9381:
URL: https://github.com/apache/arrow-rs/issues/9381#issuecomment-3881023566
> Do people think this is a problem worth solving? Any suggestions on what a good API or implementation would look like? I’m going to take crack at making something work, just to explore the space, but would appreciate any input.
What you can do with the APIs today are to create multiple ParquetRecordBatchStreams (one stream for each row group, for example) and run those streams in parallel . This is at a high level what DataFusion does to parallelize the reads from a parquet file -- it makes independent readers
The downside of multiple streams is that each stream will buffer an entire row group and thus require more memory
In your usecase of tiny row groups, that is probably a good tradeoff, but in the general case (like DataFusion) it is not always clear that multiple concurrent requests is a good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parallel Parquet Reading [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #9381: Parallel Parquet Reading
URL: https://github.com/apache/arrow-rs/issues/9381
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parallel Parquet Reading [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9381:
URL: https://github.com/apache/arrow-rs/issues/9381#issuecomment-4307491308
`label_issue.py` automatically added labels {'documentation'} from #9396
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parallel Parquet Reading [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9381:
URL: https://github.com/apache/arrow-rs/issues/9381#issuecomment-3881026939
FWIW I think this is a common enough usecase that it would be valuable to document (as an example) how to achieve it -- any chance you are interested in helping review such a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parallel Parquet Reading [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9381:
URL: https://github.com/apache/arrow-rs/issues/9381#issuecomment-3887589435
> I would be very happy to review! Also if you can give me a pointer to some existing code using the pattern, I could take a crack at making an example.
Here is what I got
- https://github.com/apache/arrow-rs/pull/9396
Let me know if that makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org