You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "masonh22 (via GitHub)" <gi...@apache.org> on 2024/10/14 19:05:04 UTC
[I] `ParquetRecordBatchStream` API to fetch the next row group while decoding [arrow-rs]
masonh22 opened a new issue, #6559:
URL: https://github.com/apache/arrow-rs/issues/6559
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I've noticed low CPU utilization when reading from filesystems with low
bandwidth using a `ParquetRecordBatchStream`. This appears to be caused by the
fact that the stream fetches row group data on demand rather than ahead of
time. In my specific scenario, I'm reading a parquet file from s3 with four
128MB row groups. It takes ~2 seconds to fetch the data and ~500ms to decode the
entire row group. In all, it takes around 10 seconds to read and decode the
entire file.
**Describe the solution you'd like**
I'd like to add the option for `ParquetRecordBatchStream` to fetch the data for
the next row group while decoding data for the current row group.
**Describe alternatives you've considered**
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `ParquetRecordBatchStream` API to fetch the next row group while decoding [arrow-rs]
Posted by "masonh22 (via GitHub)" <gi...@apache.org>.
masonh22 commented on issue #6559:
URL: https://github.com/apache/arrow-rs/issues/6559#issuecomment-2445051571
I was thinking that the pre-fetch logic could be pushed into the ParquetRecordBatchStream itself. I made a proof-of-concept of this here: https://github.com/masonh22/arrow-rs/commit/4f682b1277261dafbb40297529dc3381b0d1c32b
I didn't add any performance tests, but I confirmed that this sped things up in my own project by effectively removing the cost of decoding since reading the data takes longer than decoding it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `ParquetRecordBatchStream` API to fetch the next row group while decoding [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6559:
URL: https://github.com/apache/arrow-rs/issues/6559#issuecomment-2422214348
This is a very interesting idea -- I think it is a good one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `ParquetRecordBatchStream` API to fetch the next row group while decoding [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6559: `ParquetRecordBatchStream` API to fetch the next row group while decoding
URL: https://github.com/apache/arrow-rs/issues/6559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `ParquetRecordBatchStream` API to fetch the next row group while decoding [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6559:
URL: https://github.com/apache/arrow-rs/issues/6559#issuecomment-2422218346
Right now users can implement their own pre-fetching logic (e.g. to batch object store requests) but they don't know what ranges of data will be used.
Maybe the API could be something that calculated the required ranges based on RowGroupMetadata that were selected what file ranges would be needed
To start this this project, I would recommend that we create a motivating example (with some sort of mocked out S3 thing) showing how we wanted to read the data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org