You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Tom-Newton (via GitHub)" <gi...@apache.org> on 2023/07/03 12:58:03 UTC
[GitHub] [arrow] Tom-Newton opened a new issue, #36441: Make `CacheOptions` configurable from Python
Tom-Newton opened a new issue, #36441:
URL: https://github.com/apache/arrow/issues/36441
### Describe the enhancement requested
When doing parquet reads from high latency storage e.g. S3 or Azure blob storage its beneficial to use `fragment_scan_options=ParquetFragmentScanOptions(pre_buffer=True)`.
As far as I can tell `pre_buffer=True` does 2 important things:
1. It reads all the parquet metadata at the beginning then makes all the data block reads.
2. It coalesces adjacent data blocks so that it can read the data in a smaller number of larger reads. Coalescing is done based on `CacheOptions`, in particular `hole_size_limit` and `range_size_limit`.
In my use-case I have found that while point 1 is very beneficial for performance point 2 is actually detrimental. Point 1 though is still enough to make `pre_buffer=True` a net advantage. I found that setting `range_size_limit` to a much smaller value (in combination with `pyarrow.set_io_thread_count(<large number>)`) can give me ~6X performance improvement on my usecase because it increases the parallelism. I would therefore like to make these parameters configurable from python.
If you are interested in my use-case I'm loading a single parquet file from US East Azure blob storage to a workstation in the UK. Probably the current defaults would be sensible if it wasn't for the trans-Atlantic latency. (Pyarrow does not natively support Azure yet but I'm making use of https://github.com/apache/arrow/pull/12914 even though its not in an official release yet)
I can make a PR that implements this if others agree that its a sensible idea.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #36441: [Python] Make `CacheOptions` configurable from Python
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #36441:
URL: https://github.com/apache/arrow/issues/36441#issuecomment-1621628109
It seems reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Make `CacheOptions` configurable from Python [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche closed issue #36441: [Python] Make `CacheOptions` configurable from Python
URL: https://github.com/apache/arrow/issues/36441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #36441: [Python] Make `CacheOptions` configurable from Python
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36441:
URL: https://github.com/apache/arrow/issues/36441#issuecomment-1621652122
The CacheOptions is also already explicitly exposed in the `IpcFragmentScanOptions` (on the C++ side, this class it not yet exposed in Python).
All the other options of `ArrowReaderProperties` are exposed in the pyarrow equivalent (in `ParquetReader` in _parquet.pyx), so indeed seems to make sense to expose the cache options as well.
In the ParquetFragmentScanOptions, currently only `pre_buffer` is exposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #36441: [Python] Make `CacheOptions` configurable from Python
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36441:
URL: https://github.com/apache/arrow/issues/36441#issuecomment-1621246704
cc @westonpace @lidavidm @AlenkaF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Tom-Newton commented on issue #36441: [Python] Make `CacheOptions` configurable from Python
Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on issue #36441:
URL: https://github.com/apache/arrow/issues/36441#issuecomment-1625852466
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Tom-Newton commented on issue #36441: [Python] Make `CacheOptions` configurable from Python
Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on issue #36441:
URL: https://github.com/apache/arrow/issues/36441#issuecomment-1632114356
I've opened a PR for review to address this https://github.com/apache/arrow/pull/36627.
I don't have much experience with C++ or Cython and this is my first contribution to arrow so hopefully what I've done makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org