You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "mjs (via GitHub)" <gi...@apache.org> on 2025/07/31 01:06:37 UTC
[I] S3 read request patterns are very inefficent [arrow]
mjs opened a new issue, #47239:
URL: https://github.com/apache/arrow/issues/47239
### Describe the enhancement requested
I've been looking at why Arrow's access of parquet files on an S3 store are slower when compared to Polars and ClickHouse. A packet capture highlighted what the problem is. For a single parquet file read the following S3 requests are made:
- HEAD
- HEAD
- HEAD
- Oversized ranged GET of the tail of the object to read the metadata block
- HEAD
- Ranged GETs to read the object data
If there's any significant latency between the Arrow client and the S3 (which is likely), all these requests translate into a performance bottleneck. I'm using MinIO and there's a very noticable difference in overall read performance for a client that's 1ms away from the server and one that 30 ms away. That's 150 ms before any data is transfered when it could be 60 ms. The impact gets worse the further apart the client and server are with AWS S3 and GCS likely being the worst cases.
Compare to what Polars does to read the same parquet file:
- HEAD
- 8 byte read at end to get metadata size
- Precise tail read to get metadata
- Ranged GETs from the start to read the table metadata
Arguably it could be even smarter to just read the last 64KB and save a request instead of doing an exact read of the metadata.
ClickHouse is smart when it comes to smaller objects, doing a HEAD and just grabbing the whole object in one go if the size is below some threshold. For larger objects, it does what Arrow does with too many HEAD requests (one less than Arrow).
I've tried `allow_delayed_open` but this seems to make no difference to S3 read requests despite the documentation hinting that it might. `allow_delayed_open` does help with the efficiency of writing smaller objects though.
Are there any plans to improve the efficient of Arrow's S3 reads?
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] S3 read request patterns are very inefficient [arrow]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@arrow.apache.org>.
github-actions[bot] closed issue #47239: [C++] S3 read request patterns are very inefficient
URL: https://github.com/apache/arrow/issues/47239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] S3 read request patterns are very inefficient [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47239:
URL: https://github.com/apache/arrow/issues/47239#issuecomment-3422285673
@mjs Thanks for the report. This is something that we would be interested in improving once we get some time. Can you mention which C++ API(s) you use to read the Parquet file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] S3 read request patterns are very inefficient [arrow]
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #47239:
URL: https://github.com/apache/arrow/issues/47239#issuecomment-3140544578
This is an issue that comes up pretty regularly, so I appreciate you taking the time to investigate the issue!
> Are there any plans to improve the efficient of Arrow's S3 reads?
C++ maintainer capacity is pretty low currently, so contributions are very welcome!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] S3 read request patterns are very inefficient [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #47239:
URL: https://github.com/apache/arrow/issues/47239#issuecomment-6314695485
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org