You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "kszlim (via GitHub)" <gi...@apache.org> on 2024/02/09 06:50:31 UTC
[I] Should object store retry on connection reset by peer? [arrow-rs]
kszlim opened a new issue, #5378:
URL: https://github.com/apache/arrow-rs/issues/5378
**Which part is this question about**
Object store
**Describe your question**
Should object store's retry logic also retry when the connection is reset by a peer?
**Additional context**
I'm querying an s3 bucket via polars (which uses object store under the hood) and I'm encountering this issue, this only happens when I'm querying many (~10k files)
```
Generic S3 error: Error after 0 retries in 221.182308ms, max_retries:10, retry_timeout:10s, source:error sending request for url (https://s3.us-west-2.amazonaws.com/bucket/some_parquet_id.parquet): connection error: Connection reset by peer (os error 104)
```
This seems to be because this error isn't covered under object store's retry policy.
It'd be very handy if this was, though I'm not certain if it'd be strictly correct behavior (ie. Should it be handled by the caller of object store instead? Which in this case is polars).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Should object store retry on connection reset by peer? [arrow-rs]
Posted by "flokli (via GitHub)" <gi...@apache.org>.
flokli commented on issue #5378:
URL: https://github.com/apache/arrow-rs/issues/5378#issuecomment-2302870472
I've been running into something like this - apparently Azure has a 30 second request duration limit, and closes the connection afterwards. I opened https://github.com/apache/arrow-rs/issues/6287 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Should object store retry on connection reset by peer? [arrow-rs]
Posted by "kszlim (via GitHub)" <gi...@apache.org>.
kszlim commented on issue #5378:
URL: https://github.com/apache/arrow-rs/issues/5378#issuecomment-1936798128
Do you think it'd be alright to modify:
https://github.com/apache/arrow-rs/blob/master/object_store/src/client/retry.rs#L266
to:
`!(e.is_parse() || e.is_parse_status() || e.is_parse_too_large() || e.is_user() || e.is_canceled())`
Which would be much more aggressive wrt what states to retry on? Ideally the error `Kind` in hyper would be publically exposed to allow deeper introspection into failure modes that can be retried, but that doesn't seem like it's going to be on the roadmap for a while (see: https://github.com/hyperium/hyper/issues/2845).
Whilst this might be more aggressive than desired, I think users could accordingly adjust their retries/backoff config to compensate, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Should object store retry on connection reset by peer? [arrow-rs]
Posted by "kszlim (via GitHub)" <gi...@apache.org>.
kszlim commented on issue #5378:
URL: https://github.com/apache/arrow-rs/issues/5378#issuecomment-1936407201
> means you are hitting rate limits and should reduce the amount of concurrent IO you are performing. There is a LimitStore
Hmm, I've tried reducing concurrency in polars (to 32) and it doesn't seem to fix this, though I doubt i'm anywhere near the stated rate limit of s3 which is 5500 GET requests per partitioned prefix per second.
If it's not incorrect behavior, do you think it makes sense to also build this condition into the retries done by object store @tustvold ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Should object store retry on connection reset by peer? [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5378:
URL: https://github.com/apache/arrow-rs/issues/5378#issuecomment-1935823412
We probably could retry, but connection reset normally means you are hitting rate limits and should reduce the amount of concurrent IO you are performing. There is a LimitStore that might achieve this, if polars can't do this itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Should object store retry on connection reset by peer? [arrow-rs]
Posted by "kszlim (via GitHub)" <gi...@apache.org>.
kszlim commented on issue #5378:
URL: https://github.com/apache/arrow-rs/issues/5378#issuecomment-1937094794
Opened a PR https://github.com/apache/arrow-rs/pull/5383, feel free to close it if you think it's unreasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Should object store retry on connection reset by peer? [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5378:
URL: https://github.com/apache/arrow-rs/issues/5378#issuecomment-1980023043
The conclusion of https://github.com/pola-rs/polars/pull/14598 appears to be that this was an upstream issue, so closing this. Feel free to reopen if I am mistaken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Should object store retry on connection reset by peer? [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5378: Should object store retry on connection reset by peer?
URL: https://github.com/apache/arrow-rs/issues/5378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org