You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "benauthor (via GitHub)" <gi...@apache.org> on 2025/05/16 15:21:54 UTC
[I] Retry [arrow-rs-object-store]
benauthor opened a new issue, #368:
URL: https://github.com/apache/arrow-rs-object-store/issues/368
According to the docs on [RetryConfig](https://docs.rs/object_store/0.12.0/object_store/struct.RetryConfig.html) the intent expressed seems to be that connection errors should be retried. However the [code](https://docs.rs/crate/object_store/0.12.0/source/src/client/retry.rs#374-403) does not implement this (if the error is not an HTTP 4xx or 5xx, it exits without retrying on L390).
A typical example of a non-retried error I saw in practice is:
```
Service(ObjectStore(Generic { store: "S3", source: RetryError { method: PUT, uri: Some(XXX-REDACTED-XXX), retries: 0, max_retries: 3, elapsed: 801.473µs, retry_timeout: 60s, inner: Http(HttpError { kind: Unknown, source: reqwest::Error { kind: Request, source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })) } }) } }))
```
Is there interest in making the behavior match the documentation? Otherwise it'd be good to edit the documentation to match the behavior. I prefer the former, but either way I'd be happy to chip in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry does not cover connection errors [arrow-rs-object-store]
Posted by "johnnyg (via GitHub)" <gi...@apache.org>.
johnnyg commented on issue #368:
URL: https://github.com/apache/arrow-rs-object-store/issues/368#issuecomment-3130208822
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry does not cover connection errors [arrow-rs-object-store]
Posted by "criccomini (via GitHub)" <gi...@apache.org>.
criccomini commented on issue #368:
URL: https://github.com/apache/arrow-rs-object-store/issues/368#issuecomment-3089855509
Just came here to say we're hitting:
```
thread 'tokio-runtime-worker' panicked at /root/.cargo/git/checkouts/slatedb-a6e73982df30678a/2fe991a/slatedb/src/compactor.rs:293:18:
fatal error loading manifest: ObjectStoreError(Generic { store: "S3", source: ListRequest { source: RetryError(RetryErrorImpl { method: GET, uri: Some(https://fsn1.your-objectstorage.com/zerofs?list-type=2&prefix=test_new_123%2Fmanifest%2F), retries: 0, max_retries: 10, elapsed: 570.685µs, retry_timeout: 180s, inner: Http(HttpError { kind: Unknown, source: reqwest::Error { kind: Request, source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })) } }) }) } })
```
On 0.12.3 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry does not cover connection errors [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #368:
URL: https://github.com/apache/arrow-rs-object-store/issues/368#issuecomment-2887462282
Possibly already fixed by https://github.com/apache/arrow-rs-object-store/pull/351 (in latest release)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry does not cover connection errors [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #368:
URL: https://github.com/apache/arrow-rs-object-store/issues/368#issuecomment-2887390417
Connection errors should actually end up on this line - https://docs.rs/crate/object_store/0.12.0/source/src/client/retry.rs#409
They will then be retried provided the request was idempotent, which in the case of a PUT request is if it is `PutMode::Overwrite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry does not cover connection errors [arrow-rs-object-store]
Posted by "benauthor (via GitHub)" <gi...@apache.org>.
benauthor commented on issue #368:
URL: https://github.com/apache/arrow-rs-object-store/issues/368#issuecomment-2887458244
Apologies for my fast and loose reading and thanks for directing me better! Given the logging `HttpError { kind: Unknown,` I mentioned, I guess in this case we are actually hitting [L410](https://docs.rs/crate/object_store/0.12.0/source/src/client/retry.rs#410) which explains the lack of retry. Given [this comment](https://docs.rs/crate/object_store/0.12.0/source/src/client/connection.rs#102) perhaps this is another reqwest error case that wants special handling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry does not cover connection errors [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #368: Retry does not cover connection errors
URL: https://github.com/apache/arrow-rs-object-store/issues/368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry does not cover connection errors [arrow-rs-object-store]
Posted by "benauthor (via GitHub)" <gi...@apache.org>.
benauthor commented on issue #368:
URL: https://github.com/apache/arrow-rs-object-store/issues/368#issuecomment-2887472741
I'm going to say it's highly plausible that's a fix for this and close. 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry does not cover connection errors [arrow-rs-object-store]
Posted by "benauthor (via GitHub)" <gi...@apache.org>.
benauthor closed issue #368: Retry does not cover connection errors
URL: https://github.com/apache/arrow-rs-object-store/issues/368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org