You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "westonpace (via GitHub)" <gi...@apache.org> on 2024/08/28 21:25:15 UTC
[I] The description of the default backoff algorithm does not match the implementation [arrow-rs]
westonpace opened a new issue, #6324:
URL: https://github.com/apache/arrow-rs/issues/6324
**Describe the bug**
In `object_store::BackoffConfig` we have:
```
Exponential backoff with jitter
See https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
```
However, the actual implementation is "decorrelated jitter" (which is not exponential).
**To Reproduce**
N/A docs bug
**Expected behavior**
Docs describe implementation
**Additional context**
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Better Document Backoff Algorithm [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6324:
URL: https://github.com/apache/arrow-rs/issues/6324#issuecomment-2317288465
I take your point, however, it is a jittered exponential backoff, i.e. the base is exponential backoff which is then jittered. This is reflected both in the wording of the linked doc, and the naming in the various AWS codebases, e.g. - https://github.com/aws-samples/aws-arch-backoff-simulator/blob/master/src/backoff_simulator.py#L40
The backoff growth is still non-linear, and this is important to convey IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Better Document Backoff Algorithm [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #6324: Better Document Backoff Algorithm
URL: https://github.com/apache/arrow-rs/issues/6324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The description of the default backoff algorithm does not match the implementation [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6324:
URL: https://github.com/apache/arrow-rs/issues/6324#issuecomment-2316307847
It is still exponential backoff, in that it multiples the last delay by some base? The linked docs also describe it thus?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Better Document Backoff Algorithm [arrow-rs]
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #6324:
URL: https://github.com/apache/arrow-rs/issues/6324#issuecomment-2318290925
I disagree (chart included), but I don't really want to fight the point. As long as the actual process is clearly documented readers should figure it out. I've accepted your PR suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Better Document Backoff Algorithm [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6324:
URL: https://github.com/apache/arrow-rs/issues/6324#issuecomment-2741681034
Migrating from arrow-rs issue [#6324](https://github.com/apache/arrow-rs/issues/6324)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Better Document Backoff Algorithm [arrow-rs]
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #6324:
URL: https://github.com/apache/arrow-rs/issues/6324#issuecomment-2316339634
Sort of...it's kind of "exponential with restarts" in the sense that it can always randomly jump back to a very small value:
It can also just randomly choose not to grow at all (e.g. 70-80 in the chart). I think my problem is that both "exponential backoff" and "exponential backoff with jitter" are well known approaches (they've been around since before AWS) and if anyone sees "exponential backoff" they aren't going to bother to read the linked document.
Also, the linked document treats exponential backoff as an algorithm of its own. E.g. it has its own bar in the legend:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org