You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "mattp5657 (via GitHub)" <gi...@apache.org> on 2026/09/07 13:21:05 UTC
[I] Retry backoff has an off-by-one delay and an unclamped post-jitter overshoot [iggy]
mattp5657 opened a new issue, #4084:
URL: https://github.com/apache/iggy/issues/4084
### Description
`exponential_backoff`/`jitter` in `core/connectors/sdk/src/retry.rs` have two
bugs, both reproduced by every call site that composes them by hand:
1. **Off-by-one doubles the first retry's delay.** `exponential_backoff(base, attempt, max_delay)` expects a 0-based `attempt` (`0` → `base`). Most call sites increment their counter *before* calling it, so the first retry passes `1` and waits `base * 2` instead of `base`, and every subsequent retry is skewed one step ahead too.
2. **Jitter is applied after the cap, so it can overshoot `max_delay`.** `exponential_backoff` clamps to `max_delay` internally, but `jitter(...)` (±20%) wraps that already-capped value with no re-clamp: a retry can sleep up to `max_delay * 1.2`, silently exceeding the configured ceiling.
### Affected area / component
Connectors, Rust SDK
### Proposed solution
Fix the interface, not every call site: change `exponential_backoff` to
accept a 1-based total-attempt count internally (subtract 1 before computing
the exponent), so the call every caller already makes naturally (an
incremented counter) becomes correct.
- **No change needed:** SDK's `HttpRetryMiddleware`/`check_connectivity_with_retry` (fixes `influxdb_sink`/`influxdb_source` too), `meilisearch_sink`, `core/connectors/sources/iggy_source`.
- **Drop the now-redundant manual `-1`:** `s3_sink`, `surrealdb_sink`, `doris_sink`, `opensearch_sink`, SDK `source.rs::nack_retry_delay`.
- **Separate fix, unaffected by the interface change:** `clickhouse_sink`'s local `jittered_backoff` (own `attempts - 1` fix, 4 call sites).
Also add `.min(max_delay)` after `jitter(...)` once, in a shared
`retry_backoff(base, attempt, max_delay) -> Duration` helper, and have every
call site use it instead of composing `jitter`/`exponential_backoff` by hand.
This closes the clamp gap in the same pass since those call sites are already
being touched.
### Alternatives considered
Fix each call site individually instead of the function.
The current 0-based contract is the one nearly every caller gets wrong (SDK's own middleware, `meilisearch_sink`, and `iggy_source` all pass an already-incremented counter). Since the interface itself is being misread by most callers, fixing it at the source removes the problem for good instead of leaving it for the next connector to repeat.
### Contribution
- [ ] I'm willing to submit a pull request to implement this feature
### Good first issue
- [x] I think this could be a good first issue for a new contributor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry backoff has an off-by-one delay and an unclamped post-jitter overshoot [iggy]
Posted by "mattp5657 (via GitHub)" <gi...@apache.org>.
mattp5657 commented on issue #4084:
URL: https://github.com/apache/iggy/issues/4084#issuecomment-5571284865
@slbotbm This stemmed from your review of OpenSearch Connector so curious on your thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry backoff has an off-by-one delay and an unclamped post-jitter overshoot [iggy]
Posted by "mattp5657 (via GitHub)" <gi...@apache.org>.
mattp5657 commented on issue #4084:
URL: https://github.com/apache/iggy/issues/4084#issuecomment-5857370057
Hey @ryankert01 🙂
Has this been closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry backoff has an off-by-one delay and an unclamped post-jitter overshoot [iggy]
Posted by "jiengup (via GitHub)" <gi...@apache.org>.
jiengup commented on issue #4084:
URL: https://github.com/apache/iggy/issues/4084#issuecomment-5605391489
@Youmanvi
To avoid duplication of work, I'd like to note that #4104 is solving this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry backoff has an off-by-one delay and an unclamped post-jitter overshoot [iggy]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on issue #4084:
URL: https://github.com/apache/iggy/issues/4084#issuecomment-5894969041
@mattp5657 I don't think I fully solve all you have in this issue. That PR is focused on a earlier issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry backoff has an off-by-one delay and an unclamped post-jitter overshoot [iggy]
Posted by "Youmanvi (via GitHub)" <gi...@apache.org>.
Youmanvi commented on issue #4084:
URL: https://github.com/apache/iggy/issues/4084#issuecomment-5572515932
Hey I'd like to work on this issue, could you assign it to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Retry backoff has an off-by-one delay and an unclamped post-jitter overshoot [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #4084:
URL: https://github.com/apache/iggy/issues/4084#issuecomment-5572635832
assigned @Youmanvi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org