You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "ryankert01 (via GitHub)" <gi...@apache.org> on 2026/07/19 10:39:13 UTC
[I] feat(connectors): shared retry_async helper in iggy_connector_sdk::retry [iggy]
ryankert01 opened a new issue, #3702:
URL: https://github.com/apache/iggy/issues/3702
### Description
The connectors codebase has three hand-written copies of the same retry skeleton: `HttpRetryMiddleware::handle` and `check_connectivity_with_retry` in `core/connectors/sdk/src/retry.rs`, and `DorisSink::load_batch` in `core/connectors/sinks/doris_sink/src/lib.rs`.
The Doris sink cannot reuse the middleware: Doris Stream Load signals failure as HTTP 200 with `{"Status":"Fail"}` in the body, and the middleware never inspects response bodies. Any connector whose backend reports errors in-band has the same problem, so each one ends up hand-rolling the loop.
The copies have already diverged on backoff convention: the SDK's two loops pass the already-incremented attempt to `exponential_backoff`, so their first retry waits `2 * base`, while the Doris sink passes `attempt - 1` so the first retry waits the configured base delay, matching the documented meaning of `retry_delay`.
Raised as a non-blocking follow-up in the review of #3574: https://github.com/apache/iggy/pull/3574#discussion_r3507968371
### Affected area / component
Connectors
### Proposed solution
Expose a generic `retry_async(operation, policy)` in `iggy_connector_sdk::retry` that owns attempt counting, the transient-error predicate, capped exponential backoff with jitter, and per-retry logging, then collapse the three call sites onto it.
The shared helper should keep the base-delay-first convention (`base * 2^(k-1)` for retry `k`), since that is what the `retry_delay` config field documents.
### Alternatives considered
- Reuse `HttpRetryMiddleware` in the Doris sink: not possible, the middleware classifies on status codes and never inspects response bodies, so an in-band `{"Status":"Fail"}` looks like success to it.
- Keep per-connector loops: the exponent divergence above shows the conventions drift as soon as the skeleton is copied.
### Contribution
- [x] I'm willing to submit a pull request to implement this feature
### Good first issue
- [ ] I think this could be a good first issue for a new contributor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(connectors): shared retry_async helper in iggy_connector_sdk::retry [iggy]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 closed issue #3702: feat(connectors): shared retry_async helper in iggy_connector_sdk::retry
URL: https://github.com/apache/iggy/issues/3702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(connectors): shared retry_async helper in iggy_connector_sdk::retry [iggy]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on issue #3702:
URL: https://github.com/apache/iggy/issues/3702#issuecomment-6167385418
this is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(connectors): shared retry_async helper in iggy_connector_sdk::retry [iggy]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on issue #3702:
URL: https://github.com/apache/iggy/issues/3702#issuecomment-5601162800
There will be 3 PRs. 1 main, 2 follow-ups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org