You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "surafel58 (via GitHub)" <gi...@apache.org> on 2026/08/20 17:05:07 UTC
[I] [Improve][Connector-V2][Prometheus] Add bounded retry with backoff to PrometheusWriter.flush() [seatunnel]
surafel58 opened a new issue, #11911:
URL: https://github.com/apache/seatunnel/issues/11911
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
`PrometheusWriter.flush()` issues a single-shot HTTP remote-write POST and throws on any non-204 response or transport error. Every flush trigger calls this same method: the `batch_size` threshold in `write()`, the Zeta engine timer flush (`registerFlushAction`), the checkpoint flush (`prepareCommit()`, added in #11827), and the final flush in `close()`.
Because there is no retry, a transient Prometheus / remote-write outage fails whichever operation triggered the flush. For the checkpoint path this means a brief blip can fail a checkpoint, and on Flink/Spark repeated checkpoint failures can eventually fail the job (subject to the restart strategy). The engine restart strategy does provide recovery, but a small bounded retry with backoff inside `flush()` would absorb short transient failures before escalating to a checkpoint/job failure.
This was raised in the review of #11827 (@SEZ9). It is intentionally scoped as a follow-up because the single-shot behavior is pre-existing (it predates #11827, which only added `prepareCommit()` as one more caller), and a retry should apply uniformly to all flush paths rather than only the checkpoint path.
**Proposal:** add a bounded retry with backoff around the remote-write POST in `PrometheusWriter.flush()` so all flush callers tolerate short transient failures. Retries should be bounded (small max attempt count + capped backoff) and still throw after exhaustion, so a genuine outage still fails loudly rather than blocking indefinitely. Consider reusing an existing retry helper if one is available in the codebase, and making the attempt count / backoff configurable via the sink options.
### Usage Scenario
A streaming job writes metrics to a remote-write endpoint that has a brief transient error (for example a rolling restart of the receiver). Today the in-flight flush fails immediately; on the checkpoint path this fails the checkpoint. With a bounded retry, the flush retries a few times with backoff and usually succeeds, avoiding an unnecessary checkpoint/job failure, while a sustained outage still fails after the retries are exhausted.
### Related issues
- #11827 (added the checkpoint flush; retry raised in review as a follow-up)
- #11816 (the checkpoint-flush issue #11827 closes)
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improve][Connector-V2][Prometheus] Add bounded retry with backoff to PrometheusWriter.flush() [seatunnel]
Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on issue #11911:
URL: https://github.com/apache/seatunnel/issues/11911#issuecomment-5422212030
I would like to work on this, starting now that #11827 has merged. Plan as discussed above: add bounded retry with backoff inside flush() so all callers benefit (batch_size, timer flush, prepareCommit, close); retry transport errors and retryable statuses (5xx and 429) while failing fast on other 4xx; treat a 400 whose body indicates a duplicate or out-of-order sample as a tolerated delivery; reuse an existing retry helper (RetryUtils) if it fits; expose max attempts and backoff (initial + cap) as sink options with conservative defaults, bounded so a genuine outage still fails after exhaustion; and add docs (EN and ZH) plus tests for transient-failure-then-success and retry-exhaustion. PR to follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improve][Connector-V2][Prometheus] Add bounded retry with backoff to PrometheusWriter.flush() [seatunnel]
Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on issue #11911:
URL: https://github.com/apache/seatunnel/issues/11911#issuecomment-5366357231
Expanding the scope of this issue based on @SEZ9's review of #11827. Besides retrying retryable failures, flush() error handling should also cover the replay case, so #11911 now covers two related behaviors:
1. Bounded retry with backoff for retryable failures (transport errors, 5xx, 429), which per the remote-write spec should be retried.
2. Treat a 400 whose body indicates a duplicate (duplicate sample for timestamp) or out-of-order sample as a tolerated/successful delivery rather than throwing. Per the spec 4xx must not be retried, and after a restore the replayed batch can contain samples the receiver already has or that are older than its head; without this, a rejected replay would fail the checkpoint again and loop the job.
Both are flush() response-handling changes, so grouping them here keeps that logic in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improve][Connector-V2][Prometheus] Add bounded retry with backoff to PrometheusWriter.flush() [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11911:
URL: https://github.com/apache/seatunnel/issues/11911#issuecomment-5366245873
Well-scoped follow-up from the #11827 review. Agreed that the retry should live inside flush() so it applies uniformly to all callers (batch_size, timer flush, prepareCommit, close) rather than only the checkpoint path.
Please check whether an existing retry utility in the codebase (e.g., common retry helpers used by other connectors such as RetryUtils) can be reused instead of adding a new implementation. Also consider which HTTP status codes should be retried (e.g., 5xx/429 and transport errors) versus failed fast (4xx client errors).
Suggest exposing max attempts and backoff (initial + cap) as sink options with conservative defaults, keeping retries bounded so a genuine outage still fails loudly. New options should be documented in the Prometheus sink docs and covered by a test simulating a transient failure.
<!-- streview-comment:426 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improve][Connector-V2][Prometheus] Add bounded retry with backoff to PrometheusWriter.flush() [seatunnel]
Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on issue #11911:
URL: https://github.com/apache/seatunnel/issues/11911#issuecomment-5366291438
Thanks @SEZ9, this scoping is helpful and I agree with all of it. Plan for the implementation:
- Put the retry inside flush() so every caller benefits (batch_size, timer flush, prepareCommit, close), not just the checkpoint path.
- Reuse an existing retry helper rather than adding a new one. I will check RetryUtils (org.apache.seatunnel.common.utils) and how sibling connectors use it, and only add something new if nothing fits.
- Retry policy: retry on transport errors and retryable HTTP statuses (5xx and 429), and fail fast on other 4xx client errors since those will not succeed on retry.
- Expose max attempts and backoff (initial + cap) as sink options with conservative defaults, keeping retries bounded so a genuine outage still fails loudly after exhaustion.
- Document the new options in the Prometheus sink docs (EN and ZH) and add a unit test that simulates a transient failure followed by success, plus one that exhausts retries and still throws.
I will open the PR after #11827 merges, since both change flush() and doing this first would conflict with that PR. I would like to work on this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improve][Connector-V2][Prometheus] Add bounded retry with backoff to PrometheusWriter.flush() [seatunnel]
Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on issue #11911:
URL: https://github.com/apache/seatunnel/issues/11911#issuecomment-5424611237
Opened PR #11974 for this. Following the discussion here, it reuses the existing retry / retry_backoff_multiplier_ms / retry_backoff_max_ms options rather than adding new ones: PrometheusSink now wires them into HttpParameter (retry defaults to 3, activating the base transport-IOException retry), and PrometheusWriter.flush() additionally retries 5xx/429 with capped exponential backoff, fails fast on other 4xx, and treats a duplicate/out-of-order 400 as delivered so a replay after restore does not loop the checkpoint. Docs (EN/ZH) and unit tests included. cc @SEZ9 @DanielLeens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improve][Connector-V2][Prometheus] Add bounded retry with backoff to PrometheusWriter.flush() [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #11911: [Improve][Connector-V2][Prometheus] Add bounded retry with backoff to PrometheusWriter.flush()
URL: https://github.com/apache/seatunnel/issues/11911
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org