You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "surafel58 (via GitHub)" <gi...@apache.org> on 2026/08/26 11:27:34 UTC
[PR] [Improve][Connector-V2][Prometheus] Retry retryable remote-write failures and tolerate duplicate/out-of-order rejections [seatunnel]
surafel58 opened a new pull request, #11974:
URL: https://github.com/apache/seatunnel/pull/11974
### Purpose of this pull request
Closes #11911. Follow-up to #11827 (raised by @SEZ9 in that review).
The checkpoint flush added in #11827 makes remote-write success gate the checkpoint. Two consequences motivated this change:
- A single transient failure (transport error, or a `5xx`/`429` response) could fail a checkpoint, and on Flink's default `tolerableCheckpointFailureNumber=0` restart the whole job.
- After a restore, the source replays from the last successful checkpoint and re-sends the buffered samples. If the receiver rejects a re-sent sample as a duplicate or out-of-order (`400`), the flush would fail the checkpoint again and loop the job.
This adds bounded retry and tolerates the replay case, **reusing the connector's existing HTTP retry options** (`retry`, `retry_backoff_multiplier_ms`, `retry_backoff_max_ms` from `HttpCommonOptions`) rather than introducing new ones.
What changed:
- `PrometheusSink` wires the existing retry options into `HttpParameter`, which activates the base `HttpClientProvider`'s transport-`IOException` retry (previously `retry` was never set, so it was effectively disabled). `retry` defaults to `3` when unset, so retries are on by default.
- `PrometheusWriter.flush()` additionally retries retryable HTTP statuses (`5xx` and `429`) — the base retryer cannot see these because they come back as responses, not exceptions — with exponential backoff capped at `retry_backoff_max_ms`. Other `4xx` responses fail fast. A `400` the receiver reports as a duplicate/out-of-order sample is treated as delivered, so a replay after restore does not fail the checkpoint or loop the job. The delivery guarantee remains at-least-once.
### Does this PR introduce _any_ user-facing change?
Yes, a behavior improvement, no new options.
- Before: the Prometheus sink never wired the `retry` option into the HTTP client, so a transient remote-write failure was not retried at all; a `5xx` response or a replay-time duplicate `400` failed the flush (and, via the #11827 checkpoint flush, could fail/loop the job).
- After: transient transport failures and `5xx`/`429` responses are retried up to `retry` times (default `3`) with exponential backoff, and a duplicate/out-of-order `400` is tolerated as delivered. The `retry` option now covers `5xx`/`429` in addition to `IOException`.
The Prometheus sink docs (EN and ZH) are updated: the `retry` option description and default, and the "Checkpoint Flush and Failure Handling" section.
### How was this patch tested?
Added unit tests in `PrometheusWriterTest`:
- `shouldRetryRetryableFailureThenSucceed` — a `503` then `204` delivers the batch (two attempts).
- `shouldThrowAfterRetriesExhausted` — a persistent `503` throws after `retry + 1` attempts.
- `shouldFailFastOnNonRetryable4xx` — a `403` throws after a single attempt despite `retry=5`.
- `shouldTreatDuplicateOrOutOfOrder400AsDelivered` — a duplicate `400` returns normally and clears the buffer.
The full `connector-prometheus` module test suite passes locally (14 tests, 0 failures) on JDK 8.
### Check list
* [x] If necessary, please update the documentation to describe the new feature. (Prometheus sink docs updated, EN and ZH)
* [ ] New Jar binary package: N/A
* [ ] New connector: N/A (modifies an existing connector; no new options, no plugin-mapping / seatunnel-dist / plugin_config changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2][Prometheus] Retry retryable remote-write failures and tolerate duplicate/out-of-order rejections [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11974:
URL: https://github.com/apache/seatunnel/pull/11974#issuecomment-5564023640
@surafel58 nothing has changed on my end since my last comment — the current head (`7e01c3b1`) still has all four review-round findings verified fixed against source, CI (`Build`, `Notify test workflow`, `labeler`) was confirmed green on that same head, and I have no outstanding source-level concerns (the `Retry-After`-on-429 idea we discussed is a fine fast-follow, not a blocker).
I'm a comment-only reviewer on this repo, so I can't set the formal approval myself. Adding my +1 here for visibility — cc @davidzollo, this one's ready for a write-access maintainer to merge whenever you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2][Prometheus] Retry retryable remote-write failures and tolerate duplicate/out-of-order rejections [seatunnel]
Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11974:
URL: https://github.com/apache/seatunnel/pull/11974#issuecomment-5477814378
Thanks @DanielLeens for the careful re-verification and the approve, and for the extra idempotency / retry-storm / critical-path analysis.
Good call on Issue 5 (honoring a Retry-After header on 429). I agree it makes the sink a better citizen against a receiver actively signaling backpressure. I would like to take it as a fast-follow rather than fold it into this PR: the current HttpResponse only carries the status code and body, not arbitrary response headers, so reading Retry-After means plumbing response headers through HttpResponse / HttpClientProvider in the shared connector-http-base module. That is a small but cross-module change that is cleaner in its own PR than expanding this one. I will open a follow-up issue for it and reference this thread.
Everything else is addressed and CI is green, so this is ready for a committer with write access to merge whenever convenient. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2][Prometheus] Retry retryable remote-write failures and tolerate duplicate/out-of-order rejections [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11974:
URL: https://github.com/apache/seatunnel/pull/11974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2][Prometheus] Retry retryable remote-write failures and tolerate duplicate/out-of-order rejections [seatunnel]
Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11974:
URL: https://github.com/apache/seatunnel/pull/11974#issuecomment-5560409296
Can it be merged @davidzollo ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2][Prometheus] Retry retryable remote-write failures and tolerate duplicate/out-of-order rejections [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11974:
URL: https://github.com/apache/seatunnel/pull/11974#issuecomment-5480366459
Thanks @surafel58 — agreed on Issue 5. Deferring `Retry-After` handling to a follow-up is the right call: it needs `HttpResponse`/`HttpClientProvider` in `connector-http-base` to carry response headers first, which is a shared-module change that shouldn't ride along with this fix. Please link the follow-up issue here once it's filed so it's easy to track from this thread.
From my side there's nothing further blocking: both review rounds' findings are verified fixed at `7e01c3b1`, and I just re-confirmed live that CI (`Build`, `Notify test workflow`, `labeler`) is green on that same head SHA. As a comment-only reviewer I can't set the formal approval, so this is now waiting on a committer with write access to approve and merge — cc @davidzollo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2][Prometheus] Retry retryable remote-write failures and tolerate duplicate/out-of-order rejections [seatunnel]
Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11974:
URL: https://github.com/apache/seatunnel/pull/11974#issuecomment-5462858730
Thanks for the thorough review @DanielLeens. Addressed all four points in 7e01c3b1:
- **Issue 1 (attempt/backoff consistency):** flush() now uses `maxAttempts = max(1, retry)` so the status-code retry uses the same total-attempt budget as the base transport retryer (`stopAfterAttempt(retry)`) rather than `retry + 1`. Added a comment noting the base uses Fibonacci backoff while the status path uses capped exponential backoff, both bounded by the same `retry_backoff_*` options. (With default `retry=3`, both paths now do 3 total attempts.)
- **Issue 2 (dup/OOO heuristic):** raised the tolerated-400 log from INFO to WARN, and noted in both the code comment and the EN/ZH docs that it is a best-effort match on receiver-specific body wording, so a receiver returning 400 with different wording is not recognized and a false positive would drop the batch.
- **Issue 3 (dropped cause):** the retries-exhausted `PrometheusConnectorException` now passes the original exception as its cause instead of only folding the message text.
- **Issue 4 (per-attempt visibility):** added a WARN log on each retried status-code attempt, matching the base transport retryer's per-attempt logging.
Also added the suggested test `shouldRetainBufferAfterRetriesExhausted`, which asserts the batch stays buffered after an exhausted flush and is re-sent by a later flush (pinning down at-least-once). Full module suite passes locally (15 tests). Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2][Prometheus] Retry retryable remote-write failures and tolerate duplicate/out-of-order rejections [seatunnel]
Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11974:
URL: https://github.com/apache/seatunnel/pull/11974#issuecomment-5477835240
@davidzollo when you have a moment, could a committer with write access help merge this? It closes #11911 and is a follow-up to the merged #11827. @DanielLeens has reviewed it across two rounds and confirmed ready to merge (all four prior findings verified fixed against source; the one remaining item, honoring Retry-After on 429, is a non-blocking fast-follow I will handle in a separate PR). CI Build is green and GitHub reports mergeable. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org