You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "bharathgunapati (via GitHub)" <gi...@apache.org> on 2026/10/08 14:46:36 UTC
[PR] [FLINK-40278][connector-http] Define HTTP sink batch retry semantics [flink-connector-http]
bharathgunapati opened a new pull request, #57:
URL: https://github.com/apache/flink-connector-http/pull/57
## What is the purpose of the change
After FLINK-40277, clarify that when the HTTP sink submits a batched request, one HTTP response applies to every request entry in that batch: a retryable response retries the whole batch, and a fatal / exhausted response fails the batch as a unit.
## Brief change log
- Document batch-level retry / failure semantics for Table and DataStream docs (en + zh).
- Clarify `SinkHttpClientResponses` and writer error messaging around request entries.
- Add tests covering batch classification and exhausted retryable batch failures.
## Compatibility note
- Behaviour aligns with the existing batch submitter (one response for the submitted batch). Messaging and docs make that explicit.
- Follow-up from the FLINK-40275 umbrella after #54.
## Verifying this change
`mvn -pl flink-connector-http -DskipITs -Dskip.archunit=true -Dspotless.check.skip=true -Dcheckstyle.skip=true -Dtest=HttpSinkWriterTest,BatchRequestSubmitterTest,JavaNetSinkHttpClientTest test`
## Documentation
- Docs updated for sink batch retry semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "bharathgunapati (via GitHub)" <gi...@apache.org>.
bharathgunapati commented on code in PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#discussion_r4371720745
##########
docs/content/docs/connectors/table/http.md:
##########
@@ -633,6 +634,22 @@ If either legacy property is set together with `http.sink.success-codes` or `htt
- `http.sink.retry-codes` (default `500,503,504`) for retryable responses
- `http.sink.ignored-response-codes` for responses that should be treated as successful without retrying
+### Retries and handling errors (Sink)
+Retries are disabled by default: `http.sink.max-retries` defaults to `0`, so the first failed request fails the job.
+
+When `http.sink.max-retries` is greater than `0`, the sink retries requests that fail before a response is received (for
+example, connection errors) and responses whose status code is listed in `http.sink.retry-codes`. The delay between
Review Comment:
Good catch. Added a paragraph on all four pages: with the legacy `http.sink.error.code` / `http.sink.error.code.exclude` properties no status code is retried, a code in the error list fails the job, every other code is success, and only requests that fail before a response is retried. That matches `HttpSinkResponseClassifier`, which returns only SUCCESS or FATAL_FAILURE on the legacy path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser merged PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#discussion_r4232161336
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -82,7 +83,21 @@ These options are specified on the builder using the setProperty method.
| http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| http.sink.request.batch.size | optional | Applicable only for `http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
-
+### Retries and handling errors (Sink)
+Retries are disabled by default: `http.sink.max-retries` defaults to `0`, so the first failed request fails the job.
+
+When `http.sink.max-retries` is greater than `0`, the sink retries requests that fail before a response is received (for
+example, connection errors) and responses whose status code is listed in `http.sink.retry-codes`. The delay between
+retries is set by `http.sink.retry-strategy.type` and the matching fixed-delay or exponential-delay options. Any other
+response that is not successful or ignored is not retried and fails the job. A request that still fails with a retryable
+error after the last retry also fails the job.
+
+In batch submission mode, each HTTP request carries several records and gets one response status, which applies to every
Review Comment:
can we get more granular information?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#discussion_r4232168261
##########
docs/content.zh/docs/connectors/table/http.md:
##########
@@ -632,6 +633,22 @@ If either legacy property is set together with `http.sink.success-codes` or `htt
- `http.sink.retry-codes` (default `500,503,504`) for retryable responses
- `http.sink.ignored-response-codes` for responses that should be treated as successful without retrying
+### Retries and handling errors (Sink)
+Retries are disabled by default: `http.sink.max-retries` defaults to `0`, so the first failed request fails the job.
+
+When `http.sink.max-retries` is greater than `0`, the sink retries requests that fail before a response is received (for
+example, connection errors) and responses whose status code is listed in `http.sink.retry-codes`. The delay between
+retries is set by `http.sink.retry-strategy.type` and the matching fixed-delay or exponential-delay options. Any other
+response that is not successful or ignored is not retried and fails the job. A request that still fails with a retryable
+error after the last retry also fails the job.
+
+In batch submission mode, each HTTP request carries several records and gets one response status, which applies to every
+record in that request. A retryable response retries all of those records, and a fatal response or an exhausted retry
+fails all of them. The sink does not retry individual records inside a batch. Use an idempotent endpoint if retried
+records must not be applied twice. When one flush is split into several HTTP requests (because of
+`http.sink.request.batch.size` or different HTTP methods), each response applies only to the records in its own request.
+Retried records are batched again, so they can be grouped into different HTTP requests than in the first attempt.
Review Comment:
have you manually tested this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#discussion_r4371210146
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -82,7 +83,21 @@ These options are specified on the builder using the setProperty method.
| http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| http.sink.request.batch.size | optional | Applicable only for `http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
-
+### Retries and handling errors (Sink)
+Retries are disabled by default: `http.sink.max-retries` defaults to `0`, so the first failed request fails the job.
+
+When `http.sink.max-retries` is greater than `0`, the sink retries requests that fail before a response is received (for
+example, connection errors) and responses whose status code is listed in `http.sink.retry-codes`. The delay between
+retries is set by `http.sink.retry-strategy.type` and the matching fixed-delay or exponential-delay options. Any other
+response that is not successful or ignored is not retried and fails the job. A request that still fails with a retryable
+error after the last retry also fails the job.
+
+In batch submission mode, each HTTP request carries several records and gets one response status, which applies to every
Review Comment:
Not today: the sink only reads the status code. Per-record results need a parser for the endpoint's response body, out of scope for this ticket. A partly applied batch is resent in full.
##########
docs/content/docs/connectors/table/http.md:
##########
@@ -633,6 +634,22 @@ If either legacy property is set together with `http.sink.success-codes` or `htt
- `http.sink.retry-codes` (default `500,503,504`) for retryable responses
- `http.sink.ignored-response-codes` for responses that should be treated as successful without retrying
+### Retries and handling errors (Sink)
+Retries are disabled by default: `http.sink.max-retries` defaults to `0`, so the first failed request fails the job.
+
+When `http.sink.max-retries` is greater than `0`, the sink retries requests that fail before a response is received (for
+example, connection errors) and responses whose status code is listed in `http.sink.retry-codes`. The delay between
Review Comment:
With the legacy `http.sink.error.code` properties, no status code is retried, only failures without a response. I think that needs a sentence here, on all four pages.
##########
docs/content.zh/docs/connectors/table/http.md:
##########
@@ -632,6 +633,22 @@ If either legacy property is set together with `http.sink.success-codes` or `htt
- `http.sink.retry-codes` (default `500,503,504`) for retryable responses
- `http.sink.ignored-response-codes` for responses that should be treated as successful without retrying
+### Retries and handling errors (Sink)
+Retries are disabled by default: `http.sink.max-retries` defaults to `0`, so the first failed request fails the job.
+
+When `http.sink.max-retries` is greater than `0`, the sink retries requests that fail before a response is received (for
+example, connection errors) and responses whose status code is listed in `http.sink.retry-codes`. The delay between
+retries is set by `http.sink.retry-strategy.type` and the matching fixed-delay or exponential-delay options. Any other
+response that is not successful or ignored is not retried and fails the job. A request that still fails with a retryable
+error after the last retry also fails the job.
+
+In batch submission mode, each HTTP request carries several records and gets one response status, which applies to every
+record in that request. A retryable response retries all of those records, and a fatal response or an exhausted retry
+fails all of them. The sink does not retry individual records inside a batch. Use an idempotent endpoint if retried
+records must not be applied twice. When one flush is split into several HTTP requests (because of
+`http.sink.request.batch.size` or different HTTP methods), each response applies only to the records in its own request.
+Retried records are batched again, so they can be grouped into different HTTP requests than in the first attempt.
Review Comment:
Checked locally with `http.sink.request.batch.size` 2 and three records: only the failed request's records are resent. Please add a test, nothing in the PR covers this yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "bharathgunapati (via GitHub)" <gi...@apache.org>.
bharathgunapati commented on PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#issuecomment-6406689748
Done. The commit now ends with the Generated-by line from the description instead of the Co-Authored-By trailer (head b4e74c7, no content change). I applied the same to #58, which had the identical trailer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#issuecomment-6084691114
in general - if we retry the whole batch is it possible for some of them to be accepted? So retrying would produce duplicates - can we know which event(s) needs to be retried?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "bharathgunapati (via GitHub)" <gi...@apache.org>.
bharathgunapati commented on PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#issuecomment-6400866012
Thanks for the reviews. Addressed in the latest push:
- Legacy error-code behaviour documented on all four pages.
- Two new tests for a flush split into several HTTP requests (batch size and method change), see the thread above for the end-to-end log.
- Test clients are now closed in an `@AfterEach`.
- AI disclosure added to the description per the new template.
@davidradl on the duplicates question: yes, if the server applied part of a batch before answering 500, the retry resends the whole batch. The sink cannot know which records were applied from a status code alone, so the docs ask for an idempotent endpoint when that matters.
The CI run for the new head is waiting for workflow approval. Could one of you approve it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "bharathgunapati (via GitHub)" <gi...@apache.org>.
bharathgunapati commented on code in PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#discussion_r4371717165
##########
docs/content.zh/docs/connectors/table/http.md:
##########
@@ -632,6 +633,22 @@ If either legacy property is set together with `http.sink.success-codes` or `htt
- `http.sink.retry-codes` (default `500,503,504`) for retryable responses
- `http.sink.ignored-response-codes` for responses that should be treated as successful without retrying
+### Retries and handling errors (Sink)
+Retries are disabled by default: `http.sink.max-retries` defaults to `0`, so the first failed request fails the job.
+
+When `http.sink.max-retries` is greater than `0`, the sink retries requests that fail before a response is received (for
+example, connection errors) and responses whose status code is listed in `http.sink.retry-codes`. The delay between
+retries is set by `http.sink.retry-strategy.type` and the matching fixed-delay or exponential-delay options. Any other
+response that is not successful or ignored is not retried and fails the job. A request that still fails with a retryable
+error after the last retry also fails the job.
+
+In batch submission mode, each HTTP request carries several records and gets one response status, which applies to every
+record in that request. A retryable response retries all of those records, and a fatal response or an exhausted retry
+fails all of them. The sink does not retry individual records inside a batch. Use an idempotent endpoint if retried
+records must not be applied twice. When one flush is split into several HTTP requests (because of
+`http.sink.request.batch.size` or different HTTP methods), each response applies only to the records in its own request.
+Retried records are batched again, so they can be grouped into different HTTP requests than in the first attempt.
Review Comment:
@davidradl yes, both with unit tests and end to end.
@MartijnVisser added two tests in `JavaNetSinkHttpClientTest` that go through the real `BatchRequestSubmitter`:
- `batchModeRetriesOnlyTheFailedRequestWhenFlushIsSplit`: batch size 2, three records. The HTTP requests are `[1,2]` -> 500, `[3]` -> 200, then only `[1,2]` is resent.
- `batchModeRegroupsRetriedEntriesIntoNewHttpRequests`: POST, PUT, POST. The two failed POST records come back as one request on retry.
Both record the entries of every HTTP request through the post-request callback and compare by identity.
The same scenarios also run end to end in a Flink mini-cluster against an embedded HTTP server in bharathgunapati/flink-http-sink-examples (`DataStreamHttpSinkIT`, commit 699e9ee). Server log:
```
POST /split-batch -> 500 body=[{"id":1},{"id":2}]
POST /split-batch -> 200 body=[{"id":3}]
POST /split-batch -> 200 body=[{"id":1},{"id":2}]
POST /regroup -> 500 body=[{"id":1}]
PUT /regroup -> 200 body=[{"id":2,"method":"PUT"}]
POST /regroup -> 500 body=[{"id":3}]
POST /regroup -> 200 body=[{"id":1},{"id":3}]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40278][connector-http] Document and test that HTTP sink retries resend the whole batch [flink-connector-http]
Posted by "bharathgunapati (via GitHub)" <gi...@apache.org>.
bharathgunapati commented on code in PR #57:
URL: https://github.com/apache/flink-connector-http/pull/57#discussion_r4371718966
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -82,7 +83,21 @@ These options are specified on the builder using the setProperty method.
| http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| http.sink.request.batch.size | optional | Applicable only for `http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
-
+### Retries and handling errors (Sink)
+Retries are disabled by default: `http.sink.max-retries` defaults to `0`, so the first failed request fails the job.
+
+When `http.sink.max-retries` is greater than `0`, the sink retries requests that fail before a response is received (for
+example, connection errors) and responses whose status code is listed in `http.sink.retry-codes`. The delay between
+retries is set by `http.sink.retry-strategy.type` and the matching fixed-delay or exponential-delay options. Any other
+response that is not successful or ignored is not retried and fails the job. A request that still fails with a retryable
+error after the last retry also fails the job.
+
+In batch submission mode, each HTTP request carries several records and gets one response status, which applies to every
Review Comment:
Confirming what Martijn said: the sink classifies a response from its status code only, so it cannot tell which records of a batch the server applied. Per-record retry needs a response-body parser for the target, which is out of scope here. The docs state the limitation and recommend an idempotent endpoint for that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org