You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "mariyageorge01 (via GitHub)" <gi...@apache.org> on 2026/03/02 17:31:43 UTC
[PR] [hotfix] Fix grammar [flink-connector-http]
mariyageorge01 opened a new pull request, #28:
URL: https://github.com/apache/flink-connector-http/pull/28
Fix grammar and avoid ambiguity in documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix grammar [flink-connector-http]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28:
URL: https://github.com/apache/flink-connector-http/pull/28#discussion_r2883725715
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -72,15 +72,15 @@ These options are specified on the builder using the setProperty method.
| flink.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
| flink.connector.http.sink.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |
| flink.connector.http.sink.writer.thread-pool.size | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used. |
-| flink.connector.http.sink.writer.request.mode | optional | Sets Http Sink request submission mode. Two modes are available to select, `single` and `batch` which is the default mode if option is not specified. |
+| flink.connector.http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| flink.connector.http.sink.request.batch.size | optional | Applicable only for `flink.connector.http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
### Batch submission mode
-By default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property and has value of 500.
-The `maxBatchSize` property sets maximal number of events that will be buffered by Flink runtime before passing it to Http Sink for processing.
+By default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property.
Review Comment:
I don't understand this sentence before! when we say batch size - I assume there is a batch size property, if so we should refer to it by property name. The way it reads is ambiguous as we day the batch size is 500 bit the same as a configurable property ? I assume we are saying that the default values for both these batch size configuration properties is 500, but the way it is phrased is not clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix grammar [flink-connector-http]
Posted by "mariyageorge01 (via GitHub)" <gi...@apache.org>.
mariyageorge01 commented on code in PR #28:
URL: https://github.com/apache/flink-connector-http/pull/28#discussion_r2902172193
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -72,15 +72,15 @@ These options are specified on the builder using the setProperty method.
| flink.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
| flink.connector.http.sink.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |
| flink.connector.http.sink.writer.thread-pool.size | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used. |
-| flink.connector.http.sink.writer.request.mode | optional | Sets Http Sink request submission mode. Two modes are available to select, `single` and `batch` which is the default mode if option is not specified. |
+| flink.connector.http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| flink.connector.http.sink.request.batch.size | optional | Applicable only for `flink.connector.http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
### Batch submission mode
-By default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property and has value of 500.
-The `maxBatchSize` property sets maximal number of events that will be buffered by Flink runtime before passing it to Http Sink for processing.
+By default, the `flink.connector.http.sink.request.batch.size` property is set to 500, which matches the default value of Http Sink's `maxBatchSize` property.
+The `maxBatchSize` property sets the maximum number of events that will be buffered by Flink runtime before passing it to Http Sink for processing.
Review Comment:
Updated the doc with more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix grammar [flink-connector-http]
Posted by "mariyageorge01 (via GitHub)" <gi...@apache.org>.
mariyageorge01 commented on code in PR #28:
URL: https://github.com/apache/flink-connector-http/pull/28#discussion_r2905914764
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -92,8 +98,10 @@ HttpSink.<String>builder()
.build();
```
-### Single submission mode
-In this mode every processed event is submitted as individual HTTP POST/PUT request.
+In this example, the default `sink.batch.max-size` of 500 is used. When Flink flushes 500 records, the HTTP Sink splits them into 10 HTTP POST requests (50 records each).
Review Comment:
I haven't changed the content for single submission mode.
I added a common header for Batch submission mode and Single submission mode (like in Table API doc) and formatted the headings for better clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix grammar [flink-connector-http]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28:
URL: https://github.com/apache/flink-connector-http/pull/28#discussion_r2905587207
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -72,15 +70,23 @@ These options are specified on the builder using the setProperty method.
| flink.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
| flink.connector.http.sink.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |
| flink.connector.http.sink.writer.thread-pool.size | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used. |
-| flink.connector.http.sink.writer.request.mode | optional | Sets Http Sink request submission mode. Two modes are available to select, `single` and `batch` which is the default mode if option is not specified. |
+| flink.connector.http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| flink.connector.http.sink.request.batch.size | optional | Applicable only for `flink.connector.http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
-### Batch submission mode
+### Request submission
+HTTP Sink by default submits events in batch. The submission mode can be changed using `flink.connector.http.sink.writer.request.mode` property using `single` or `batch` as property value.
+
+#### Batch submission mode
+
+The HTTP Sink uses a two-stage batching mechanism that decouples the rate and size of incoming records from how they are sent as HTTP requests.
-By default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property and has value of 500.
-The `maxBatchSize` property sets maximal number of events that will be buffered by Flink runtime before passing it to Http Sink for processing.
+**Stage 1 - Flink Runtime Buffering**: Controlled by `sink.batch.max-size` property (default: 500). Flink buffers records internally until reaching `sink.batch.max-size` records, `sink.flush-buffer.size`, or timeout `sink.flush-buffer.timeout`. When triggered, Flink flushes the buffered records to the HTTP Sink.
Review Comment:
nit : to the HTTP Sink. - > to the network.
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -72,15 +70,23 @@ These options are specified on the builder using the setProperty method.
| flink.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
| flink.connector.http.sink.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |
| flink.connector.http.sink.writer.thread-pool.size | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used. |
-| flink.connector.http.sink.writer.request.mode | optional | Sets Http Sink request submission mode. Two modes are available to select, `single` and `batch` which is the default mode if option is not specified. |
+| flink.connector.http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| flink.connector.http.sink.request.batch.size | optional | Applicable only for `flink.connector.http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
-### Batch submission mode
+### Request submission
+HTTP Sink by default submits events in batch. The submission mode can be changed using `flink.connector.http.sink.writer.request.mode` property using `single` or `batch` as property value.
+
+#### Batch submission mode
+
+The HTTP Sink uses a two-stage batching mechanism that decouples the rate and size of incoming records from how they are sent as HTTP requests.
-By default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property and has value of 500.
-The `maxBatchSize` property sets maximal number of events that will be buffered by Flink runtime before passing it to Http Sink for processing.
+**Stage 1 - Flink Runtime Buffering**: Controlled by `sink.batch.max-size` property (default: 500). Flink buffers records internally until reaching `sink.batch.max-size` records, `sink.flush-buffer.size`, or timeout `sink.flush-buffer.timeout`. When triggered, Flink flushes the buffered records to the HTTP Sink.
Review Comment:
nit : to the HTTP Sink. - > to the network.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix grammar [flink-connector-http]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl merged PR #28:
URL: https://github.com/apache/flink-connector-http/pull/28
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix grammar [flink-connector-http]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28:
URL: https://github.com/apache/flink-connector-http/pull/28#discussion_r2889038903
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -72,15 +72,15 @@ These options are specified on the builder using the setProperty method.
| flink.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
| flink.connector.http.sink.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |
| flink.connector.http.sink.writer.thread-pool.size | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used. |
-| flink.connector.http.sink.writer.request.mode | optional | Sets Http Sink request submission mode. Two modes are available to select, `single` and `batch` which is the default mode if option is not specified. |
+| flink.connector.http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| flink.connector.http.sink.request.batch.size | optional | Applicable only for `flink.connector.http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
### Batch submission mode
-By default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property and has value of 500.
-The `maxBatchSize` property sets maximal number of events that will be buffered by Flink runtime before passing it to Http Sink for processing.
+By default, the `flink.connector.http.sink.request.batch.size` property is set to 500, which matches the default value of Http Sink's `maxBatchSize` property.
+The `maxBatchSize` property sets the maximum number of events that will be buffered by Flink runtime before passing it to Http Sink for processing.
Review Comment:
I was asking myself :what do these mean. It looks like the connector buffers records internally up to size maxBatchSize, this will be flushed when it reaches maxBatchSize or sink.flush-buffer.timeout is met.
Example without timeout:
maxBatchSize = 500
batch.size = 100
What happens:
Flink buffers up to 500 records (maxBatchSize)
When flush is triggered due to reaching 500, those 500 records are split into 5 HTTP requests
So this mechanism decouples the rate and size of records coming in vs how we send them as http requests.
We should try to say this in the text.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix grammar [flink-connector-http]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28:
URL: https://github.com/apache/flink-connector-http/pull/28#discussion_r2905630595
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -92,8 +98,10 @@ HttpSink.<String>builder()
.build();
```
-### Single submission mode
-In this mode every processed event is submitted as individual HTTP POST/PUT request.
+In this example, the default `sink.batch.max-size` of 500 is used. When Flink flushes 500 records, the HTTP Sink splits them into 10 HTTP POST requests (50 records each).
Review Comment:
I think the single submission mode, is does as the original text said. The `PerRequestSubmitter` class does not look at the batching configuration. This is all done in the `BatchRequestSubmitter ` class.
I suggest we add what the config parameter is that means we are in Single submission mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hotfix] Fix grammar [flink-connector-http]
Posted by "mariyageorge01 (via GitHub)" <gi...@apache.org>.
mariyageorge01 commented on code in PR #28:
URL: https://github.com/apache/flink-connector-http/pull/28#discussion_r2883840178
##########
docs/content.zh/docs/connectors/datastream/http.md:
##########
@@ -72,15 +72,15 @@ These options are specified on the builder using the setProperty method.
| flink.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |
| flink.connector.http.sink.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |
| flink.connector.http.sink.writer.thread-pool.size | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used. |
-| flink.connector.http.sink.writer.request.mode | optional | Sets Http Sink request submission mode. Two modes are available to select, `single` and `batch` which is the default mode if option is not specified. |
+| flink.connector.http.sink.writer.request.mode | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| flink.connector.http.sink.request.batch.size | optional | Applicable only for `flink.connector.http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |
### Batch submission mode
-By default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property and has value of 500.
-The `maxBatchSize` property sets maximal number of events that will be buffered by Flink runtime before passing it to Http Sink for processing.
+By default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property.
Review Comment:
I added the property name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org