You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "frankvicky (via GitHub)" <gi...@apache.org> on 2024/11/08 06:51:41 UTC
[PR] KAFKA-17927: Disallow users to configure `max.in.flight.requests.perconnection` bigger than 5 [kafka]
frankvicky opened a new pull request, #17717:
URL: https://github.com/apache/kafka/pull/17717
JIRA: KAFKA-17927
> As idempotent producer is the default behavior in 4.0, we should add a upper bound (5) to `max.in.flight.requests.per.connection`.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17927: Disallow users to configure `max.in.flight.requests.perconnection` bigger than 5 [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #17717:
URL: https://github.com/apache/kafka/pull/17717#discussion_r1833848063
##########
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java:
##########
@@ -582,13 +582,8 @@ private void postProcessAndValidateIdempotenceConfigs(final Map<String, Object>
final int inFlightConnection = this.getInt(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION);
if (MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE < inFlightConnection) {
- if (userConfiguredIdempotence) {
- throw new ConfigException("Must set " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + " to at most 5" +
- " to use the idempotent producer.");
- }
- log.warn("Idempotence will be disabled because {} is set to {}, which is greater than 5. " +
- "Please note that in v4.0.0 and onward, this will become an error.", MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, inFlightConnection);
- shouldDisableIdempotence = true;
+ throw new ConfigException("To use the idempotent producer, " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION +
Review Comment:
How about updating the validator from `atLeast(1)` to `between(1, 5)`, so we don't need the check here?
```java
.define(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION,
Type.INT,
5,
between(1, 5),
Importance.LOW,
MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_DOC)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17927: Disallow users to configure `max.in.flight.requests.perconnection` bigger than 5 [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #17717:
URL: https://github.com/apache/kafka/pull/17717#discussion_r1833893802
##########
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java:
##########
@@ -582,13 +582,8 @@ private void postProcessAndValidateIdempotenceConfigs(final Map<String, Object>
final int inFlightConnection = this.getInt(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION);
if (MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE < inFlightConnection) {
- if (userConfiguredIdempotence) {
- throw new ConfigException("Must set " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + " to at most 5" +
- " to use the idempotent producer.");
- }
- log.warn("Idempotence will be disabled because {} is set to {}, which is greater than 5. " +
- "Please note that in v4.0.0 and onward, this will become an error.", MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, inFlightConnection);
- shouldDisableIdempotence = true;
+ throw new ConfigException("To use the idempotent producer, " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION +
Review Comment:
IIUC, broker side only contains at most 5 batches. Event if the `enable.idempotence` is `false`, the maximum `MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION` can't be more than `5`.
https://github.com/apache/kafka/blob/af53758746485daa06c4d4366036c04f7f9f0e63/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateEntry.java#L29-L36
https://github.com/apache/kafka/blob/af53758746485daa06c4d4366036c04f7f9f0e63/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateEntry.java#L107-L110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17927: Disallow users to configure `max.in.flight.requests.perconnection` bigger than 5 [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #17717:
URL: https://github.com/apache/kafka/pull/17717#discussion_r1833859572
##########
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java:
##########
@@ -582,13 +582,8 @@ private void postProcessAndValidateIdempotenceConfigs(final Map<String, Object>
final int inFlightConnection = this.getInt(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION);
if (MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE < inFlightConnection) {
- if (userConfiguredIdempotence) {
- throw new ConfigException("Must set " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + " to at most 5" +
- " to use the idempotent producer.");
- }
- log.warn("Idempotence will be disabled because {} is set to {}, which is greater than 5. " +
- "Please note that in v4.0.0 and onward, this will become an error.", MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, inFlightConnection);
- shouldDisableIdempotence = true;
+ throw new ConfigException("To use the idempotent producer, " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION +
Review Comment:
Hmmm… This logic indeed covers the case where `enable.idempotence` is set to `true`, but we don’t want this constraint if `enable.idempotence` is set to `false`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17927: Disallow users to configure `max.in.flight.requests.perconnection` bigger than 5 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17717:
URL: https://github.com/apache/kafka/pull/17717#issuecomment-2480363251
@frankvicky please update `upgrade.html` :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17927: Disallow users to configure `max.in.flight.requests.perconnection` bigger than 5 [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #17717:
URL: https://github.com/apache/kafka/pull/17717#discussion_r1833893802
##########
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java:
##########
@@ -582,13 +582,8 @@ private void postProcessAndValidateIdempotenceConfigs(final Map<String, Object>
final int inFlightConnection = this.getInt(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION);
if (MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE < inFlightConnection) {
- if (userConfiguredIdempotence) {
- throw new ConfigException("Must set " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + " to at most 5" +
- " to use the idempotent producer.");
- }
- log.warn("Idempotence will be disabled because {} is set to {}, which is greater than 5. " +
- "Please note that in v4.0.0 and onward, this will become an error.", MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, inFlightConnection);
- shouldDisableIdempotence = true;
+ throw new ConfigException("To use the idempotent producer, " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION +
Review Comment:
IIUC, broker side only contains at most 5 batches. If the `enable.idempotence` is `false` and the config allows `max.in.flight.requests.perconnection` to be more than 5, there may have data loss.
https://github.com/apache/kafka/blob/af53758746485daa06c4d4366036c04f7f9f0e63/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateEntry.java#L29-L36
https://github.com/apache/kafka/blob/af53758746485daa06c4d4366036c04f7f9f0e63/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateEntry.java#L107-L110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17927: Disallow users to configure `max.in.flight.requests.perconnection` bigger than 5 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #17717:
URL: https://github.com/apache/kafka/pull/17717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17927: Disallow users to configure `max.in.flight.requests.perconnection` bigger than 5 [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #17717:
URL: https://github.com/apache/kafka/pull/17717#discussion_r1833859572
##########
clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java:
##########
@@ -582,13 +582,8 @@ private void postProcessAndValidateIdempotenceConfigs(final Map<String, Object>
final int inFlightConnection = this.getInt(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION);
if (MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_FOR_IDEMPOTENCE < inFlightConnection) {
- if (userConfiguredIdempotence) {
- throw new ConfigException("Must set " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + " to at most 5" +
- " to use the idempotent producer.");
- }
- log.warn("Idempotence will be disabled because {} is set to {}, which is greater than 5. " +
- "Please note that in v4.0.0 and onward, this will become an error.", MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, inFlightConnection);
- shouldDisableIdempotence = true;
+ throw new ConfigException("To use the idempotent producer, " + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION +
Review Comment:
Hmmm… This logic indeed covers the case where `enable.idempotence` is set to `true`, but we don’t want this constraint if `enable.idempotence` is explicit set to `false`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org