You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "m1a2st (via GitHub)" <gi...@apache.org> on 2024/09/08 15:37:50 UTC
[PR] KAFKA-17492: Can't register 3.9+ broker to 3.8 controller since 3.8 controller assume the default version of feature is zero [kafka]
m1a2st opened a new pull request, #17128:
URL: https://github.com/apache/kafka/pull/17128
Jira: https://issues.apache.org/jira/browse/KAFKA-17492
Due to the BrokerRegistrationRequest minVersion is setting from 0 to 1, thus `ClusterControlManger#processRegistrationFeature` check the version will throw an error, fix it by below suggestion
1. make 3.9+ broker be able to register to 3.8 controller
skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4
2. make 3.9+ controller be able to accept unknown features from future broker
skip check of unknown features. see following sample code
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2341430553
Thanks folks for fixing this 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1750743769
##########
clients/src/main/java/org/apache/kafka/common/requests/BrokerRegistrationRequest.java:
##########
@@ -46,21 +46,7 @@ public short oldestAllowedVersion() {
@Override
public BrokerRegistrationRequest build(short version) {
- if (version < 4) {
- // Workaround for KAFKA-17011: for BrokerRegistrationRequest versions older than 4,
- // translate minSupportedVersion = 0 to minSupportedVersion = 1.
- BrokerRegistrationRequestData newData = data.duplicate();
- for (Iterator<BrokerRegistrationRequestData.Feature> iter = newData.features().iterator();
- iter.hasNext(); ) {
- BrokerRegistrationRequestData.Feature feature = iter.next();
- if (feature.minSupportedVersion() == 0) {
- feature.setMinSupportedVersion((short) 1);
- }
- }
- return new BrokerRegistrationRequest(newData, version);
- } else {
- return new BrokerRegistrationRequest(data, version);
- }
+ return new BrokerRegistrationRequest(data, version);
Review Comment:
Sorry, I don't understand the change here. The correct change (that matches the PR description) would be to remove features that had a minVersion of 0 when handling older RPCs, right? But instead the code just seems to pass everything through, ignoring RPC version. What am I missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2339343202
I did some cleanup and added the `removeIf` simplification that was suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2341509975
The last commit does not impact source code, so I'm going to merge it to test full e2e with #17084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1752293086
##########
clients/src/main/resources/common/message/BrokerRegistrationRequest.json:
##########
@@ -48,7 +48,7 @@
]
},
{ "name": "Features", "type": "[]Feature",
- "about": "The features on this broker. Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.", "versions": "0+", "fields": [
+ "about": "The features on this broker. Note: in v0-v3, features MinSupportedVersion = 0 are omitted.", "versions": "0+", "fields": [
Review Comment:
features MinSupportedVersion = 0 are omitted => features with MinSupportedVersion = 0 are omitted
##########
clients/src/main/resources/common/message/ApiVersionsResponse.json:
##########
@@ -47,7 +47,7 @@
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "SupportedFeatures", "type": "[]SupportedFeatureKey", "ignorable": true,
"versions": "3+", "tag": 0, "taggedVersions": "3+",
- "about": "Features supported by the broker. Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.",
+ "about": "Features supported by the broker. Note: in v0-v3, features MinSupportedVersion = 0 are omitted.",
Review Comment:
features MinSupportedVersion = 0 are omitted => features with MinSupportedVersion = 0 are omitted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1751152536
##########
clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsResponse.java:
##########
@@ -289,19 +289,19 @@ private static SupportedFeatureKeyCollection maybeFilterSupportedFeatureKeys(
SupportedFeatureKeyCollection converted = new SupportedFeatureKeyCollection();
for (Map.Entry<String, SupportedVersionRange> feature : latestSupportedFeatures.features().entrySet()) {
final SupportedVersionRange versionRange = feature.getValue();
- final SupportedFeatureKey key = new SupportedFeatureKey();
- key.setName(feature.getKey());
if (alterV0 && versionRange.min() == 0) {
- // Some older clients will have deserialization problems if a feature's
- // minimum supported level is 0. Therefore, when preparing ApiVersionResponse
- // at versions less than 4, we must set the minimum version for these features
- // to 1 rather than 0. See KAFKA-17011 for details.
- key.setMinVersion((short) 1);
+ // Note: For API versions v0-v3, features with MinSupportedVersion = 0
Review Comment:
This comment is not accurate. Could you consider using @cmccabe's comment?
```
// Some older clients will have deserialization problems if a feature's
// minimum supported level is 0. Therefore, when preparing ApiVersionResponse
// at versions less than 4, we must omit these features. See KAFKA-17492.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492: Can't register 3.9+ broker to 3.8 controller since 3.8 controller assume the default version of feature is zero [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1750624496
##########
clients/src/main/java/org/apache/kafka/common/requests/BrokerRegistrationRequest.java:
##########
@@ -46,21 +46,7 @@ public short oldestAllowedVersion() {
@Override
public BrokerRegistrationRequest build(short version) {
- if (version < 4) {
- // Workaround for KAFKA-17011: for BrokerRegistrationRequest versions older than 4,
- // translate minSupportedVersion = 0 to minSupportedVersion = 1.
- BrokerRegistrationRequestData newData = data.duplicate();
- for (Iterator<BrokerRegistrationRequestData.Feature> iter = newData.features().iterator();
- iter.hasNext(); ) {
- BrokerRegistrationRequestData.Feature feature = iter.next();
- if (feature.minSupportedVersion() == 0) {
- feature.setMinSupportedVersion((short) 1);
- }
- }
- return new BrokerRegistrationRequest(newData, version);
- } else {
- return new BrokerRegistrationRequest(data, version);
- }
+ return new BrokerRegistrationRequest(data, version);
Review Comment:
for another: please update the following comment:
```
// Workaround for KAFKA-17011: for BrokerRegistrationRequest versions older than 4,
// translate minSupportedVersion = 0 to minSupportedVersion = 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1750771062
##########
clients/src/main/java/org/apache/kafka/common/requests/BrokerRegistrationRequest.java:
##########
@@ -47,14 +47,14 @@ public short oldestAllowedVersion() {
@Override
public BrokerRegistrationRequest build(short version) {
if (version < 4) {
- // Workaround for KAFKA-17011: for BrokerRegistrationRequest versions older than 4,
- // translate minSupportedVersion = 0 to minSupportedVersion = 1.
+ // Workaround for KAFKA-17492: for BrokerRegistrationRequest versions older than 4,
+ // remove features with minSupportedVersion = 0.
BrokerRegistrationRequestData newData = data.duplicate();
for (Iterator<BrokerRegistrationRequestData.Feature> iter = newData.features().iterator();
iter.hasNext(); ) {
BrokerRegistrationRequestData.Feature feature = iter.next();
if (feature.minSupportedVersion() == 0) {
- feature.setMinSupportedVersion((short) 1);
+ iter.remove();
Review Comment:
How about using `removeIf`?
```java
newData.features().removeIf(feature -> feature.minSupportedVersion() == 0);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2339574666
@m1a2st please add 3.8 version to following e2e:
https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/zookeeper_migration_test.py#L89
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2342408525
I will update this KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-1022%3A+Formatting+and+Updating+Features
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1750743248
##########
metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java:
##########
@@ -501,14 +501,16 @@ BrokerFeature processRegistrationFeature(
FinalizedControllerFeatures finalizedFeatures,
BrokerRegistrationRequestData.Feature feature
) {
- int defaultVersion = feature.name().equals(MetadataVersion.FEATURE_NAME) ? 1 : 0; // The default value for MetadataVersion is 1 not 0.
- short finalized = finalizedFeatures.versionOrDefault(feature.name(), (short) defaultVersion);
- if (!VersionRange.of(feature.minSupportedVersion(), feature.maxSupportedVersion()).contains(finalized)) {
- throw new UnsupportedVersionException("Unable to register because the broker " +
- "does not support finalized version " + finalized + " of " + feature.name() +
- ". The broker wants a version between " + feature.minSupportedVersion() + " and " +
- feature.maxSupportedVersion() + ", inclusive.");
- }
+ finalizedFeatures.get(feature.name()).ifPresent(
Review Comment:
@m1a2st please take a look at https://github.com/apache/kafka/pull/16421#discussion_r1750717947. We don't need to remove the check of `ClusterControlManager`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1751149794
##########
clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsResponse.java:
##########
@@ -289,19 +289,17 @@ private static SupportedFeatureKeyCollection maybeFilterSupportedFeatureKeys(
SupportedFeatureKeyCollection converted = new SupportedFeatureKeyCollection();
for (Map.Entry<String, SupportedVersionRange> feature : latestSupportedFeatures.features().entrySet()) {
final SupportedVersionRange versionRange = feature.getValue();
- final SupportedFeatureKey key = new SupportedFeatureKey();
- key.setName(feature.getKey());
if (alterV0 && versionRange.min() == 0) {
// Some older clients will have deserialization problems if a feature's
// minimum supported level is 0. Therefore, when preparing ApiVersionResponse
- // at versions less than 4, we must set the minimum version for these features
- // to 1 rather than 0. See KAFKA-17011 for details.
- key.setMinVersion((short) 1);
+ // at versions less than 4, we must omit these features. See KAFKA-17492.
Review Comment:
@m1a2st Could you please address @junrao comment? That includes following files.
https://github.com/apache/kafka/blob/b22e848ddef975785846af13685ee50ce40d4df9/clients/src/main/resources/common/message/ApiVersionsResponse.json#L50
https://github.com/apache/kafka/blob/b22e848ddef975785846af13685ee50ce40d4df9/clients/src/main/resources/common/message/ApiVersionsResponse.json#L50
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1751495140
##########
clients/src/main/resources/common/message/ApiVersionsResponse.json:
##########
@@ -47,7 +47,7 @@
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "SupportedFeatures", "type": "[]SupportedFeatureKey", "ignorable": true,
"versions": "3+", "tag": 0, "taggedVersions": "3+",
- "about": "Features supported by the broker. Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.",
+ "about": "Features supported by the broker. Note: in v0-v3, features blocked with MinSupportedVersion = 0.",
Review Comment:
Pardon me, I don't catch "features blocked with MinSupportedVersion = 0."
Also, please fix `BrokerRegistrationRequest.json` too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2341287540
The failed tests pass on my local, and following links are the related jira. The new e2e works on my local. I will merge it to trunk and 3.9 later if no objections.
https://issues.apache.org/jira/browse/KAFKA-17514
https://issues.apache.org/jira/browse/KAFKA-16174
https://issues.apache.org/jira/browse/KAFKA-15104
https://issues.apache.org/jira/browse/KAFKA-14533
https://issues.apache.org/jira/browse/KAFKA-17515
https://issues.apache.org/jira/browse/KAFKA-15772
https://issues.apache.org/jira/browse/KAFKA-15411
https://issues.apache.org/jira/browse/KAFKA-16501
https://issues.apache.org/jira/browse/KAFKA-15146
https://issues.apache.org/jira/browse/KAFKA-16377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2338835994
I pushed a PR which does what the PR description says (skip features with minVersion of 0)...
take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492: Can't register 3.9+ broker to 3.8 controller since 3.8 controller assume the default version of feature is zero [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2336762735
> make 3.9+ broker be able to register to 3.8 controller
skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4
noticed that I had raised a question for it: https://github.com/apache/kafka/pull/16421#discussion_r1748904248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1750744353
##########
metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java:
##########
@@ -501,14 +501,16 @@ BrokerFeature processRegistrationFeature(
FinalizedControllerFeatures finalizedFeatures,
BrokerRegistrationRequestData.Feature feature
) {
- int defaultVersion = feature.name().equals(MetadataVersion.FEATURE_NAME) ? 1 : 0; // The default value for MetadataVersion is 1 not 0.
- short finalized = finalizedFeatures.versionOrDefault(feature.name(), (short) defaultVersion);
- if (!VersionRange.of(feature.minSupportedVersion(), feature.maxSupportedVersion()).contains(finalized)) {
- throw new UnsupportedVersionException("Unable to register because the broker " +
- "does not support finalized version " + finalized + " of " + feature.name() +
- ". The broker wants a version between " + feature.minSupportedVersion() + " and " +
- feature.maxSupportedVersion() + ", inclusive.");
- }
+ finalizedFeatures.get(feature.name()).ifPresent(
Review Comment:
This change is incorrect. If the minVersion is 0, the feature must be set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2339424501
Thanks for @cmccabe, @chia7712, @junrao helping for this issue, I will complete this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1751505082
##########
clients/src/main/resources/common/message/ApiVersionsResponse.json:
##########
@@ -47,7 +47,7 @@
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "SupportedFeatures", "type": "[]SupportedFeatureKey", "ignorable": true,
"versions": "3+", "tag": 0, "taggedVersions": "3+",
- "about": "Features supported by the broker. Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.",
+ "about": "Features supported by the broker. Note: in v0-v3, features blocked with MinSupportedVersion = 0.",
Review Comment:
> Pardon me, I don't catch "features blocked with MinSupportedVersion = 0."
I change it to omit, it is more accurancy in this case
##########
clients/src/main/resources/common/message/ApiVersionsResponse.json:
##########
@@ -47,7 +47,7 @@
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "SupportedFeatures", "type": "[]SupportedFeatureKey", "ignorable": true,
"versions": "3+", "tag": 0, "taggedVersions": "3+",
- "about": "Features supported by the broker. Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.",
+ "about": "Features supported by the broker. Note: in v0-v3, features blocked with MinSupportedVersion = 0.",
Review Comment:
> Pardon me, I don't catch "features blocked with MinSupportedVersion = 0."
I change it to omit, it is more accurancy in this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2341924550
@m1a2st : Could we update https://cwiki.apache.org/confluence/display/KAFKA/KIP-1022%3A+Formatting+and+Updating+Features and the mailing list with the changes in this PR?
```
In order to resolve this dilemma, we will create a new version 4 for ApiVersionsRequest. Clients which send v4 promise to be able to handle ranges including 0. Clients which send v3 will never see a range that starts with 0. Such ranges will be translated into ranges starting with 1 instead.
Similarly, for BrokerRegistrationRequest, we will create a new version 4. Servers which accept v4 promise to be able to handle ranges including 0. Servers which do not will get ranges that only start with 1, rather than 0.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1751071116
##########
clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsResponse.java:
##########
@@ -289,19 +289,17 @@ private static SupportedFeatureKeyCollection maybeFilterSupportedFeatureKeys(
SupportedFeatureKeyCollection converted = new SupportedFeatureKeyCollection();
for (Map.Entry<String, SupportedVersionRange> feature : latestSupportedFeatures.features().entrySet()) {
final SupportedVersionRange versionRange = feature.getValue();
- final SupportedFeatureKey key = new SupportedFeatureKey();
- key.setName(feature.getKey());
if (alterV0 && versionRange.min() == 0) {
// Some older clients will have deserialization problems if a feature's
// minimum supported level is 0. Therefore, when preparing ApiVersionResponse
- // at versions less than 4, we must set the minimum version for these features
- // to 1 rather than 0. See KAFKA-17011 for details.
- key.setMinVersion((short) 1);
+ // at versions less than 4, we must omit these features. See KAFKA-17492.
Review Comment:
We need to change the comment like the following in ApiVersionResponse.json and BrokerRegistrationRequest.json.
`Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.
`
##########
core/src/test/scala/unit/kafka/server/ApiVersionsResponseIntegrationTest.scala:
##########
@@ -72,4 +72,12 @@ class ApiVersionsResponseIntegrationTest extends BaseRequestTest {
assertEquals(name, key.name())
assertEquals(expectedMinVersion, key.minVersion())
}
+
+ def assertFeatureMissing(
Review Comment:
Could this be private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492: Can't register 3.9+ broker to 3.8 controller since 3.8 controller assume the default version of feature is zero [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1750604290
##########
clients/src/main/java/org/apache/kafka/common/requests/BrokerRegistrationRequest.java:
##########
@@ -46,21 +46,7 @@ public short oldestAllowedVersion() {
@Override
public BrokerRegistrationRequest build(short version) {
- if (version < 4) {
- // Workaround for KAFKA-17011: for BrokerRegistrationRequest versions older than 4,
- // translate minSupportedVersion = 0 to minSupportedVersion = 1.
- BrokerRegistrationRequestData newData = data.duplicate();
- for (Iterator<BrokerRegistrationRequestData.Feature> iter = newData.features().iterator();
- iter.hasNext(); ) {
- BrokerRegistrationRequestData.Feature feature = iter.next();
- if (feature.minSupportedVersion() == 0) {
- feature.setMinSupportedVersion((short) 1);
- }
- }
- return new BrokerRegistrationRequest(newData, version);
- } else {
- return new BrokerRegistrationRequest(data, version);
- }
+ return new BrokerRegistrationRequest(data, version);
Review Comment:
We need to skip features with minSupportedVersion of 0 in v3 or less.
We need to do the same in ApiVersionResponse.
We need to change the comment like the following in ApiVersionResponse.json and BrokerRegistrationRequest.json.
`Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17128:
URL: https://github.com/apache/kafka/pull/17128#issuecomment-2339036805
Run the `transactions_upgrade_test.py`, `transactions_mixed_versions_test.py`, and `kraft_upgrade_test.py` with this patch and #17084 on my local. all pass.
I will merge it if no failed tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1751528209
##########
clients/src/main/resources/common/message/BrokerRegistrationRequest.json:
##########
@@ -48,7 +48,7 @@
]
},
{ "name": "Features", "type": "[]Feature",
- "about": "The features on this broker. Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.", "versions": "0+", "fields": [
+ "about": "The features on this broker. Note: in v0-v3, features omit with MinSupportedVersion = 0.", "versions": "0+", "fields": [
Review Comment:
How about "features with MinSupportedVersion = 0 are omitted"?
##########
clients/src/main/resources/common/message/ApiVersionsResponse.json:
##########
@@ -47,7 +47,7 @@
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "SupportedFeatures", "type": "[]SupportedFeatureKey", "ignorable": true,
"versions": "3+", "tag": 0, "taggedVersions": "3+",
- "about": "Features supported by the broker. Note: in v0-v3, features with MinSupportedVersion = 0 show up with MinSupportedVersion = 1.",
+ "about": "Features supported by the broker. Note: in v0-v3, features omit with MinSupportedVersion = 0.",
Review Comment:
How about "features with MinSupportedVersion = 0 are omitted"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17128:
URL: https://github.com/apache/kafka/pull/17128#discussion_r1751149794
##########
clients/src/main/java/org/apache/kafka/common/requests/ApiVersionsResponse.java:
##########
@@ -289,19 +289,17 @@ private static SupportedFeatureKeyCollection maybeFilterSupportedFeatureKeys(
SupportedFeatureKeyCollection converted = new SupportedFeatureKeyCollection();
for (Map.Entry<String, SupportedVersionRange> feature : latestSupportedFeatures.features().entrySet()) {
final SupportedVersionRange versionRange = feature.getValue();
- final SupportedFeatureKey key = new SupportedFeatureKey();
- key.setName(feature.getKey());
if (alterV0 && versionRange.min() == 0) {
// Some older clients will have deserialization problems if a feature's
// minimum supported level is 0. Therefore, when preparing ApiVersionResponse
- // at versions less than 4, we must set the minimum version for these features
- // to 1 rather than 0. See KAFKA-17011 for details.
- key.setMinVersion((short) 1);
+ // at versions less than 4, we must omit these features. See KAFKA-17492.
Review Comment:
@m1a2st Could you please address @junrao comment? That includes following files.
https://github.com/apache/kafka/blob/b22e848ddef975785846af13685ee50ce40d4df9/clients/src/main/resources/common/message/ApiVersionsResponse.json#L50
https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/BrokerRegistrationRequest.json#L51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17492 skip features with minVersion of 0 instead of replacing 0 with 1 when BrokerRegistrationRequest < 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #17128:
URL: https://github.com/apache/kafka/pull/17128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org