You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mumrah (via GitHub)" <gi...@apache.org> on 2023/06/20 18:37:43 UTC
[GitHub] [kafka] mumrah opened a new pull request, #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
mumrah opened a new pull request, #13890:
URL: https://github.com/apache/kafka/pull/13890
Since the KRaft controller is sending LISR to ZK brokers while in migration mode, we can't skip the leader epoch bump introduced in #13765.
This patch creates an exception for the new logic if we're running in ZK migration mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13890:
URL: https://github.com/apache/kafka/pull/13890#discussion_r1237291087
##########
metadata/src/main/java/org/apache/kafka/controller/PartitionChangeBuilder.java:
##########
@@ -273,13 +282,15 @@ private void tryElection(PartitionChangeRecord record) {
* that required that the leader epoch be bump whenever the ISR shrank. In MV 3.6 this leader
* bump is not required when the ISR shrinks. Note, that the leader epoch is never increased if
* the ISR expanded.
+ *
+ * When the controller is in ZK migration mode, the leader epoch must be bumped during ISR
Review Comment:
Maybe add something like "even in MV 3.6 and later, when the controller is in ZK migration mode..."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13890:
URL: https://github.com/apache/kafka/pull/13890#discussion_r1235713408
##########
metadata/src/main/java/org/apache/kafka/controller/PartitionChangeBuilder.java:
##########
@@ -87,13 +89,16 @@ public PartitionChangeBuilder(
Uuid topicId,
int partitionId,
IntPredicate isAcceptableLeader,
- MetadataVersion metadataVersion
+ MetadataVersion metadataVersion,
+ boolean zkMigrationEnabled
Review Comment:
I think this should be descriptive, like "bumpLeaderEpochOnIsrShrink". Then it can be initialized to `!metadataVersion.isSkipLeaderEpochBumpSupported()`. Then we can have a setter function that overrides it, and we can do
```
if (zkMigrationEnabled) {
builder.setBumpLeaderEpochOnIsrShrink(true);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13890:
URL: https://github.com/apache/kafka/pull/13890#discussion_r1239113984
##########
metadata/src/main/java/org/apache/kafka/controller/PartitionChangeBuilder.java:
##########
@@ -140,6 +144,11 @@ public PartitionChangeBuilder setTargetLeaderRecoveryState(LeaderRecoveryState t
return this;
}
+ public PartitionChangeBuilder setBumpLeaderEpochOnIsrShrink(boolean bumpLeaderEpochOnIsrShrink) {
+ this.bumpLeaderEpochOnIsrShrink = bumpLeaderEpochOnIsrShrink;
+ return this;
+ }
Review Comment:
> Hmm. Doesn't this ignore the MV?
No. The MV is consulted in the constructor.
> Checking the MV is important to make sure that all of the brokers support this feature before the controller can skip leader epoch bumps
This function is only used to enable bumping leader epoch, which is always safe to do, even on older MVs.
If you are really concerned about foolproofing this you could make a PR that changes this function from `setBumpLeaderEpochOnIsrShrink` to `enableBumpLeaderEpochOnIsrShrink` (i.e. remove the option to disable this way)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mumrah merged pull request #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah merged PR #13890:
URL: https://github.com/apache/kafka/pull/13890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13890:
URL: https://github.com/apache/kafka/pull/13890#discussion_r1237291930
##########
metadata/src/test/java/org/apache/kafka/controller/PartitionChangeBuilderTest.java:
##########
@@ -217,6 +228,16 @@ public void testTriggerLeaderEpochBumpIfNeeded() {
new PartitionChangeRecord(),
1
);
+
+ // Shrinking the ISR while in ZK migration mode does increase the leader epoch
Review Comment:
Add a reference to KAFKA-15109. Also a reason: I guess it's because the ZK brokers need this information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13890:
URL: https://github.com/apache/kafka/pull/13890#discussion_r1239115751
##########
metadata/src/main/java/org/apache/kafka/controller/PartitionChangeBuilder.java:
##########
@@ -140,6 +144,11 @@ public PartitionChangeBuilder setTargetLeaderRecoveryState(LeaderRecoveryState t
return this;
}
+ public PartitionChangeBuilder setBumpLeaderEpochOnIsrShrink(boolean bumpLeaderEpochOnIsrShrink) {
+ this.bumpLeaderEpochOnIsrShrink = bumpLeaderEpochOnIsrShrink;
+ return this;
+ }
Review Comment:
Oops, the intention here was to only call this API when ZK migration was enabled.
To fix this, I’d suggest removing setBumpLeaderEpochOnIsrShrink and changing it to enableBumpLeaderEpochOnIsrShrink. Basically, the API can be used to enable the bump, but it cannot be used to disable. This should avoid this kind of problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jsancio commented on a diff in pull request #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #13890:
URL: https://github.com/apache/kafka/pull/13890#discussion_r1239082472
##########
metadata/src/main/java/org/apache/kafka/controller/PartitionChangeBuilder.java:
##########
@@ -140,6 +144,11 @@ public PartitionChangeBuilder setTargetLeaderRecoveryState(LeaderRecoveryState t
return this;
}
+ public PartitionChangeBuilder setBumpLeaderEpochOnIsrShrink(boolean bumpLeaderEpochOnIsrShrink) {
+ this.bumpLeaderEpochOnIsrShrink = bumpLeaderEpochOnIsrShrink;
+ return this;
+ }
Review Comment:
Hmm. Doesn't this ignore the MV? Checking the MV is important to make sure that all of the brokers support this feature before the controller can skip leader epoch bumps. I think you want the following code:
```java
public PartitionChangeBuilder setBumpLeaderEpochOnIsrShrink(boolean bumpLeaderEpochOnIsrShrink) {
// If the MV doesn't support skipping leader epoch bump the caller cannot override it.
if (metadataVersion.isSkipLeaderEpochBumpSupported()) {
this.bumpLeaderEpochOnIsrShrink = bumpLeaderEpochOnIsrShrink;
}
return this;
}
```
Because of this, it is not clear to me if there is even the right API. You probably want `setInZKMigrationMode` to make all of this more explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mumrah commented on pull request #13890: KAFKA-15109 Don't skip leader epoch bump while in migration mode
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #13890:
URL: https://github.com/apache/kafka/pull/13890#issuecomment-1599310423
This system test was failing on trunk, with this patch it's passing again.
```
================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.11.3
session_id: 2023-06-20--002
run time: 5 minutes 48.107 seconds
tests run: 2
passed: 2
flaky: 0
failed: 0
ignored: 0
================================================================================
test_id: kafkatest.tests.core.zookeeper_migration_test.TestMigration.test_online_migration.roll_controller=False
status: PASS
run time: 2 minutes 51.979 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.core.zookeeper_migration_test.TestMigration.test_online_migration.roll_controller=True
status: PASS
run time: 2 minutes 55.931 seconds
--------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org