You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "FrankYang0529 (via GitHub)" <gi...@apache.org> on 2025/03/19 10:24:33 UTC
[PR] KAFKA-18845: Fix flaky QuorumControllerTest#testUncleanShutdownBrokerElrEnabled [kafka]
FrankYang0529 opened a new pull request, #19240:
URL: https://github.com/apache/kafka/pull/19240
There're two root causes:
1. When we unclean shutdown `brokerToBeTheLeader`, we didn't wait for the result. That means when we send heartbeat to unfence broker, it has chance to use stale broker epoch to send the request. [0]
2. We use different replica directory to unclean shutdown broker. Even if broker is unfenced, it cannot get an online directory, so the `brokerToBeTheLeader` cannot be elected as a new leader. [1]
[0] https://github.com/apache/kafka/blob/a5325e029e2493f22925af99482ad9fa1eb06947/metadata/src/test/java/org/apache/kafka/controller/QuorumControllerTest.java#L484-L497
[1] https://github.com/apache/kafka/blob/a5325e029e2493f22925af99482ad9fa1eb06947/metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java#L2470-L2477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18845: Fix flaky QuorumControllerTest#testUncleanShutdownBrokerElrEnabled [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah merged PR #19240:
URL: https://github.com/apache/kafka/pull/19240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18845: Fix flaky QuorumControllerTest#testUncleanShutdownBrokerElrEnabled [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #19240:
URL: https://github.com/apache/kafka/pull/19240#discussion_r2005767333
##########
metadata/src/test/java/org/apache/kafka/controller/QuorumControllerTest.java:
##########
@@ -474,24 +475,28 @@ public void testUncleanShutdownBrokerElrEnabled() throws Throwable {
setClusterId(active.clusterId()).
setFeatures(features).
setIncarnationId(Uuid.randomUuid()).
- setLogDirs(List.of(Uuid.randomUuid())).
+ setLogDirs(List.of(brokerLogDirs.get(brokerToUncleanShutdown))).
setListeners(listeners));
brokerEpochs.put(brokerToUncleanShutdown, reply.get().epoch());
partition = active.replicationControl().getPartition(topicIdFoo, 0);
assertArrayEquals(new int[]{brokerToBeTheLeader}, partition.elr, partition.toString());
assertArrayEquals(lastKnownElr, partition.lastKnownElr, partition.toString());
// Unclean shutdown should not remove the last known ELR members.
- active.registerBroker(
+ CompletableFuture<BrokerRegistrationReply> replyLeader = active.registerBroker(
anonymousContextFor(ApiKeys.BROKER_REGISTRATION),
new BrokerRegistrationRequestData().
setBrokerId(brokerToBeTheLeader).
setClusterId(active.clusterId()).
setFeatures(features).
setIncarnationId(Uuid.randomUuid()).
setPreviousBrokerEpoch(brokerEpochs.get(brokerToBeTheLeader)).
- setLogDirs(List.of(Uuid.randomUuid())).
+ setLogDirs(List.of(brokerLogDirs.get(brokerToBeTheLeader))).
setListeners(listeners));
+ brokerEpochs.put(brokerToBeTheLeader, replyLeader.get().epoch());
+ partition = active.replicationControl().getPartition(topicIdFoo, 0);
+ assertArrayEquals(new int[]{brokerToBeTheLeader}, partition.elr, partition.toString());
Review Comment:
Add more content for it. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18845: Fix flaky QuorumControllerTest#testUncleanShutdownBrokerElrEnabled [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #19240:
URL: https://github.com/apache/kafka/pull/19240#discussion_r2004040049
##########
metadata/src/test/java/org/apache/kafka/controller/QuorumControllerTest.java:
##########
@@ -474,24 +475,28 @@ public void testUncleanShutdownBrokerElrEnabled() throws Throwable {
setClusterId(active.clusterId()).
setFeatures(features).
setIncarnationId(Uuid.randomUuid()).
- setLogDirs(List.of(Uuid.randomUuid())).
+ setLogDirs(List.of(brokerLogDirs.get(brokerToUncleanShutdown))).
setListeners(listeners));
brokerEpochs.put(brokerToUncleanShutdown, reply.get().epoch());
partition = active.replicationControl().getPartition(topicIdFoo, 0);
assertArrayEquals(new int[]{brokerToBeTheLeader}, partition.elr, partition.toString());
assertArrayEquals(lastKnownElr, partition.lastKnownElr, partition.toString());
// Unclean shutdown should not remove the last known ELR members.
- active.registerBroker(
+ CompletableFuture<BrokerRegistrationReply> replyLeader = active.registerBroker(
anonymousContextFor(ApiKeys.BROKER_REGISTRATION),
new BrokerRegistrationRequestData().
setBrokerId(brokerToBeTheLeader).
setClusterId(active.clusterId()).
setFeatures(features).
setIncarnationId(Uuid.randomUuid()).
setPreviousBrokerEpoch(brokerEpochs.get(brokerToBeTheLeader)).
- setLogDirs(List.of(Uuid.randomUuid())).
+ setLogDirs(List.of(brokerLogDirs.get(brokerToBeTheLeader))).
setListeners(listeners));
+ brokerEpochs.put(brokerToBeTheLeader, replyLeader.get().epoch());
+ partition = active.replicationControl().getPartition(topicIdFoo, 0);
+ assertArrayEquals(new int[]{brokerToBeTheLeader}, partition.elr, partition.toString());
Review Comment:
How about a real message instead of just the partition if the assertion fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18845: Fix flaky QuorumControllerTest#testUncleanShutdownBrokerElrEnabled [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #19240:
URL: https://github.com/apache/kafka/pull/19240#discussion_r2004033069
##########
metadata/src/test/java/org/apache/kafka/controller/QuorumControllerTest.java:
##########
@@ -376,10 +375,12 @@ public void testElrEnabledByDefault() throws Throwable {
}
}
- @Flaky("KAFKA-18845")
Review Comment:
@FrankYang0529 we need to keep the annotation here until the test proves itself to no longer be flaky. I have updated the wiki https://cwiki.apache.org/confluence/display/KAFKA/Flaky+Test+Management to reflect this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18845: Fix flaky QuorumControllerTest#testUncleanShutdownBrokerElrEnabled [kafka]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #19240:
URL: https://github.com/apache/kafka/pull/19240#discussion_r2005766732
##########
metadata/src/test/java/org/apache/kafka/controller/QuorumControllerTest.java:
##########
@@ -376,10 +375,12 @@ public void testElrEnabledByDefault() throws Throwable {
}
}
- @Flaky("KAFKA-18845")
Review Comment:
Thanks for the reminder. I added Flaky back and will remove it after we have sufficient data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org