You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mumrah (via GitHub)" <gi...@apache.org> on 2023/03/09 17:54:48 UTC
[GitHub] [kafka] mumrah opened a new pull request, #13372: MINOR: Improved error handling in ZK migration
mumrah opened a new pull request, #13372:
URL: https://github.com/apache/kafka/pull/13372
This patch fixes many small issues to improve error handling and logging during the ZK migration. A test was added to simulate a ZK session expiration to ensure the correctness of the migration driver.
With this change, ZK errors thrown during the migration will not hit the fault handler registered with with KRaftMigrationDriver, but they will be logged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #13372:
URL: https://github.com/apache/kafka/pull/13372#issuecomment-1470482252
@mumrah : `KRaftMigrationDriverTest.testMigrationWithClientException` is failing for me with this PR. Can you take a look?
```
Gradle Test Run :metadata:test > Gradle Test Executor 3 > KRaftMigrationDriverTest > testMigrationWithClientException(boolean) > org.apache.kafka.metadata.migration.KRaftMigrationDriverTest.testMigrationWithClientException(boolean)[1] FAILED
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
at app//org.apache.kafka.metadata.migration.KRaftMigrationDriverTest.testMigrationWithClientException(KRaftMigrationDriverTest.java:358)```
(Also Jenkins did something silly again, but hopefully the next build will work.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mumrah commented on pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #13372:
URL: https://github.com/apache/kafka/pull/13372#issuecomment-1472608583
@cmccabe the latest test run fixed the unit test. The other failures will be addressed by #13407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134589780
##########
core/src/main/scala/kafka/zk/ZkMigrationClient.scala:
##########
@@ -47,24 +46,44 @@ import scala.jdk.CollectionConverters._
*/
class ZkMigrationClient(zkClient: KafkaZkClient) extends MigrationClient with Logging {
- override def getOrCreateMigrationRecoveryState(initialState: ZkMigrationLeadershipState): ZkMigrationLeadershipState = {
- zkClient.createTopLevelPaths()
- zkClient.getOrCreateMigrationState(initialState)
+ @throws(classOf[MigrationClientException])
+ def wrapZkException[T](fn: => T): T = {
Review Comment:
Should we make it so that `wrapZkException(wrapZkException(e))` is the same as `wrapZkException(e)`? In case we inadvertently end up nesting these somehow.
(Concretely I'm just saying: let's add a pass-through case for MigrationClientAuthException / MigrationClientException)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134591764
##########
core/src/test/scala/integration/kafka/zk/ZkMigrationIntegrationTest.scala:
##########
@@ -85,7 +85,7 @@ class ZkMigrationIntegrationTest {
quotas.add(new ClientQuotaAlteration(
new ClientQuotaEntity(Map("ip" -> "8.8.8.8").asJava),
List(new ClientQuotaAlteration.Op("connection_creation_rate", 10.0)).asJava))
- admin.alterClientQuotas(quotas)
+ admin.alterClientQuotas(quotas).all().get(60, TimeUnit.SECONDS)
Review Comment:
nitpick: Could also do 1 TimeUnit.MINUTES
:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134591050
##########
core/src/main/scala/kafka/zk/ZkMigrationClient.scala:
##########
@@ -47,24 +46,44 @@ import scala.jdk.CollectionConverters._
*/
class ZkMigrationClient(zkClient: KafkaZkClient) extends MigrationClient with Logging {
Review Comment:
Can we add JavaDoc that you need to wrap zkClient uses in wrapZkException?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134594893
##########
metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java:
##########
@@ -132,26 +132,45 @@ private boolean isControllerQuorumReadyForMigration() {
return true;
}
+ private boolean imageDoesNotContainAllBrokers(MetadataImage image, Set<Integer> brokerIds) {
+ for (BrokerRegistration broker : image.cluster().brokers().values()) {
+ if (broker.isMigratingZkBroker()) {
+ brokerIds.remove(broker.id());
+ }
+ }
+ return !brokerIds.isEmpty();
+ }
+
private boolean areZkBrokersReadyForMigration() {
if (image == MetadataImage.EMPTY) {
// TODO maybe add WAIT_FOR_INITIAL_METADATA_PUBLISH state to avoid this kind of check?
log.info("Waiting for initial metadata publish before checking if Zk brokers are registered.");
return false;
}
- Set<Integer> zkRegisteredZkBrokers = zkMigrationClient.readBrokerIdsFromTopicAssignments();
- for (BrokerRegistration broker : image.cluster().brokers().values()) {
- if (broker.isMigratingZkBroker()) {
- zkRegisteredZkBrokers.remove(broker.id());
- }
+
+ // First check the brokers registered in ZK
+ Set<Integer> zkBrokerRegistrations = new HashSet<>(zkMigrationClient.readBrokerIds());
Review Comment:
is it necessary to copy the `Set<Integer>` we get back from `zkMigrationClient.readBrokerIds`? Seems like it should be fine to use directly...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134590727
##########
core/src/main/scala/kafka/zk/ZkMigrationClient.scala:
##########
@@ -211,19 +230,21 @@ class ZkMigrationClient(zkClient: KafkaZkClient) extends MigrationClient with Lo
}
}
- override def readAllMetadata(batchConsumer: Consumer[util.List[ApiMessageAndVersion]],
- brokerIdConsumer: Consumer[Integer]): Unit = {
+ override def readAllMetadata(
+ batchConsumer: Consumer[util.List[ApiMessageAndVersion]],
+ brokerIdConsumer: Consumer[Integer]
+ ): Unit = wrapZkException {
Review Comment:
I think it would be better to have the `wrapZkException` blocks inside `migrateTopics`, `migrateBrokerConfigs`, `migrateClientQuotas`, `migrateProducerId`. Then we can have a simple mental rule like: if you do something with zkClient, you need to wrap it. And not have to analyze control flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134596431
##########
metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java:
##########
@@ -269,15 +288,24 @@ public void close() throws Exception {
// Events handled by Migration Driver.
abstract class MigrationEvent implements EventQueue.Event {
+ @SuppressWarnings("ThrowableNotThrown")
@Override
public void handleException(Throwable e) {
- if (e instanceof RejectedExecutionException) {
- log.info("Not processing {} because the event queue is closed.", this);
+ if (e instanceof MigrationClientAuthException) {
+ KRaftMigrationDriver.this.faultHandler.handleFault("Encountered client auth error in " + this, e);
+ } else if (e instanceof MigrationClientException) {
+ log.debug(String.format("Encountered client error during event %s. Will retry.", this), e.getCause());
Review Comment:
I think this should be `info`... for now, at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134595911
##########
metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java:
##########
@@ -269,15 +288,24 @@ public void close() throws Exception {
// Events handled by Migration Driver.
abstract class MigrationEvent implements EventQueue.Event {
+ @SuppressWarnings("ThrowableNotThrown")
@Override
public void handleException(Throwable e) {
- if (e instanceof RejectedExecutionException) {
- log.info("Not processing {} because the event queue is closed.", this);
+ if (e instanceof MigrationClientAuthException) {
+ KRaftMigrationDriver.this.faultHandler.handleFault("Encountered client auth error in " + this, e);
Review Comment:
maybe "Encountered ZooKeeper authentication error" to be as specific as possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134593684
##########
metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java:
##########
@@ -93,7 +93,7 @@ public KRaftMigrationDriver(
this.log = LoggerFactory.getLogger(KRaftMigrationDriver.class);
this.migrationState = MigrationDriverState.UNINITIALIZED;
this.migrationLeadershipState = ZkMigrationLeadershipState.EMPTY;
- this.eventQueue = new KafkaEventQueue(Time.SYSTEM, new LogContext("KRaftMigrationDriver"), "kraft-migration");
+ this.eventQueue = new KafkaEventQueue(Time.SYSTEM, new LogContext("KRaftMigrationDriver "), "kraft-migration");
Review Comment:
btw `KRaftMigrationDriver.log` should also be created by this logContext, so we know what node our logs are coming from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mumrah commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1135709978
##########
metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java:
##########
@@ -132,26 +132,45 @@ private boolean isControllerQuorumReadyForMigration() {
return true;
}
+ private boolean imageDoesNotContainAllBrokers(MetadataImage image, Set<Integer> brokerIds) {
+ for (BrokerRegistration broker : image.cluster().brokers().values()) {
+ if (broker.isMigratingZkBroker()) {
+ brokerIds.remove(broker.id());
+ }
+ }
+ return !brokerIds.isEmpty();
+ }
+
private boolean areZkBrokersReadyForMigration() {
if (image == MetadataImage.EMPTY) {
// TODO maybe add WAIT_FOR_INITIAL_METADATA_PUBLISH state to avoid this kind of check?
log.info("Waiting for initial metadata publish before checking if Zk brokers are registered.");
return false;
}
- Set<Integer> zkRegisteredZkBrokers = zkMigrationClient.readBrokerIdsFromTopicAssignments();
- for (BrokerRegistration broker : image.cluster().brokers().values()) {
- if (broker.isMigratingZkBroker()) {
- zkRegisteredZkBrokers.remove(broker.id());
- }
+
+ // First check the brokers registered in ZK
+ Set<Integer> zkBrokerRegistrations = new HashSet<>(zkMigrationClient.readBrokerIds());
Review Comment:
Ah, yea this is needed because the Set returned by the client was an immutable Scala thing originally. I'll change the client to return a mutable set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe merged pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe merged PR #13372:
URL: https://github.com/apache/kafka/pull/13372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13372: MINOR: Improved error handling in ZK migration
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13372:
URL: https://github.com/apache/kafka/pull/13372#discussion_r1134592688
##########
metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java:
##########
@@ -93,7 +93,7 @@ public KRaftMigrationDriver(
this.log = LoggerFactory.getLogger(KRaftMigrationDriver.class);
this.migrationState = MigrationDriverState.UNINITIALIZED;
this.migrationLeadershipState = ZkMigrationLeadershipState.EMPTY;
- this.eventQueue = new KafkaEventQueue(Time.SYSTEM, new LogContext("KRaftMigrationDriver"), "kraft-migration");
+ this.eventQueue = new KafkaEventQueue(Time.SYSTEM, new LogContext("KRaftMigrationDriver "), "kraft-migration");
Review Comment:
can we add nodeId here to the LogContext? Maybe like "[KRaftMigrationDriver nodeId=1] " etc. etc. to match our usual style in the controller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org