You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "gaurav-narula (via GitHub)" <gi...@apache.org> on 2024/01/25 13:56:32 UTC

[PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

gaurav-narula opened a new pull request, #15262:
URL: https://github.com/apache/kafka/pull/15262

   This change ensures we check that the broker is running in Kraft mode or is undergoing a migration while handling a failure for metadata.log.dir.
   
   This avoids halting a broker in ZK mode when the metadata.log.dir fails. When unconfigured, it defaults to the first log directory.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on PR #15262:
URL: https://github.com/apache/kafka/pull/15262#issuecomment-1936789321

   The issues started for 3.7 on the same day so it is one of the 3 commits backported feb 2 
   https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=America%2FLos_Angeles&search.values=3.7&tests.container=kafka.server.LogDirFailureTest
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on PR #15262:
URL: https://github.com/apache/kafka/pull/15262#issuecomment-1936787524

   Hey folks. We've seen a large increase in LogDirFailureTest after this PR. Can we take a look and see if something here caused it?
   
   gradle enterprise: https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=America%2FLos_Angeles&search.values=trunk&tests.container=kafka.server.LogDirFailureTest
   jira: https://issues.apache.org/jira/browse/KAFKA-16225


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on code in PR #15262:
URL: https://github.com/apache/kafka/pull/15262#discussion_r1473210077


##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2586,7 +2586,7 @@ class ReplicaManager(val config: KafkaConfig,
     // retrieve the UUID here because logManager.handleLogDirFailure handler removes it
     val uuid = logManager.directoryId(dir)
     logManager.handleLogDirFailure(dir)
-    if (dir == config.metadataLogDir) {
+    if (dir == new File(config.metadataLogDir).getAbsolutePath && (zkClient.isEmpty || config.migrationEnabled)) {

Review Comment:
   Addressed in https://github.com/apache/kafka/pull/15262/commits/b28e21a5f96d580e4463e20eb73738a157eb440c



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on PR #15262:
URL: https://github.com/apache/kafka/pull/15262#issuecomment-1911962461

   > I think the impact of this bug is that if there are more than 1 log dirs in ZK broker, and when the 1st of them is failed, we will shutdown the broker unexpectedly. But if there's only 1 log dir, it should be fine to shutdown the broker since no available online log dir. (just have a strange log saying _Shutdown broker because the metadata log dir_. Is my understanding correct?
   > 
   > If so, I think the change makes sense. Could you add tests for it?
   
   That's correct. The case where all log directories fail is handled in `logManager.handleLogDirFailure` which is invoked on line 2588.
   
   Will follow up with a test shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #15262:
URL: https://github.com/apache/kafka/pull/15262#discussion_r1473174695


##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2586,7 +2586,7 @@ class ReplicaManager(val config: KafkaConfig,
     // retrieve the UUID here because logManager.handleLogDirFailure handler removes it
     val uuid = logManager.directoryId(dir)
     logManager.handleLogDirFailure(dir)
-    if (dir == config.metadataLogDir) {
+    if (dir == new File(config.metadataLogDir).getAbsolutePath && (zkClient.isEmpty || config.migrationEnabled)) {

Review Comment:
   This is not the correct check... you should check `config.processRoles`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on PR #15262:
URL: https://github.com/apache/kafka/pull/15262#issuecomment-1910271761

   CC: @cmccabe @showuon 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on PR #15262:
URL: https://github.com/apache/kafka/pull/15262#issuecomment-1912105919

   @showuon added a test with commit https://github.com/apache/kafka/pull/15262/commits/d0858deb5500ebdec071bc06ca6f98726ae6f602. Please take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #15262:
URL: https://github.com/apache/kafka/pull/15262#issuecomment-1938859808

   > The issues started for 3.7 on the same day so it is one of the 3 commits backported feb 2 https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=America%2FLos_Angeles&search.values=3.7&tests.container=kafka.server.LogDirFailureTest
   
   Hi @jolshan just introduced a fix here #15354  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "OmniaGM (via GitHub)" <gi...@apache.org>.
OmniaGM commented on PR #15262:
URL: https://github.com/apache/kafka/pull/15262#issuecomment-1914824513

   Tests Compilation is failing with 
   ```
   [2024-01-26T16:18:05.648Z] > Task :core:compileTestScala
   [2024-01-26T16:18:05.648Z] [Error] /home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-15262/core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:6410:23: Invalid literal number
   [2024-01-26T16:18:06.555Z] [Error] /home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka-pr_PR-15262/core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:6413:5: ')' expected but '}' found.
   [2024-01-26T16:18:07.582Z] two errors found
   [2024-01-26T16:18:07.582Z] 
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe merged PR #15262:
URL: https://github.com/apache/kafka/pull/15262


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-16195: ignore metadata.log.dir failure in ZK mode [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #15262:
URL: https://github.com/apache/kafka/pull/15262#discussion_r1473174695


##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2586,7 +2586,7 @@ class ReplicaManager(val config: KafkaConfig,
     // retrieve the UUID here because logManager.handleLogDirFailure handler removes it
     val uuid = logManager.directoryId(dir)
     logManager.handleLogDirFailure(dir)
-    if (dir == config.metadataLogDir) {
+    if (dir == new File(config.metadataLogDir).getAbsolutePath && (zkClient.isEmpty || config.migrationEnabled)) {

Review Comment:
   This is not quite the correct check... you should check `config.processRoles` (probably `config.processRoles.isNotEmpty || config.migrationEnabled`



##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2586,7 +2586,7 @@ class ReplicaManager(val config: KafkaConfig,
     // retrieve the UUID here because logManager.handleLogDirFailure handler removes it
     val uuid = logManager.directoryId(dir)
     logManager.handleLogDirFailure(dir)
-    if (dir == config.metadataLogDir) {
+    if (dir == new File(config.metadataLogDir).getAbsolutePath && (zkClient.isEmpty || config.migrationEnabled)) {

Review Comment:
   This is not quite the correct check... you should check `config.processRoles` (probably `config.processRoles.isNotEmpty || config.migrationEnabled` )



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org