You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "clolov (via GitHub)" <gi...@apache.org> on 2023/02/16 13:03:57 UTC
[GitHub] [kafka] clolov opened a new pull request, #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
clolov opened a new pull request, #13260:
URL: https://github.com/apache/kafka/pull/13260
Implementation of https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1643528374
Heya @mimaison! Okay, I can think of and propose two ways forward:
1. I go back and modify the KIP based on the outcome of these tests to say it is safe to carry out the upgrade without a "bridge" upgrade.
2. We treat the tests passing as a happy-path upgrade and continue recommending a "bridge" upgrade.
As far as I know the upgrade/downgrade tests do not test all interactions that Kafka has with Zookeeper, so asking people to carry out two upgrades won't break their clusters even if only one is needed. As mentioned, the statement made in the KIP is a statement based on Zookeeper's documentation.
As such, I have a preference for 2. What would you like to go for? If you disagree with both what would your recommendation for going forward be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1660410928
Unrelated test failures at https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13260/9/pipeline/ . Merging this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1653704812
It appears that Jenkins is failing on `**/build/test-results/**/TEST-*.xml— Archive JUnit-formatted test results`. There are some failures in the tests, but the step itself has an output of `BUILD SUCCESSFUL in 54m 57s`. I will check whether other recent pull requests appear to have the same failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1599080971
I've not setup the system tests recently. If the provided instructions are not clear enough, let's discuss it on the dev mailing list. You can also open Jiras for the issues you encounter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1642274928
My point is that this really contradicts the KIP. In the KIP we say "it does not work" but then we have a test that relies on the upgrade working. This is a bit counter intuitive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1658242751
It appears that in Zookeeper 3.8.0 the Zookeeper community moved away from log4j to logback (source: https://issues.apache.org/jira/browse/ZOOKEEPER-4427)
In the failing test cases we attach an appender to the root logger (Kafka uses a log4j logger) and assert that we see certain log messages. Because of the mismatch in expectation (log4j) and reality (logback) these tests have started failing as the test-appender has no messages.
The difference in Zookeeper dependencies between current trunk (31 July 2023) and this pull request is:
```
trunk
org.apache.zookeeper:zookeeper:3.6.4
> io.netty:netty-handler:4.1.94.Final
> io.netty:netty-transport-native-epoll:4.1.94.Final
> org.apache.yetus:audience-annotations:0.13.0
> org.apache.zookeeper:zookeeper-jute:3.6.4
> org.slf4j:slf4j-api:1.7.36
pull request
org.apache.zookeeper:zookeeper:3.8.2
> ch.qos.logback:logback-classic:1.2.10 <- NEW
> ch.qos.logback:logback-core:1.2.10 <- NEW
> commons-io:commons-io:2.11.0 <- NEW
> io.netty:netty-handler:4.1.94.Final
> io.netty:netty-transport-native-epoll:4.1.94.Final
> org.apache.yetus:audience-annotations:0.12.0
> org.apache.zookeeper:zookeeper-jute:3.8.2
> org.slf4j:slf4j-api:1.7.36
```
I have removed the `ch.qos.logback:*` dependencies in the latest commit and tests have started passing locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1622020208
I'm not familiar with these tests so please correct me if I'm not understanding something correctly.
First, the KIP states (Compatibility, Deprecation, and Migration Plan section):
```
Users of Kafka clusters with Zookeeper clients older than 3.5.x won't be able to communicate with a 3.8.1 Zookeeper cluster.
```
Now looking at the tests in `upgrade_test.py`, `test_upgrade()` tries deploying a Kafka cluster and ZooKeeper from all previous releases and upgrade them to the latest version.
So for example, it deploys a 0.10.0.0 Kafka cluster with ZooKeeper 3.4.6. Then proceeds to upgrade ZooKeeper to the version from the latest Kafka which in this case should be 3.8.1. At this stage we should be in the scenario described in the KIP, Kafka brokers (still running 0.10.0.0) have ZooKeeper clients older than 3.5.x and have to communicate with ZooKeeper 3.8.1 which should not work.
Can you provide the output you got when running these tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1604416528
I ran https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/upgrade_test.py#L33 and all tests passed. I have confirmed that as part of the tests we start with an older version of Zookeeper
```
jinfo output
...
/opt/kafka-3.3.1/bin/../libs/zookeeper-3.6.3.jar
```
and we end up with a new one
```
jinfo output
...
/opt/kafka-dev/bin/../core/build/dependant-libs-2.13.10/zookeeper-3.8.1.jar
```
I was expecting upgrade tests to fail as detailed in the KIP, but they haven't done this. I will run https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/downgrade_test.py#L25 as well and report back my findings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1606959011
Heya @mimaison! I ran the upgrade/downgrade tests over the weekend and they passed as well. Unless there is something else which you would like me to run (or there is a place where you would like me to upload the results from the tests) I believe this is ready to be merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on a diff in pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on code in PR #13260:
URL: https://github.com/apache/kafka/pull/13260#discussion_r1280697282
##########
build.gradle:
##########
@@ -893,6 +893,10 @@ project(':core') {
implementation libs.dropwizardMetrics
exclude module: 'slf4j-log4j12'
exclude module: 'log4j'
+ // ZooKeeper moved from log4j to logback in v3.8.0,
+ // but Kafka relies on log4j. We are removing this additional dependency.
+ exclude module: 'logback-classic'
+ exclude module: 'logback-core'
}
// ZooKeeperMain depends on commons-cli but declares the dependency as `provided`
Review Comment:
I don't know whether commons-cli is the same as commons-io. I will check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1655753598
At least the below tests have started to continuously fail. I am still looking as to why:
```
Gradle Test Run :core:test > Gradle Test Executor 3 > ControllerIntegrationTest > testControllerMoveOnPreferredReplicaElection() FAILED
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnPreferredReplicaElection(ControllerIntegrationTest.scala:1885)
Gradle Test Run :core:test > Gradle Test Executor 3 > ControllerIntegrationTest > testControllerMoveOnTopicCreation() FAILED
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnTopicCreation(ControllerIntegrationTest.scala:1885)
Gradle Test Run :core:test > Gradle Test Executor 3 > ControllerIntegrationTest > testControllerMoveOnPartitionReassignment() FAILED
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnPartitionReassignment(ControllerIntegrationTest.scala:1885)
Gradle Test Run :core:test > Gradle Test Executor 3 > ControllerIntegrationTest > testControllerMoveOnTopicDeletion() FAILED
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnTopicDeletion(ControllerIntegrationTest.scala:1885)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1655821383
At least on a first look it appears something changed/is wrong with the loggers:
```
[2023-02-22T17:35:34.655Z] Gradle Test Run :core:unitTest > Gradle Test Executor 136 > ConfigAdminManagerTest > testPreprocessIncrementalWithLoggerChanges() FAILED
[2023-02-22T17:35:34.655Z] org.opentest4j.AssertionFailedError: expected: <{AlterConfigsResource(resourceType=8, resourceName='1', configs=[AlterableConfig(name='kafka.server.ConfigAdminManagerTest', configOperation=0, value='INFO')])=ApiError(error=NONE, message=null)}> but was: <{AlterConfigsResource(resourceType=8, resourceName='1', configs=[AlterableConfig(name='kafka.server.ConfigAdminManagerTest', configOperation=0, value='INFO')])=ApiError(error=INVALID_CONFIG, message=Logger kafka.server.ConfigAdminManagerTest does not exist!)}>
[2023-02-22T17:35:34.655Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T17:35:34.655Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T17:35:34.655Z] at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T17:35:34.655Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T17:35:34.655Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T17:35:34.655Z] at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T17:35:34.656Z] at app//kafka.server.ConfigAdminManagerTest.testPreprocessIncrementalWithLoggerChanges(ConfigAdminManagerTest.scala:352)
...
[2023-02-22T17:35:34.656Z] Gradle Test Run :core:unitTest > Gradle Test Executor 136 > ConfigAdminManagerTest > testValidateLogLevelConfigs() FAILED
[2023-02-22T17:35:34.656Z] org.apache.kafka.common.errors.InvalidConfigurationException: Logger kafka.server.ConfigAdminManagerTest does not exist!
...
[2023-02-22T17:36:10.045Z] Gradle Test Run :core:unitTest > Gradle Test Executor 136 > LoggingTest > testLoggerLevelIsResolved() FAILED
[2023-02-22T17:36:10.045Z] org.opentest4j.AssertionFailedError: expected: <TRACE> but was: <No such logger.>
[2023-02-22T17:36:10.045Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T17:36:10.045Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T17:36:10.045Z] at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T17:36:10.045Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T17:36:10.045Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T17:36:10.045Z] at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T17:36:10.045Z] at app//kafka.utils.LoggingTest.testLoggerLevelIsResolved(LoggingTest.scala:81)
...
[2023-02-22T17:37:30.427Z] Gradle Test Run :core:unitTest > Gradle Test Executor 173 > ConfigAdminManagerTest > testPreprocessIncrementalWithLoggerChanges() FAILED
[2023-02-22T17:37:30.427Z] org.opentest4j.AssertionFailedError: expected: <{AlterConfigsResource(resourceType=8, resourceName='1', configs=[AlterableConfig(name='kafka.server.ConfigAdminManagerTest', configOperation=0, value='INFO')])=ApiError(error=NONE, message=null)}> but was: <{AlterConfigsResource(resourceType=8, resourceName='1', configs=[AlterableConfig(name='kafka.server.ConfigAdminManagerTest', configOperation=0, value='INFO')])=ApiError(error=INVALID_CONFIG, message=Logger kafka.server.ConfigAdminManagerTest does not exist!)}>
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T17:37:30.427Z] at app//kafka.server.ConfigAdminManagerTest.testPreprocessIncrementalWithLoggerChanges(ConfigAdminManagerTest.scala:352)
...
[2023-02-22T17:37:30.427Z] Gradle Test Run :core:unitTest > Gradle Test Executor 173 > ConfigAdminManagerTest > testValidateLogLevelConfigs() FAILED
[2023-02-22T17:37:30.427Z] org.apache.kafka.common.errors.InvalidConfigurationException: Logger kafka.server.ConfigAdminManagerTest does not exist!
...
[2023-02-22T17:37:30.427Z] Gradle Test Run :core:unitTest > Gradle Test Executor 173 > LoggingTest > testLoggerLevelIsResolved() FAILED
[2023-02-22T17:37:30.427Z] org.opentest4j.AssertionFailedError: expected: <TRACE> but was: <No such logger.>
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T17:37:30.427Z] at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T17:37:30.427Z] at app//kafka.utils.LoggingTest.testLoggerLevelIsResolved(LoggingTest.scala:81)
...
[2023-02-22T17:37:30.427Z] 2757 tests completed, 6 failed, 5 skipped
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1655850239
Zoning in on ControllerIntegrationTest#testControllerMoveOnTopicCreation:
```
...
val event = appender.getMessages.find(e => e.getLevel == Level.INFO
&& e.getThrowableInformation != null
&& e.getThrowableInformation.getThrowable.getClass.getName.equals(classOf[ControllerMovedException].getName))
assertTrue(event.isDefined) <- THIS IS THE ASSERTION WHICH FAILS
...
```
However from the test logs:
```
16:03:02.258 [controller-event-thread] INFO kafka.controller.KafkaController - [Controller id=0] Controller moved to another broker when processing TopicChange.
org.apache.kafka.common.errors.ControllerMovedException: Controller epoch zkVersion check fails. Expected zkVersion = 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1652100097
@clolov we have CI failures. Can you please check what's going on here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1658538656
The failures in JDK 8/Scala 2.12 have to do with formatting on code which hasn't been touched as part of this pull request, so I believe it is safe to go ahead.
There are 2 failures in JDK 20/Scala 2.13 - one concerned with tiered storage (RemoteIndexCacheTest - cleaner did not shutdown) and one with simple production (PlaintextProducerSendTest - not enough brokers for specified RF). I believe these are transient failures as they do not manifest when running locally, but if people are unconvinced we can rerun the tests.
Otherwise, I think we are ready to merge this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on a diff in pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #13260:
URL: https://github.com/apache/kafka/pull/13260#discussion_r1280704453
##########
build.gradle:
##########
@@ -893,6 +893,10 @@ project(':core') {
implementation libs.dropwizardMetrics
exclude module: 'slf4j-log4j12'
exclude module: 'log4j'
+ // ZooKeeper moved from log4j to logback in v3.8.0,
+ // but Kafka relies on log4j. We are removing this additional dependency.
+ exclude module: 'logback-classic'
+ exclude module: 'logback-core'
}
// ZooKeeperMain depends on commons-cli but declares the dependency as `provided`
Review Comment:
My bad! you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1598704961
Heya @mimaison! I have been struggling with setting up the system test infrastructure as detailed in https://github.com/apache/kafka/tree/trunk/tests - I am trying to run them on EC2. Is there someone you know how has successfully set things up so I can speak with them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] forlack commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "forlack (via GitHub)" <gi...@apache.org>.
forlack commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1640943945
Should we bump to ZK 3.8.2 to include https://issues.apache.org/jira/browse/ZOOKEEPER-3860? Have seen a few kafka customers hit this issue recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on a diff in pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #13260:
URL: https://github.com/apache/kafka/pull/13260#discussion_r1280702334
##########
build.gradle:
##########
@@ -893,6 +893,10 @@ project(':core') {
implementation libs.dropwizardMetrics
exclude module: 'slf4j-log4j12'
exclude module: 'log4j'
+ // ZooKeeper moved from log4j to logback in v3.8.0,
+ // but Kafka relies on log4j. We are removing this additional dependency.
Review Comment:
Thank you. That clarifies things. This is a safe change now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1655828383
```
[2023-02-22T18:18:23.694Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 175 > PlaintextAdminIntegrationTest > testDescribeConfigsForLog4jLogLevels(String) > kafka.api.PlaintextAdminIntegrationTest.testDescribeConfigsForLog4jLogLevels(String)[1] FAILED
[2023-02-22T18:18:23.694Z] java.lang.NullPointerException
[2023-02-22T18:18:23.694Z] at kafka.api.PlaintextAdminIntegrationTest.testDescribeConfigsForLog4jLogLevels(PlaintextAdminIntegrationTest.scala:2260)
...
[2023-02-22T18:18:28.429Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 175 > PlaintextAdminIntegrationTest > testDescribeConfigsForLog4jLogLevels(String) > kafka.api.PlaintextAdminIntegrationTest.testDescribeConfigsForLog4jLogLevels(String)[2] FAILED
[2023-02-22T18:18:28.429Z] java.lang.NullPointerException
[2023-02-22T18:18:28.429Z] at kafka.api.PlaintextAdminIntegrationTest.testDescribeConfigsForLog4jLogLevels(PlaintextAdminIntegrationTest.scala:2260)
...
[2023-02-22T18:42:58.206Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 176 > ControllerIntegrationTest > testControllerMoveOnTopicDeletion() FAILED
[2023-02-22T18:42:58.206Z] org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[2023-02-22T18:42:58.206Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T18:42:58.206Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T18:42:58.206Z] at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2023-02-22T18:42:58.206Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2023-02-22T18:42:58.206Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
[2023-02-22T18:42:58.206Z] at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
[2023-02-22T18:42:58.206Z] at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnTopicDeletion(ControllerIntegrationTest.scala:1884)
...
[2023-02-22T18:42:58.207Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 176 > ControllerIntegrationTest > testControllerMoveOnPartitionReassignment() FAILED
[2023-02-22T18:42:58.207Z] org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[2023-02-22T18:42:58.207Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T18:42:58.207Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T18:42:58.207Z] at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2023-02-22T18:42:58.207Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2023-02-22T18:42:58.207Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
[2023-02-22T18:42:58.207Z] at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
[2023-02-22T18:42:58.207Z] at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnPartitionReassignment(ControllerIntegrationTest.scala:1884)
...
[2023-02-22T18:43:01.009Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 176 > ControllerIntegrationTest > testControllerMoveOnTopicCreation() FAILED
[2023-02-22T18:43:01.009Z] org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[2023-02-22T18:43:01.009Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T18:43:01.009Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T18:43:01.009Z] at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2023-02-22T18:43:01.009Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2023-02-22T18:43:01.009Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
[2023-02-22T18:43:01.009Z] at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
[2023-02-22T18:43:01.009Z] at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnTopicCreation(ControllerIntegrationTest.scala:1884)
...
[2023-02-22T18:43:52.201Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 176 > ControllerIntegrationTest > testControllerMoveOnPreferredReplicaElection() FAILED
[2023-02-22T18:43:52.201Z] org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[2023-02-22T18:43:52.201Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T18:43:52.201Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T18:43:52.201Z] at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2023-02-22T18:43:52.201Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2023-02-22T18:43:52.201Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
[2023-02-22T18:43:52.201Z] at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
[2023-02-22T18:43:52.201Z] at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnPreferredReplicaElection(ControllerIntegrationTest.scala:1884)
...
[2023-02-22T19:02:05.196Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 176 > KRaftClusterTest > testSetLog4jConfigurations() FAILED
[2023-02-22T19:02:05.196Z] org.opentest4j.AssertionFailedError: expected: <List(ApiError(error=NONE, message=null), ApiError(error=INVALID_REQUEST, message=APPEND operation is not allowed for the BROKER_LOGGER resource))> but was: <List(ApiError(error=INVALID_CONFIG, message=Logger kafka.server.KRaftClusterTest does not exist!), ApiError(error=INVALID_REQUEST, message=APPEND operation is not allowed for the BROKER_LOGGER resource))>
[2023-02-22T19:02:05.196Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T19:02:05.196Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T19:02:05.196Z] at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T19:02:05.196Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T19:02:05.196Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T19:02:05.196Z] at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T19:02:05.196Z] at app//kafka.server.KRaftClusterTest.testSetLog4jConfigurations(KRaftClusterTest.scala:645)
...
[2023-02-22T19:33:55.457Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 179 > ControllerIntegrationTest > testControllerMoveOnTopicDeletion() FAILED
[2023-02-22T19:33:55.457Z] org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[2023-02-22T19:33:55.457Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T19:33:55.457Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T19:33:55.457Z] at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2023-02-22T19:33:55.457Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2023-02-22T19:33:55.457Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
[2023-02-22T19:33:55.457Z] at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
[2023-02-22T19:33:55.457Z] at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnTopicDeletion(ControllerIntegrationTest.scala:1884)
...
[2023-02-22T19:33:57.499Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 179 > ControllerIntegrationTest > testControllerMoveOnPartitionReassignment() FAILED
[2023-02-22T19:33:57.499Z] org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[2023-02-22T19:33:57.499Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T19:33:57.499Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T19:33:57.499Z] at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2023-02-22T19:33:57.499Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2023-02-22T19:33:57.499Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
[2023-02-22T19:33:57.499Z] at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
[2023-02-22T19:33:57.499Z] at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnPartitionReassignment(ControllerIntegrationTest.scala:1884)
...
[2023-02-22T19:33:57.499Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 180 > PlaintextAdminIntegrationTest > testDescribeConfigsForLog4jLogLevels(String) > kafka.api.PlaintextAdminIntegrationTest.testDescribeConfigsForLog4jLogLevels(String)[1] FAILED
[2023-02-22T19:33:57.499Z] java.lang.NullPointerException
[2023-02-22T19:33:57.499Z] at kafka.api.PlaintextAdminIntegrationTest.testDescribeConfigsForLog4jLogLevels(PlaintextAdminIntegrationTest.scala:2260)
...
[2023-02-22T19:33:59.539Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 179 > ControllerIntegrationTest > testControllerMoveOnTopicCreation() FAILED
[2023-02-22T19:33:59.539Z] org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[2023-02-22T19:33:59.539Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T19:33:59.539Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T19:33:59.539Z] at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2023-02-22T19:33:59.539Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2023-02-22T19:33:59.539Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
[2023-02-22T19:33:59.539Z] at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
[2023-02-22T19:33:59.539Z] at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnTopicCreation(ControllerIntegrationTest.scala:1884)
...
[2023-02-22T19:34:00.561Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 179 > ControllerIntegrationTest > testControllerMoveOnPreferredReplicaElection() FAILED
[2023-02-22T19:34:00.561Z] org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[2023-02-22T19:34:00.561Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T19:34:00.561Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T19:34:00.561Z] at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2023-02-22T19:34:00.561Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2023-02-22T19:34:00.561Z] at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
[2023-02-22T19:34:00.561Z] at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
[2023-02-22T19:34:00.561Z] at app//kafka.controller.ControllerIntegrationTest.testControllerMoveOnPreferredReplicaElection(ControllerIntegrationTest.scala:1884)
...
[2023-02-22T19:34:05.885Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 180 > PlaintextAdminIntegrationTest > testDescribeConfigsForLog4jLogLevels(String) > kafka.api.PlaintextAdminIntegrationTest.testDescribeConfigsForLog4jLogLevels(String)[2] FAILED
[2023-02-22T19:34:05.885Z] java.lang.NullPointerException
[2023-02-22T19:34:05.885Z] at kafka.api.PlaintextAdminIntegrationTest.testDescribeConfigsForLog4jLogLevels(PlaintextAdminIntegrationTest.scala:2260)
[2023-02-22T19:34:12.144Z] kafka.server.KRaftClusterTest.testSetLog4jConfigurations() failed, log available in /home/jenkins/workspace/Kafka_kafka-pr_PR-13260/core/build/reports/testOutput/kafka.server.KRaftClusterTest.testSetLog4jConfigurations().test.stdout
...
[2023-02-22T19:34:12.144Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 179 > KRaftClusterTest > testSetLog4jConfigurations() FAILED
[2023-02-22T19:34:12.144Z] org.opentest4j.AssertionFailedError: expected: <List(ApiError(error=NONE, message=null), ApiError(error=INVALID_REQUEST, message=APPEND operation is not allowed for the BROKER_LOGGER resource))> but was: <List(ApiError(error=INVALID_CONFIG, message=Logger kafka.server.KRaftClusterTest does not exist!), ApiError(error=INVALID_REQUEST, message=APPEND operation is not allowed for the BROKER_LOGGER resource))>
[2023-02-22T19:34:12.144Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2023-02-22T19:34:12.144Z] at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2023-02-22T19:34:12.144Z] at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
[2023-02-22T19:34:12.144Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
[2023-02-22T19:34:12.144Z] at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
[2023-02-22T19:34:12.144Z] at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1142)
[2023-02-22T19:34:12.144Z] at app//kafka.server.KRaftClusterTest.testSetLog4jConfigurations(KRaftClusterTest.scala:645)
...
[2023-02-22T19:34:12.144Z] 1970 tests completed, 14 failed, 5 skipped
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1576955162
Thanks @clolov for the PR. I think some system tests do upgrades from previous versions to the latest. Have you checked these system tests still work with this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on a diff in pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #13260:
URL: https://github.com/apache/kafka/pull/13260#discussion_r1280597042
##########
build.gradle:
##########
@@ -893,6 +893,10 @@ project(':core') {
implementation libs.dropwizardMetrics
exclude module: 'slf4j-log4j12'
exclude module: 'log4j'
+ // ZooKeeper moved from log4j to logback in v3.8.0,
+ // but Kafka relies on log4j. We are removing this additional dependency.
+ exclude module: 'logback-classic'
+ exclude module: 'logback-core'
}
// ZooKeeperMain depends on commons-cli but declares the dependency as `provided`
Review Comment:
Is this still true with the latest release? (I guess not since commons-cli is explicitly pulled as a dependency of Zk)
##########
build.gradle:
##########
@@ -893,6 +893,10 @@ project(':core') {
implementation libs.dropwizardMetrics
exclude module: 'slf4j-log4j12'
exclude module: 'log4j'
+ // ZooKeeper moved from log4j to logback in v3.8.0,
+ // but Kafka relies on log4j. We are removing this additional dependency.
Review Comment:
> but Kafka relies on log4j
this is not correct. Kafka uses slf4j APIs with reload4j as the logging framework. I would suggest to rephrase this as "Zookeeper uses logback as the logging framework but Kafka uses reload4j, hence we want to avoid having multiple frameworks in dependency chain, so removing logback".
Next, can you check why Zk logs are working even if we are using reload4j at runtime? (the answer might be that Zk uses slf4j and underlying framework is pluggable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya merged pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya merged PR #13260:
URL: https://github.com/apache/kafka/pull/13260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1622084674
FYI @mimaison , @clolov is on vacation this week. I will bring this comment to his attention when he is back next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1623903798
Also we need to update the `LICENSE-binary` file as mentioned in https://github.com/apache/kafka/blob/trunk/gradle/dependencies.gradle#L73-L74
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1516273036
Hello @mimaison, @ijuma and @cmccabe! Since [KIP-902](https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1) has been accepted I have rebased this pull request and added to the documentation for 3.6. I would be grateful for all suggestions for improvements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1628903388
> I'm not familiar with these tests so please correct me if I'm not understanding something correctly.
>
> First, the KIP states (Compatibility, Deprecation, and Migration Plan section):
>
> ```
> Users of Kafka clusters with Zookeeper clients older than 3.5.x won't be able to communicate with a 3.8.1 Zookeeper cluster.
> ```
>
> Now looking at the tests in `upgrade_test.py`, `test_upgrade()` tries deploying a Kafka cluster and ZooKeeper from all previous releases and upgrade them to the latest version.
>
> So for example, it deploys a 0.10.0.0 Kafka cluster with ZooKeeper 3.4.6. Then proceeds to upgrade ZooKeeper to the version from the latest Kafka which in this case should be 3.8.1. At this stage we should be in the scenario described in the KIP, Kafka brokers (still running 0.10.0.0) have ZooKeeper clients older than 3.5.x and have to communicate with ZooKeeper 3.8.1 which should not work.
>
> Can you provide the output you got when running these tests?
Heya @mimaison! Yes, as mentioned in a previous comment of mine, I have confirmed that we start with an older version of Zookeeper and as part of the tests we upgrade to the latest. The KIP was written based on information from Zookeeper's official documentation as I was unaware of these tests at the time. I suspect the tests do not fail because the Zookeeper APIs Kafka uses are compatible across all versions despite what Zookeeper's documentation says. This being said, these upgrade/downgrade tests only test production + consumption so I would rather be safe than sorry and go forward with recommending to do a double-rolling-upgrade as documented in the KIP. The current server.log output of the tests is
```
ubuntu@ip-172-31-94-245:~/kafka/results$ du -sh -- */ | sort -rh
50G 2023-06-23--003/
50G 2023-06-22--002/
```
so I suspect it will be difficult to share via GitHub. Where would you like me to put the results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1641957263
Hey @forlack! Thank you for the suggestion, I have upgraded the dependency so that tests can run.
Heya @mimaison, I have addressed your comment about adding the latest version to the LICENSE-binary file and I would be grateful for any last review as this has been open for sometime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on a diff in pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on code in PR #13260:
URL: https://github.com/apache/kafka/pull/13260#discussion_r1280697282
##########
build.gradle:
##########
@@ -893,6 +893,10 @@ project(':core') {
implementation libs.dropwizardMetrics
exclude module: 'slf4j-log4j12'
exclude module: 'log4j'
+ // ZooKeeper moved from log4j to logback in v3.8.0,
+ // but Kafka relies on log4j. We are removing this additional dependency.
+ exclude module: 'logback-classic'
+ exclude module: 'logback-core'
}
// ZooKeeperMain depends on commons-cli but declares the dependency as `provided`
Review Comment:
I think commons-cli and commons-io are two different libraries so I would prefer to keep commons-cli.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on a diff in pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on code in PR #13260:
URL: https://github.com/apache/kafka/pull/13260#discussion_r1280689006
##########
build.gradle:
##########
@@ -893,6 +893,10 @@ project(':core') {
implementation libs.dropwizardMetrics
exclude module: 'slf4j-log4j12'
exclude module: 'log4j'
+ // ZooKeeper moved from log4j to logback in v3.8.0,
+ // but Kafka relies on log4j. We are removing this additional dependency.
Review Comment:
Yup, this is the exact reason. Kafka uses slf4j + reload4j. Zookeeper uses slf4j + logback. When we remove logback we fallback to reload4j for both Zookeeper and Kafka. I will update the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] clolov commented on pull request #13260: KAFKA-14661: Upgrade Zookeeper to 3.8.1
Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on PR #13260:
URL: https://github.com/apache/kafka/pull/13260#issuecomment-1581120917
I have not, but I will aim to carry out the check in the upcoming days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org