You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "frankvicky (via GitHub)" <gi...@apache.org> on 2024/09/09 04:48:03 UTC

[PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

frankvicky opened a new pull request, #17131:
URL: https://github.com/apache/kafka/pull/17131

   JIRA: [KAFKA-17497
   ](https://issues.apache.org/jira/browse/KAFKA-17497)
   > This is follow-up of [KAFKA-17011](https://issues.apache.org/jira/browse/KAFKA-17011) that zk migration will encounter min=0 issue when the cluster consists of "new" broker and "old" controller.
   The e2e `zookeeper_migration_test` [0] is a good test for online migration, but it does not test hybrid version. Hence, we can leverage it to increase test coverage.
   [0] https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/zookeeper_migration_test.py#L91
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17131:
URL: https://github.com/apache/kafka/pull/17131#issuecomment-2339562305

   I'm going to merge this PR. The e2e case of 3.8 will be addressed by #17128


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17131:
URL: https://github.com/apache/kafka/pull/17131#discussion_r1749964870


##########
tests/kafkatest/tests/core/zookeeper_migration_test.py:
##########
@@ -86,20 +86,23 @@ def do_migration(self, roll_controller = False, downgrade_to_zk = False):
                 controller.start_node(node)
 
     @cluster(num_nodes=7)
-    @parametrize(roll_controller = True)
-    @parametrize(roll_controller = False)
-    def test_online_migration(self, roll_controller):
+    @matrix(roll_controller=[True, False], from_kafka_version=[str(DEV_BRANCH), str(LATEST_3_7)])
+    def test_online_migration(self, roll_controller, from_kafka_version):
         zk_quorum = partial(ServiceQuorumInfo, zk)
         self.zk = ZookeeperService(self.test_context, num_nodes=1, version=DEV_BRANCH)
+
+        server_prop_overrides = [["zookeeper.metadata.migration.enable", "false"]]
+
+        if from_kafka_version != str(DEV_BRANCH):
+            server_prop_overrides.append([config_property.INTER_BROKER_PROTOCOL_VERSION, "3.7"])

Review Comment:
   My point was - please use `str(from_kafka_version)` to replace `3.7`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on PR #17131:
URL: https://github.com/apache/kafka/pull/17131#issuecomment-2339512609

   Hi @mumrah,
   
   As @chia7712 mentioned, this test aims to ensure that the controller can work with brokers of higher versions. We need to verify that the higher version brokers can set the minimum supported version to 1 and communicate with the controller normally, as discussed in this https://github.com/apache/kafka/pull/16421#discussion_r1745300168
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17131:
URL: https://github.com/apache/kafka/pull/17131#issuecomment-2338762869

   > Is the purpose of the new test case to verify that the migration can happen on the latest software version with an older MV/IBP? E.g., 3.7 MV on 3.9 code base
   
   The purpose is to verify the migration can happen on the latest broker with "older quorum controller". There was a min=0 issue in migrating (https://github.com/apache/kafka/pull/16420#issuecomment-2192619405), and we did not add a e2e for the scenario before.
   
   for another: the motivation of this jira/PR is from this comment (https://github.com/apache/kafka/pull/16421#discussion_r1746680267) that I realize the scenario is not easy to understand. Having a e2e case can help us to discuss the migration case


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #17131:
URL: https://github.com/apache/kafka/pull/17131


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17131:
URL: https://github.com/apache/kafka/pull/17131#discussion_r1749562688


##########
tests/kafkatest/tests/core/zookeeper_migration_test.py:
##########
@@ -86,9 +86,8 @@ def do_migration(self, roll_controller = False, downgrade_to_zk = False):
                 controller.start_node(node)
 
     @cluster(num_nodes=7)
-    @parametrize(roll_controller = True)
-    @parametrize(roll_controller = False)
-    def test_online_migration(self, roll_controller):
+    @matrix(roll_controller=[True, False], from_kafka_version=[str(DEV_BRANCH), str(LATEST_3_7)])
+    def test_online_migration(self, roll_controller, from_kafka_version):
         zk_quorum = partial(ServiceQuorumInfo, zk)
         self.zk = ZookeeperService(self.test_context, num_nodes=1, version=DEV_BRANCH)
         self.kafka = KafkaService(self.test_context,

Review Comment:
   Please set IBP if the version is NOT `DEV_BRANCH`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #17131:
URL: https://github.com/apache/kafka/pull/17131#issuecomment-2339559850

   > We need to verify that the higher version brokers can set the minimum supported version to 1 
   
   This description is correct before #17128 gets merged. The new approach (#17128) will skip the features instead of changing the min version.  The new approach can fix the issue of min=0 check in 3.8 controller
   https://github.com/apache/kafka/blob/3.8/metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java#L470
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17497: Add e2e for zk migration with old controller [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17131:
URL: https://github.com/apache/kafka/pull/17131#discussion_r1749950020


##########
tests/kafkatest/tests/core/zookeeper_migration_test.py:
##########
@@ -86,20 +86,23 @@ def do_migration(self, roll_controller = False, downgrade_to_zk = False):
                 controller.start_node(node)
 
     @cluster(num_nodes=7)
-    @parametrize(roll_controller = True)
-    @parametrize(roll_controller = False)
-    def test_online_migration(self, roll_controller):
+    @matrix(roll_controller=[True, False], from_kafka_version=[str(DEV_BRANCH), str(LATEST_3_7)])
+    def test_online_migration(self, roll_controller, from_kafka_version):
         zk_quorum = partial(ServiceQuorumInfo, zk)
         self.zk = ZookeeperService(self.test_context, num_nodes=1, version=DEV_BRANCH)
+
+        server_prop_overrides = [["zookeeper.metadata.migration.enable", "false"]]
+
+        if from_kafka_version != str(DEV_BRANCH):
+            server_prop_overrides.append([config_property.INTER_BROKER_PROTOCOL_VERSION, "3.7"])

Review Comment:
   Please use `str(from_kafka_version`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org