You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/02/23 22:33:37 UTC

[GitHub] [kafka] rondagostino opened a new pull request #10199: MINOR: Fix security_test system test for Raft case

rondagostino opened a new pull request #10199:
URL: https://github.com/apache/kafka/pull/10199


   The test with SSL as the security protocol between the broker and raft-based controller when there is a hostname mismatch was failing because this case simply won't work -- the broker will not restart.  This patch updates the test to make sure this behavior occurs.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on a change in pull request #10199: KAFKA-12374: Add missing config sasl.mechanism.controller.protocol

Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#discussion_r583240035



##########
File path: tests/kafkatest/services/security/security_config.py
##########
@@ -250,14 +252,16 @@ def setup_sasl(self, node):
                     'is_ibm_jdk': any('IBM' in line for line in java_version),
                     'SecurityConfig': SecurityConfig,
                     'client_sasl_mechanism': self.client_sasl_mechanism,
-                    'enabled_sasl_mechanisms': self.enabled_sasl_mechanisms
+                    'enabled_sasl_mechanisms': self.enabled_sasl_mechanisms,
+                    'usable_sasl_mechanisms': self.usable_sasl_mechanisms
                 }
             )
         else:
             jaas_conf = self.properties['sasl.jaas.config']
 
         if self.static_jaas_conf:
             node.account.create_file(SecurityConfig.JAAS_CONF_PATH, jaas_conf)
+            print(jaas_conf)

Review comment:
       Will revert this unintentional change




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on pull request #10199: MINOR: Fix security_test system test for Raft case

Posted by GitBox <gi...@apache.org>.
rondagostino commented on pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#issuecomment-785265318


   Hi @abbccdda.  We used to see this error message in `verifiable_producer.log` when `security_protocol='PLAINTEXT', interbroker_security_protocol='SSL'`:
   
   ```
   WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {test_topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
   ```
   The test does a `grep LEADER_NOT_AVAILABLE` on the log in this case, and it used to pass.
   
   Now we are instead seeing this in the log file:
   
   ```
   WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {test_topic=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)
   ```
   
   And of course now the test fails.
   
   The `INVALID_REPLICATION_FACTOR` is coming from the auto topic creation manager as I described above.
   
   It is a simple matter to make the test pass -- I have confirmed that it passes if we `grep` for `INVALID_REPLICATION_FACTOR` in the log file instead of `LEADER_NOT_AVAILABLE`.
   
   I think we just need to decide if this change in behavior is acceptable or not.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on a change in pull request #10199: KAFKA-12374: Add missing config sasl.mechanism.controller.protocol

Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#discussion_r583226737



##########
File path: core/src/main/scala/kafka/server/KafkaConfig.scala
##########
@@ -675,6 +676,7 @@ object KafkaConfig {
     "KIP-500. If it is not set, the metadata log is placed in the first log directory from log.dirs."
   val ControllerListenerNamesDoc = "A comma-separated list of the names of the listeners used by the KIP-500 controller. This is required " +
     "if this process is a KIP-500 controller. The ZK-based controller will not use this configuration."
+  val SaslMechanismControllerProtocolDoc = "SASL mechanism used for broker-to-raft-controller communication. Default is GSSAPI."

Review comment:
       Should probably say what KIP-631 says, which is "SASL mechanism used for communication with controllers. 
    Default is GSSAPI." It is used by Raft controllers to talk to other controllers, after all (assuming we are using SASL for inter-controller communication).  Will fix.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on pull request #10199: MINOR: Fix security_test system test for Raft case

Posted by GitBox <gi...@apache.org>.
rondagostino commented on pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#issuecomment-785311895


   @mumrah I separated out the communication-to-quorum failure cases into a separate test -- so now we confirm that hostname verification failure to both ZK and Raft Controller causes Kafka to be unable to start.
   
   I did not add the test where communication to the Raft quorum is okay but the inter-broker protocol is SSL and failing due to hostname verification.  Let me know if you think it is necessary and I can add it.
   
   @abbccdda I "fixed" the ZooKeeper test as mentioned above -- we can revert if you decide you need to fix something related to the auto topic creation manager instead.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on a change in pull request #10199: MINOR: Fix security_test system test for Raft case

Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#discussion_r582029624



##########
File path: tests/kafkatest/tests/core/security_test.py
##########
@@ -82,7 +82,19 @@ def test_client_ssl_endpoint_validation_failure(self, security_protocol, interbr
 
         # now set the certs to have invalid hostnames so we can run the actual test
         SecurityConfig.ssl_stores.valid_hostname = False
-        self.kafka.restart_cluster()
+        if quorum.for_test(self.test_context) == quorum.zk or interbroker_security_protocol != 'SSL':
+            self.kafka.restart_cluster()
+        else:
+            # Raft-based metadata quorum with SSL communication between quorum and broker
+            # will simply fail to work due to TLS hostname mismatch
+            self.kafka.remote_controller_quorum.restart_cluster()
+            try:
+                self.kafka.restart_cluster()
+                raise RuntimeError("Kafka restarted successfully but should not have!"
+                                   " Endpoint validation did not fail with invalid hostname")
+            except TimeoutError:

Review comment:
       The test burns 120 seconds waiting because the restart and shutdown timeouts are both 60 seconds by default.  We can adjust the restart timeout to 30 seconds if we wanted to, but the shutdown timeout is set by ducktape during the teardown and can't be changed.  So we burn 120 seconds but could probably drop that to 90 seconds.  The test currently runs like this: `run time:         2 minutes 43.263 seconds`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on a change in pull request #10199: KAFKA-12374: Add missing config sasl.mechanism.controller.protocol

Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#discussion_r583228799



##########
File path: tests/kafkatest/sanity_checks/test_verifiable_producer.py
##########
@@ -96,4 +96,59 @@ def test_simple_run(self, producer_version, security_protocol = 'PLAINTEXT', sas
         num_produced = self.producer.num_acked
         assert num_produced == self.num_messages, "num_produced: %d, num_messages: %d" % (num_produced, self.num_messages)
 
+    @cluster(num_nodes=4)
+    @matrix(inter_broker_security_protocol=['PLAINTEXT', 'SSL'], metadata_quorum=[quorum.remote_raft])
+    @matrix(inter_broker_security_protocol=['SASL_SSL'], inter_broker_sasl_mechanism=['PLAIN', 'GSSAPI'],
+            metadata_quorum=[quorum.remote_raft])
+    def test_multiple_raft_security_protocols(
+            self, inter_broker_security_protocol, inter_broker_sasl_mechanism='GSSAPI', metadata_quorum=quorum.remote_raft):
+        """
+        Test for remote Raft cases that we can start VerifiableProducer on the current branch snapshot version, and
+        verify that we can produce a small number of messages.  The inter-controller and broker-to-controller
+        security protocols are defined to be different (which differs from the above test, where they were the same).
+        """
+        self.kafka.security_protocol = self.kafka.interbroker_security_protocol = inter_broker_security_protocol
+        self.kafka.client_sasl_mechanism = self.kafka.interbroker_sasl_mechanism = inter_broker_sasl_mechanism
+        controller_quorum = self.kafka.controller_quorum
+        sasl_mechanism = 'PLAIN' if inter_broker_sasl_mechanism == 'GSSAPI' else 'GSSAPI'
+        if inter_broker_security_protocol == 'PLAINTEXT':
+            controller_security_protocol = 'SSL'
+            intercontroller_security_protocol = 'SASL_SSL'
+        elif inter_broker_security_protocol == 'SSL':
+            controller_security_protocol = 'SASL_SSL'
+            intercontroller_security_protocol = 'PLAINTEXT'
+        else: # inter_broker_security_protocol == 'SASL_SSL'
+            controller_security_protocol = 'PLAINTEXT'
+            intercontroller_security_protocol = 'SSL'
+        controller_quorum.controller_security_protocol = controller_security_protocol
+        controller_quorum.controller_sasl_mechanism = sasl_mechanism
+        controller_quorum.intercontroller_security_protocol = intercontroller_security_protocol
+        controller_quorum.intercontroller_sasl_mechanism = sasl_mechanism
+        self.kafka.start()
+
+        node = self.producer.nodes[0]
+        node.version = KafkaVersion(str(DEV_BRANCH))
+        self.producer.start()
+        wait_until(lambda: self.producer.num_acked > 5, timeout_sec=15,
+             err_msg="Producer failed to start in a reasonable amount of time.")
+
+        # using version.vstring (distutils.version.LooseVersion) is a tricky way of ensuring
+        # that this check works with DEV_BRANCH
+        # When running VerifiableProducer 0.8.X, both the current branch version and 0.8.X should show up because of the
+        # way verifiable producer pulls in some development directories into its classpath
+        #
+        # If the test fails here because 'ps .. | grep' couldn't find the process it means
+        # the login and grep that is_version() performs is slower than
+        # the time it takes the producer to produce its messages.
+        # Easy fix is to decrease throughput= above, the good fix is to make the producer
+        # not terminate until explicitly killed in this case.
+        if node.version <= LATEST_0_8_2:
+            assert is_version(node, [node.version.vstring, DEV_BRANCH.vstring], logger=self.logger)
+        else:

Review comment:
       This stuff can probably be removed.  Remote Raft isn't supported on older versions, and the comment appears above anyway.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] mumrah commented on a change in pull request #10199: MINOR: Fix security_test system test for Raft case

Posted by GitBox <gi...@apache.org>.
mumrah commented on a change in pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#discussion_r582017351



##########
File path: tests/kafkatest/tests/core/security_test.py
##########
@@ -82,7 +82,19 @@ def test_client_ssl_endpoint_validation_failure(self, security_protocol, interbr
 
         # now set the certs to have invalid hostnames so we can run the actual test
         SecurityConfig.ssl_stores.valid_hostname = False
-        self.kafka.restart_cluster()
+        if quorum.for_test(self.test_context) == quorum.zk or interbroker_security_protocol != 'SSL':
+            self.kafka.restart_cluster()
+        else:
+            # Raft-based metadata quorum with SSL communication between quorum and broker
+            # will simply fail to work due to TLS hostname mismatch
+            self.kafka.remote_controller_quorum.restart_cluster()
+            try:
+                self.kafka.restart_cluster()
+                raise RuntimeError("Kafka restarted successfully but should not have!"
+                                   " Endpoint validation did not fail with invalid hostname")
+            except TimeoutError:

Review comment:
       How long does this test have to wait for the timeout?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] mumrah commented on a change in pull request #10199: MINOR: Fix security_test system test for Raft case

Posted by GitBox <gi...@apache.org>.
mumrah commented on a change in pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#discussion_r582025258



##########
File path: tests/kafkatest/tests/core/security_test.py
##########
@@ -82,7 +82,19 @@ def test_client_ssl_endpoint_validation_failure(self, security_protocol, interbr
 
         # now set the certs to have invalid hostnames so we can run the actual test
         SecurityConfig.ssl_stores.valid_hostname = False
-        self.kafka.restart_cluster()
+        if quorum.for_test(self.test_context) == quorum.zk or interbroker_security_protocol != 'SSL':
+            self.kafka.restart_cluster()
+        else:
+            # Raft-based metadata quorum with SSL communication between quorum and broker
+            # will simply fail to work due to TLS hostname mismatch
+            self.kafka.remote_controller_quorum.restart_cluster()

Review comment:
       As I understand it, this test wants to mess up the inter-broker listener and see that client's get the correct error. However, in raft mode, we can't load any metadata if the inter-broker listener is non-functional since we cannot connect to the raft quorum. This is analogous to breaking the connection to ZK in ZK mode. 
   
   Maybe we should just take the raft mode out of this test and write a new test that exercises similar behavior? Perhaps we could let the inter-broker listener stay properly configured, but mis-configure the controller listener. WDYT?
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe merged pull request #10199: KAFKA-12374: Add missing config sasl.mechanism.controller.protocol

Posted by GitBox <gi...@apache.org>.
cmccabe merged pull request #10199:
URL: https://github.com/apache/kafka/pull/10199


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on pull request #10199: MINOR: Fix security_test system test for Raft case

Posted by GitBox <gi...@apache.org>.
rondagostino commented on pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#issuecomment-784563557


   @abbccdda The ZooKeeper version of this test is failing because the producer is never seeing `LEADER_NOT_AVAILABLE`.  The issue seems to be that when the broker sees a `METADATA` request for `test_topic` after it restarts it is determining that the topic needs to be created, and then the auto topic creation manager is determining that there aren't enough brokers available -- so it returns `INVALID_REPLICATION_FACTOR` for that topic in the Metadata response.  In other words, the flow has changed and the issue is not manifesting as it was before.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on a change in pull request #10199: KAFKA-12374: Add missing config sasl.mechanism.controller.protocol

Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#discussion_r583240514



##########
File path: tests/kafkatest/services/security/security_config.py
##########
@@ -343,6 +347,17 @@ def interbroker_sasl_mechanism(self):
     def enabled_sasl_mechanisms(self):
         return set([self.client_sasl_mechanism, self.interbroker_sasl_mechanism])
 
+    @property
+    def usable_sasl_mechanisms(self):
+        """
+        Differs from enabled_sasl_mechanisms in that it also includes SASL mechanisms that
+        are used for Raft communication
+        :return: enabled_sasl_mechanisms plus any mechanisms that are used for Raft communication
+        """
+        retval = set([self.client_sasl_mechanism, self.interbroker_sasl_mechanism] + self.raft_sasl_mechanisms)
+        print(retval)

Review comment:
       will revert




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] cmccabe commented on pull request #10199: KAFKA-12374: Add missing config sasl.mechanism.controller.protocol

Posted by GitBox <gi...@apache.org>.
cmccabe commented on pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#issuecomment-786970988


   I filed https://issues.apache.org/jira/browse/KAFKA-12381 to follow up on the auto-topic-creation behavior change discussion.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] rondagostino commented on a change in pull request #10199: MINOR: Fix security_test system test for Raft case

Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10199:
URL: https://github.com/apache/kafka/pull/10199#discussion_r582046667



##########
File path: tests/kafkatest/tests/core/security_test.py
##########
@@ -82,7 +82,19 @@ def test_client_ssl_endpoint_validation_failure(self, security_protocol, interbr
 
         # now set the certs to have invalid hostnames so we can run the actual test
         SecurityConfig.ssl_stores.valid_hostname = False
-        self.kafka.restart_cluster()
+        if quorum.for_test(self.test_context) == quorum.zk or interbroker_security_protocol != 'SSL':
+            self.kafka.restart_cluster()
+        else:
+            # Raft-based metadata quorum with SSL communication between quorum and broker
+            # will simply fail to work due to TLS hostname mismatch
+            self.kafka.remote_controller_quorum.restart_cluster()

Review comment:
       Yeah, I agree that mucking with the broker-controller connectivity is analogous to mucking with the non-Raft brokers' connection to ZooKeeper.  We currently don't check that Kafka will not start if hostname verification fails to a TLS-enabled Zookeeper quorum, but that type of a test -- Raft and Zookeeper -- would fit in a "security system test" as we are doing here.  I think there's room for checking to make sure we can't start with hostname verification failing to the Raft controller quorum.  I don't know if it's worth adding a ZooKeeper case given that TLS to ZooKeeper just became supported in AK 2.7 (meaning adoption is light) and we are moving away from ZooKeeper.  Reaction to these thoughts?
   
   Aside from the above, we could consider leaving Broker-to-Raft-Controller connectivity unperturbed/functional and make hostname verification fail for the inter-broker security protocol as you suggest.  We could add that after we get the ZooKeeper version working again.  I could "fix" that test and add the Raft equivalent, but Boyang needs to take a look to see if the auto topic creation manager changes are causing something to happen that should not.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org