Python client failed to connect secured Kafka: SSL handshake failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number

[Calvin Chen <pi...@hotmail.com>]

Hi Kafka Team

Recently I moved Kafka cluster from CentOS8 to UbuntuServer20.04, same Kafka version(2.13-3.0.0), same Kafka configuration(check below), same JDK(openjdk-11-jdk) in server, but I get python client failed to connect.
# SASL-SSL
security.inter.broker.protocol=SASL_SSL
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
ssl.client.auth=required
ssl.endpoint.identification.algorithm=
ssl.keystore.location=/data/ssl/2022-03-25/kafka.server.keystore.jks
ssl.keystore.password=sasl_ssl
ssl.key.password=sasl_ssl
ssl.truststore.location=/data/ssl/2022-03-25/kafka.server.truststore.jks
ssl.truststore.password=sasl_ssl
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.truststore.type=JKS
ssl.keystore.type=JKS

I create client jks file, and convert ca-cert to python pem, my java application can send/recv message from/to Kafka successfully.
keytool -keystore kafka.truststore.jks -alias CARoot -import -file ca-cert -storepass sasl_ssl -keypass sasl_ssl -noprompt
openssl x509 -in ca-cert -out ca-cert.pem

But confluent python client failed to talk to Kafka cluster, python client shows:
%3|1648225510.555|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://sc-dev-kafka01a.eng.vmware.com:9093/bootstrap]: sasl_ssl://sc-dev-kafka01a.eng.vmware.com:9093/bootstrap: SSL handshake failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (after 215ms in state SSL_HANDSHAKE)
client error: KafkaError{code=_SSL,val=-181,str="sasl_ssl://sc-dev-kafka01a.eng.vmware.com:9093/bootstrap: SSL handshake failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (after 215ms in state SSL_HANDSHAKE)"}
client error: KafkaError{code=_ALL_BROKERS_DOWN,val=-187,str="1/1 brokers are down"}
Traceback (most recent call last):
  File "consumer.py", line 44, in <module>
    msg = consumer.poll(timeout=1.0)
  File "consumer.py", line 7, in error_callback
    raise KafkaException(err)
cimpl.KafkaException: KafkaError{code=_SSL,val=-181,str="sasl_ssl://sc-dev-kafka01a.eng.vmware.com:9093/bootstrap: SSL handshake failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (after 215ms in state SSL_HANDSHAKE)"}

And Kafka server said:
[2022-03-25 16:25:10,554] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /10.117.238.223 (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Python client code:
config = {
    "bootstrap.servers": "sc-dev-kafka01a.eng.vmware.com:9093",
    # "bootstrap.servers": "sc-dev-kafka01a.eng.vmware.com:9093, \
    #                       sc-dev-kafka01b.eng.vmware.com:9093, \
    #                       sc-dev-kafka01c.eng.vmware.com:9093",
    "group.id": "event-cg-mirror",
    "security.protocol": "SASL_SSL",
    "ssl.ca.location": "/home/pingc/KafkaSecurity/kafka-client/certs/2022-03-25/ca-cert.pem",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": "kms-user",
    "sasl.password": "test",
    "error_cb": error_callback,
}


Could anyone help to check, why after move to UbuntuServer, python client doesn't work while java client can, and there is no change in Kafka server/config and client side, thanks.

-Calvin