Kafka SSL setup - producer timing out

["Milakovic, Srdjan" <sm...@plex.com>]

I am trying to set up a three node Kafka v1.1 cluster with SSL. I can consume messages via the SSL port but I cannot produce. The command is timing out after 60 seconds with the following error message:

ERROR Error when sending message to topic myTopic with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

I am using the kafka-console-producer.sh and kafka-console-consumer.sh scripts on one of the broker nodes in order to do my testing. Both consumer and producer are using the same properties file (client-ssl-auth.properties) which looks like this:

security.protocol=SSL
ssl.truststore.location=/home/sshuser/ssl/kafka.server.truststore.jks
ssl.truststore.password=dummy_password
ssl.keystore.location=/home/sshuser/ssl/kafka.server.keystore.jks
ssl.keystore.password=dummy_password
ssl.key.password=dummy_password

This is how the consumer command looks:

./kafka-console-consumer.sh --bootstrap-server 10.0.0.36:9093 --topic myTopic --from-beginning --consumer.config client-ssl-auth.properties

This is how the producer command looks:

./kafka-console-producer.sh --broker-list 10.0.0.36:9093 --topic myTopic --producer.config client-ssl-auth.properties

I can verify the functionality of the consumer because I can consume messages that are produced over the non-encrypted port.

The advertised.listeners configuration is set to the broker IP instead of the broker FQDN like so:

advertised.listeners=PLAINTEXT://10.0.0.36:9092,SSL://10.0.0.36:9093

There are no SSL handshake errors in the server.log. This is what I was able to find in the server.log file on one of the nodes:

[2019-03-25 14:55:25,428] WARN [GroupCoordinator 1003]: Failed to write empty metadata for group console-consumer-21670: The group is rebalancing, so a rejoin is needed...

The commands behave the same regardless on which node I execute them on and regardless of the broker IP I use.

I can produce and consume messages on that topic via non-encrypted ports without issues.

Any help will be appreciated.

Thank you.