Dynamic update of server-side keystores - apache kafka 2.3.1

[PraneethKumar <pr...@gmail.com>]

I have few queries related to enabling SSL/TLS on kafka with dynamic update
of certs using kafka-configs.sh and i am running into issues when certs are
expired

Server Side
===================
I have generated short lived keystore and truststore certificates (4hrs
expiry time) and placed details in a file(abc.properties) with below content
ssl.keystore.location=/tmp/kafka.server.keystore.jks
ssl.keystore.password=<<KEYSTORE_PASSWORD>>
ssl.truststore.location=/tmp/kafka.server.truststore.jks
ssl.truststore.password=<<TRUSTSTORE_PASSWORD>>

kafka 2.3.1 version is used and kafka server.properites configuration as
below

security_protocol=SSL
client_auth=required
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
listeners=PLAINTEXT://host:9092,SSL://host:9094
advertised.listeners=PLAINTEXT://host:9092,SSL://host:9094
security.inter.broker.protocol=PLAINTEXT
# keystore/truststore location and password values are read from configFile
provider
ssl.keystore.location=${file:abc.properties:ssl.keystore.location}
ssl.keystore.password=${file:abc.properties:ssl.keystore.password}
ssl.truststore.location=${file:abc.properties:ssl.truststore.location}
ssl.truststore.password=${file:abc.properties:ssl.truststore.password}
config.providers=file
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider


At Client Side
====================
Components which connects to kafka was started successfully and able to
establish connections to server over Mutual SSL/TLS


Server Side
========================
I have a process which does the following when certs are nearing its expiry
time
1. generates certificates with cert validity of 4 hours
2. updates new keystore and truststore cert details dynamically using
kafka-configs.sh

sh ./bin/kafka-configs.sh --bootstrap-server host:9094
--entity-name {broker_id} --entity-type brokers --alter --add-config
'listener.name.SSL.ssl.truststore.location=/var/private/ssl/kafka.server.truststore.20200818120021.jks,
listener.name.SSL.ssl.truststore.password=<<TRUSTSTORE_PASSWORD>>,listener.name.SSL.ssl.keystore.location=/var/private/ssl/kafka.server.keystore.20200818120021.jks,listener.name.SSL.ssl.keystore.password=<<KEYSTORE_PASSWORD>>'

--command-config /tmp/client-ssl.properties

contents of client-ssl.properties
security.protocol=SSL
ssl.keystore.location=/tmp/kafka.client.keystore.jks
ssl.keystore.password=<<KEYSTOREPASSWORD>>
ssl.truststore.location=/tmp/kafka.client.truststore.jks
ssl.truststore.password=<<TRUSTSTOREPASSWORD>>

above command provides below output
Completed updating config for broker: 0.

Post that when i do a describe of broker configs using kafka-configs.sh I
can see DYNAMIC BROKER config has values with new cert details and
STATIC BROKER CONFIG has cert details which were generated at server startup

sh ./bin/kafka-configs.sh --bootstrap-server hostname:9094 --entity-name 0
--entity-type brokers --describe --command-config /tmp/client-ssl.properties
Configs for broker 0 are:

listener.name.SSL.ssl.truststore.location=/var/private/ssl/kafka.server.truststore.20200818120021.jks
sensitive=false
synonyms={DYNAMIC_BROKER_CONFIG:listener.name.SSL.ssl.truststore.location=/var/private/ssl/kafka.server.truststore.20200818120021.jks,
STATIC_BROKER_CONFIG:ssl.truststore.location=/tmp/kafka.server.truststore.jks}
  listener.name.SSL.ssl.truststore.password=null sensitive=true
synonyms={DYNAMIC_BROKER_CONFIG:listener.name.SSL.ssl.truststore.password=null,
STATIC_BROKER_CONFIG:ssl.truststore.password=null}

listener.name.SSL.ssl.keystore.location=/var/private/ssl/kafka.server.keystore.20200818120021.jks
sensitive=false
synonyms={DYNAMIC_BROKER_CONFIG:listener.name.SSL.ssl.keystore.location=/var/private/ssl/kafka.server.keystore.20200818120021.jks,
STATIC_BROKER_CONFIG:ssl.keystore.location=/tmp/kafka.server.keystore.jks}
  listener.name.SSL.ssl.keystore.password=null sensitive=true
synonyms={DYNAMIC_BROKER_CONFIG:listener.name.SSL.ssl.keystore.password=null,
STATIC_BROKER_CONFIG:ssl.keystore.password=null}


After 4 hours client threw below exception
============================================================
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Tue
Aug 18 13:09:13 BST 2020

Tue Aug 18 13:09:13 BST 2020 is the expiry time of the certificate which
was generated during server startup

I have also tried below options
1. updating keystore details and then truststore details as two different
commands but it does not work and i get below error
missing entries: Set(listener.name.SSL.ssl.keystore.password)

2. changing client_auth to requested and none - I get same certificate
expiry error

I expect the client to connect successfully with no exceptions as the
dynamic broker update was successful

Any help appreciated

Thanks & Regards
Praneeth