Kafka Dynamic Broker Configuration errors

[Daniel Guymon <vt...@gmail.com>]

 Hello all,

I'm currently playing around with Zookeeper's ability to store dynamic
Kafka configurations to protect sensitive Kafka settings (like keystore and
truststore passwords).  Working with a simple example of a single EC2
instance in AWS that has both a single Kafka broker and a single Zookeeper
node installed on it.

During my bootstrap script, while Zookeeper is up and running and while
Kafka is not yet started, I run the below with no issue:

+ /opt/kafka/latest/bin/kafka-configs.sh --zookeeper 10.99.215.93:2281
--zk-tls-config-file
/opt/kafka/latest/config/kafka-to-zookeeper-tls.properties --entity-type
brokers --entity-name 0 --alter --add-config
listener.name.ssl.ssl.truststore.password=changeit,password.encoder.secret=changeit
Warning: --zookeeper is deprecated and will be removed in a future version
of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: brokers '0'.
+ /opt/kafka/latest/bin/kafka-configs.sh --zookeeper 10.99.215.93:2281
--zk-tls-config-file
/opt/kafka/latest/config/kafka-to-zookeeper-tls.properties --entity-type
brokers --entity-name 0 --alter --add-config
listener.name.ssl.ssl.key.password=changeit,password.encoder.secret=changeit
Warning: --zookeeper is deprecated and will be removed in a future version
of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: brokers '0'.
+ /opt/kafka/latest/bin/kafka-configs.sh --zookeeper 10.99.215.93:2281
--zk-tls-config-file
/opt/kafka/latest/config/kafka-to-zookeeper-tls.properties --entity-type
brokers --entity-name 0 --alter --add-config
listener.name.ssl.ssl.keystore.password=changeit,password.encoder.secret=changeit
Warning: --zookeeper is deprecated and will be removed in a future version
of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: brokers '0'.

Then, during Kafka start-up, I see the below errors which ultimately causes
Kafka to enter a failed state:

[2021-10-19 15:16:10,485] ERROR Dynamic password config
listener.name.ssl.ssl.key.password could not be decoded, ignoring.
(kafka.server.DynamicBrokerConfig)
org.apache.kafka.common.config.ConfigException: Invalid value
javax.crypto.BadPaddingException: Given final block not properly padded.
Such issues can arise if a bad key is used during decryption. for
configuration Password could not be decoded
        at kafka.utils.PasswordEncoder.decode(PasswordEncoder.scala:104)
        at
kafka.server.DynamicBrokerConfig.decodePassword$1(DynamicBrokerConfig.scala:386)
        at
kafka.server.DynamicBrokerConfig.$anonfun$fromPersistentProps$5(DynamicBrokerConfig.scala:397)
        at
kafka.server.DynamicBrokerConfig.$anonfun$fromPersistentProps$5$adapted(DynamicBrokerConfig.scala:395)
        at
kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.MapOps.foreachEntry(Map.scala:211)
        at scala.collection.MapOps.foreachEntry$(Map.scala:207)
        at scala.collection.AbstractMap.foreachEntry(Map.scala:372)
        at
kafka.server.DynamicBrokerConfig.fromPersistentProps(DynamicBrokerConfig.scala:395)
        at
kafka.server.DynamicBrokerConfig.$anonfun$updateBrokerConfig$1(DynamicBrokerConfig.scala:293)
        at
kafka.server.DynamicBrokerConfig.updateBrokerConfig(DynamicBrokerConfig.scala:292)
        at
kafka.server.DynamicBrokerConfig.initialize(DynamicBrokerConfig.scala:216)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:227)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2021-10-19 15:16:10,551] ERROR Per-broker configs of 0 could not be
applied:
{listener.name.ssl.ssl.key.password=encryptedPassword:mfTt1/beJojQXOSdv11jVQ==,keyLength:128,cipherAlgorithm:AES/CBC/PKCS5Padding,initializationVector:1xhgT4bOgHEA0GzL5kPJkQ==,keyFactoryAlgorithm:PBKDF2WithHmacSHA512,passwordLength:8,salt:l+4hnx+Ia91VpGvyrU2A2dFhLHSRv5Pb1OAm+4TmDpxnsBjDOcPUMUmUnIe07vq0UWBpVcX5gXk/JVrEEAuSFcTOeOelbmMJ12guwbOgfiJCvQaYscPk+nasFBWN/kHryM94BBKgwil5obWXzDRIKuUJithro2Hoh4L0UKwxU9V9C9BH87AF94SAjxjVV8sMghgncJUDNLkfE1Fqe4mxnZJyzt6zzZAcoOMlkHYgG0leEYlPLwR1mm/Bv/5mBKrPUJdc/+lSQhHo6+3pzEl9HGv6a/uR/VX89vCP8LrqrZmYgJTPtvawFYx0feg6J8NIGqorfuTzQNRZJmD0X1vQVQ==,iterations:4096,
listener.name.ssl.ssl.truststore.password=encryptedPassword:fER6tx8eEZJWx/GGGn3z0w==,keyLength:128,cipherAlgorithm:AES/CBC/PKCS5Padding,initializationVector:wL3ZAN5xPhwy3LsPryK0Tg==,keyFactoryAlgorithm:PBKDF2WithHmacSHA512,passwordLength:8,salt:QA21rnyDHCrbBdB7PVEX0xUQbbkOSUFhtchd1V7DQsOx/L0JgSHZGk4tg3i8397tosUaGrX0ihQFVJeZkQb1rCNI5ifc2eIExjopKhV3ztY6sM6PUWRwf1CVQbfXhog6x082TI1k6H+1ua/O/KbeJ2btlgprzxhiuchOtLJmIR5v17h25zmDUyyZA7XCFZdWglFJWLnHCuGeXqREj0zQ9s6hd46aVnwUnxdqirlVjfLv9GaU8SocHjPwDGEVCvx1UL7P+jaV+Bi9OIVZVvvrRogu5KjlxvHWWRYAd6XSlwW6dlIMiShXCZbfo+FqFjj+pqVcUFq2/T12DbZFZGRhfw==,iterations:4096,
listener.name.ssl.ssl.keystore.password=encryptedPassword:icoGVBqyOLshplKCPSV8iw==,keyLength:128,cipherAlgorithm:AES/CBC/PKCS5Padding,initializationVector:mzBcrVWdbiJuyWTGf4bZfQ==,keyFactoryAlgorithm:PBKDF2WithHmacSHA512,passwordLength:8,salt:lHtk9e6lFIX0Gat38pLER8Pv115X68DzHB9uqV4royM3OUk9VN2YH/WSlqEtplpX82Me8FMMLZsIwxNw49ycco5U20FsATZ3DyAnTj9+ADHeRx8t4wpGj9apUbZncMTV6WeMPmJfA411ezh/PyPEP4oD56eOc2mKtMUg3ryPQT/oefrZcm2A0p1yJHELnlU8FD5y5Qs5ET29UtHkQFDPLElt6TCdZ1jtPDQxyAPSf1PsQBjJ9wweuaS9xB1heRUauS+5kg7Ykpp8tvi5PEl+x9KlmVSPSA8bJBiqwYqjIbYgCA8TIsXX/MBQqkibU70p4vDL3zoS91Fgx/gF3r2mcw==,iterations:4096}
(kafka.server.DynamicBrokerConfig)
java.util.ConcurrentModificationException
        at java.util.Hashtable$Enumerator.next(Hashtable.java:1387)
        at
scala.collection.convert.JavaCollectionWrappers$JPropertiesWrapper$$anon$6.next(JavaCollectionWrappers.scala:518)
        at
scala.collection.convert.JavaCollectionWrappers$JPropertiesWrapper$$anon$6.next(JavaCollectionWrappers.scala:514)
        at scala.collection.MapOps.foreachEntry(Map.scala:210)
        at scala.collection.MapOps.foreachEntry$(Map.scala:207)
        at scala.collection.AbstractMap.foreachEntry(Map.scala:372)
        at
kafka.server.DynamicBrokerConfig.fromPersistentProps(DynamicBrokerConfig.scala:395)
        at
kafka.server.DynamicBrokerConfig.$anonfun$updateBrokerConfig$1(DynamicBrokerConfig.scala:293)
        at
kafka.server.DynamicBrokerConfig.updateBrokerConfig(DynamicBrokerConfig.scala:292)
        at
kafka.server.DynamicBrokerConfig.initialize(DynamicBrokerConfig.scala:216)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:227)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)


Appreciate any assistance!

Regards,

-Danny