Re: Does the kafka controller protocol have a fallback?

[Dima Brodsky <dd...@gmail.com>]

Just following up, I realized I forgot to add some information.

This is using kafka 3.5.1,

I am in the process of setting up a kafka cluster which is configured to
> use KRaft.  There is a set of three controller nodes and a set of six
> brokers.  Both the controllers and the brokers are configured to use mTLS
> (Mutual TLS).  So the part of the controller config looks like:
>
> listeners=CONTROLLER://:9097
> listener.security.protocol.map=CONTROLLER:SSL
> controller.listener.names=CONTROLLER
>
> Now the certificates initially were missing a SAN that corresponded to the
> fqdn of the nodes.  The fqdn was used in creating the controller quorum
> voters config.
>
> When the controllers started up I did not see any errors or issues.  When
> the brokers started up I saw a couple of SSL connection errors when it
> tried to connect to the controllers, giving the controller hostname was
> missing from the SAN of the certificate.  But the whole cluster seemed to
> function normally.  No other errors and everything was in sync.  And the
> kafka-metadata-quorum.sh ... describe --status showed the correct status of
> the controllers and the brokers.
>

I saw some errors such as:

2024-04-16 20:05:57,924 [broker-4-to-controller-forwarding-channel-manager]
ERROR (kafka.server.BrokerToControllerRequestThread) -
[broker-4-to-controller-forwarding-channel-manager]: Request
EnvelopeRequestData(requestData=java.nio.HeapByteBuffer[pos=0 lim=169
cap=169], requestPrincipal=[0, 0, 5, 85, 115, 101, 114, 23, 97, 106, 110,
97, 108, 111, 99, 97, 108, 116, 101, 115, 116, 49, 46, 97, 106, 110, 97,
107, 109, 102, 0, 0], clientHostAddress=[10, 16, -125, -115]) failed due to
authentication error with controller

2024-04-16 20:05:57,924 [broker-4-to-controller-forwarding-channel-manager]
ERROR (kafka.server.BrokerToControllerRequestThread) -
[broker-4-to-controller-forwarding-channel-manager]: Failed to send the
following request due to authentication error:
ClientRequest(expectResponse=true,
callback=kafka.server.BrokerToControllerRequestThread$$Lambda$695/0x00000007c0703440@35746167,
destination=900, correlationId=7, clientId=4, createdTimeMs=1713297957917,
requestBuilder=EnvelopeRequestData(requestData=java.nio.HeapByteBuffer[pos=0
lim=169 cap=169], requestPrincipal=[0, 0, 5, 85, 115, 101, 114, 23, 97,
106, 110, 97, 108, 111, 99, 97, 108, 116, 101, 115, 116, 49, 46, 97, 106,
110, 97, 107, 109, 102, 0, 0], clientHostAddress=[10, 16, -125, -115]))

2024-04-16 20:05:57,923 [broker-4-to-controller-forwarding-channel-manager]
ERROR (org.apache.kafka.clients.NetworkClient) -
[BrokerToControllerChannelManager id=4 name=forwarding] Connection to node
900
(kafka-controller-0.kafka-controller-internal-svc.ajnalocaltest1.svc.cluster.local/
10.16.135.173:9097) failed authentication due to: SSL handshake failed


 I fixed the SAN in the cert and the errors went away on the brokers.
>
> My question is if the certs prevented the SSL connection from being
> established between the brokers and the controllers, or even between the
> controllers, is there some fallback that was used?  PLAINTEXT or was some
> of the validation skipped?
>

Thanks!
ttyl
Dima