[jira] [Created] (KAFKA-7450) kafka controller RequestSendThread stuck in infinite loop after SSL handshake failure with peer brokers

["Yu Yang (JIRA)" <ji...@apache.org>]

Yu Yang created KAFKA-7450:
------------------------------

             Summary: kafka controller RequestSendThread stuck in infinite loop after SSL handshake failure with peer brokers
                 Key: KAFKA-7450
                 URL: https://issues.apache.org/jira/browse/KAFKA-7450
             Project: Kafka
          Issue Type: Bug
          Components: controller
    Affects Versions: 2.0.0
            Reporter: Yu Yang


After updating security.inter.broker.protocol to SSL for our cluster, we observed that the controller can get into almost 100% cpu usage. 

{code}
listeners=PLAINTEXT://:9092,SSL://:9093
security.inter.broker.protocol=SSL
{code}

There is no obvious error in server.log. But in controller.log, there is repetitive SSL handshare failure error as below: 

{code}
[2018-09-28 05:53:10,821] WARN [RequestSendThread controllerId=6042] Controller 6042's connection to broker datakafka06176.ec2.pin220.com:9093 (id: 6176 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1487)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:468)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:125)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
        at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
        at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
        at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:196)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
        ... 10 more

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)