[jira] [Created] (KAFKA-7958) Transactions are broken with kubernetes hosted brokers

["Thomas Dickinson (JIRA)" <ji...@apache.org>]

Thomas Dickinson created KAFKA-7958:
---------------------------------------

             Summary: Transactions are broken with kubernetes hosted brokers
                 Key: KAFKA-7958
                 URL: https://issues.apache.org/jira/browse/KAFKA-7958
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 2.1.1
         Environment: cp-kakfka 2.1.1-1, kafka-streams 2.1.1
            Reporter: Thomas Dickinson


After a rolling re-start in a kubernetes-like environment, brokers may change IP address.  From our logs it seems that the transaction manager in the brokers never re-resolves the DNS name of other brokers, keeping stale pod IPs.  Thus transactions stop working.  

??[2019-02-20 02:20:20,085] WARN [TransactionCoordinator id=1001] Connection to node 0 (khaki-joey-kafka-0.khaki-joey-kafka-headless.hyperspace-dev/[10.233.124.181:9092|http://10.233.124.181:9092/]) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)??

??[2019-02-20 02:20:57,205] WARN [TransactionCoordinator id=1001] Connection to node 1 (khaki-joey-kafka-1.khaki-joey-kafka-headless.hyperspace-dev/[10.233.122.67:9092|http://10.233.122.67:9092/]) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)??

This is from the log from broker 1001 which was restarted first, followed by 1 and then 0.  The log entries are from the day after the rolling restart.

I note a similar issue was fixed for clients 2.1.1  https://issues.apache.org/jira/browse/KAFKA-7755.  We are using streams lib 2.1.1

We have turned off EOS in our stream applications to work-around this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)