[jira] [Created] (KAFKA-7845) NotLeaderForPartitionException error when publishing after a broker has died

["Jennifer Thompson (JIRA)" <ji...@apache.org>]

Jennifer Thompson created KAFKA-7845:
----------------------------------------

             Summary: NotLeaderForPartitionException error when publishing after a broker has died
                 Key: KAFKA-7845
                 URL: https://issues.apache.org/jira/browse/KAFKA-7845
             Project: Kafka
          Issue Type: Bug
          Components: clients
    Affects Versions: 2.1.0
            Reporter: Jennifer Thompson


When one of our Kafka brokers dies and a new one replaces it (via an aws ASG), the clients that publish to Kafka still try to publish to the old brokers.

We see errors like 
{code:java}
2019-01-18 20:16:16 WARN NetworkClient:721 - [Producer clientId=producer-1] Connection to node 2 (/10.130.98.111:9092) could not be established. Broker may not be available.
2019-01-18 20:19:09 WARN Sender:596 - [Producer clientId=producer-1] Got error produce response with correlation id 3414 on topic-partition aa.pga-2, retrying (4 attempts left). Error: NOT_LEADER_FOR_PARTITION
2019-01-18 20:19:09 WARN Sender:641 - [Producer clientId=producer-1] Received invalid metadata error in produce request on partition aa.pga-2 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now
2019-01-18 20:21:19 WARN NetworkClient:721 - [Producer clientId=producer-1] Connection to node 2 (/10.130.98.111:9092) could not be established. Broker may not be available.
2019-01-18 20:21:50 ERROR ProducerBatch:233 - Error executing user-provided callback on message for topic-partition 'aa.test-liz-0'{code}
and
{code:java}
[2019-01-18 20:28:47,732] ERROR WorkerSourceTask{id=rabbit-vpc-2-kafka-1} Failed to flush, timed out while waiting for producer to flush outstanding 27 messages (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2019-01-18 20:28:47,732] ERROR WorkerSourceTask{id=rabbit-vpc-2-kafka-1} Failed to commit offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter)
{code}
The ip address referenced is for the broker that died. We have Kafka Manager running as well, and that picks up the new broker.

This started happening after we upgraded to 2.1. When had Kafka 1.1, brokers could failover without a problem.

One thing that might be considered unusual about our deployment is that we reuse the same broker id and EBS volume for the new broker, so that partitions do not have to be reassigned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)