You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "venkatakumar puvvada (JIRA)" <ji...@apache.org> on 2018/12/12 13:57:00 UTC

[jira] [Updated] (KAFKA-7727) kafka-producer-network-thread throwing Error: NOT_LEADER_FOR_PARTITION after adding new broker to kafka cluster

     [ https://issues.apache.org/jira/browse/KAFKA-7727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

venkatakumar puvvada updated KAFKA-7727:
----------------------------------------
    Description: 
Kafka producer network thread throwing Error:NOT_LEADER_FOR_PARTITION  after adding a broker to the cluster in the consumer group during the load.

Below are the steps followed 

Env: 3 brokers and 3 zookeeper 

1. Sending  4K messages/sec

2. Performed HA on one node (say node1).

3. continue to run the load

4. No problem in the consumer group.

5.start node 1 after sometime (say 10mins).

after adding the Kafka broker to the cluster below error throwing continuously.

 

2018-12-11 12:30:43.505 sp-46-e695c9a3-764f-44ec-a018-885005a21034-StreamThread-4 ERROR - Failed to process key x by worker:sp-work with exception

org.apache.kafka.streams.errors.StreamsException: task [0_19] Abort sending since an error caught with a previous record (key x value [B@4c4a09b7 timestamp 1544531442264) to topic x-changelog due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition..
at org.apache.kafka.streams.processor.internals.RecordCollectorImpl$1.onCompletion(RecordCollectorImpl.java:118)
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204)
at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596)
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557)
at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481)
at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74)
at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692)
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.

  was:
Kafka producer network thread throwing Error:NOT_LEADER_FOR_PARTITION  after adding a broker to the cluster in the consumer group during the load.

Below are the steps followed 

Env: 3 brokers and 3 zookeeper 

1. Sending  4K messages/sec

2. Performed HA on one node (say node1).

3. continue to run the load

4. No problem in the consumer group.

5.start node 1 after sometime (say 10mins).

after adding the Kafka broker to the cluster below error throwing continuously.

 

2018-12-11 12:30:43.505 sp-46-e695c9a3-764f-44ec-a018-885005a21034-StreamThread-4 ERROR - Failed to process key x by worker:y with exception

org.apache.kafka.streams.errors.StreamsException: task [0_19] Abort sending since an error caught with a previous record (key x value [B@4c4a09b7 timestamp 1544531442264) to topic x-changelog due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition..
at org.apache.kafka.streams.processor.internals.RecordCollectorImpl$1.onCompletion(RecordCollectorImpl.java:118)
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204)
at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596)
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557)
at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481)
at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74)
at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692)
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.


> kafka-producer-network-thread  throwing Error: NOT_LEADER_FOR_PARTITION after adding new broker to kafka cluster
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-7727
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7727
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 1.0.0
>         Environment: docker
>            Reporter: venkatakumar puvvada
>            Priority: Major
>
> Kafka producer network thread throwing Error:NOT_LEADER_FOR_PARTITION  after adding a broker to the cluster in the consumer group during the load.
> Below are the steps followed 
> Env: 3 brokers and 3 zookeeper 
> 1. Sending  4K messages/sec
> 2. Performed HA on one node (say node1).
> 3. continue to run the load
> 4. No problem in the consumer group.
> 5.start node 1 after sometime (say 10mins).
> after adding the Kafka broker to the cluster below error throwing continuously.
>  
> 2018-12-11 12:30:43.505 sp-46-e695c9a3-764f-44ec-a018-885005a21034-StreamThread-4 ERROR - Failed to process key x by worker:sp-work with exception
> org.apache.kafka.streams.errors.StreamsException: task [0_19] Abort sending since an error caught with a previous record (key x value [B@4c4a09b7 timestamp 1544531442264) to topic x-changelog due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition..
> at org.apache.kafka.streams.processor.internals.RecordCollectorImpl$1.onCompletion(RecordCollectorImpl.java:118)
> at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204)
> at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187)
> at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627)
> at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596)
> at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557)
> at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481)
> at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74)
> at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692)
> at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
> at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)