You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jiangjie Qin (JIRA)" <ji...@apache.org> on 2019/07/12 03:51:00 UTC
[jira] [Comment Edited] (FLINK-11736) flink kafka producer failed
with NOT_LEADER_FOR_PARTITION
[ https://issues.apache.org/jira/browse/FLINK-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883502#comment-16883502 ]
Jiangjie Qin edited comment on FLINK-11736 at 7/12/19 3:50 AM:
---------------------------------------------------------------
[~debugman] NOT_LEADER_FOR_PARTITION indicates that there was a leader migration in the Kafka cluster for some partition. Such information of the new leader will be propagated to all the brokers in the cluster asynchronously. It may take some time, typically within 1 second, for all the brokers in the Kafka cluster to receive the updated leader information.
From the timestamp in the log, it seems the producer was retrying with a backoff around 3 ms. This may exhaust the 10 retries in 30 ms, which may be too aggressive. The default value of \{{retry.backoff.ms}} is actually 100ms. Did you happened to change that?
was (Author: becket_qin):
[~debugman] NOT_LEADER_FOR_PARTITION indicates that there was a leader migration in the Kafka cluster for some partition. Such information of the new leader will be propagated to all the brokers in the cluster asynchronously. It may take some time, typically within 1 second, for all the brokers in the Kafka cluster to receive the updated leader information.
From the timestamp in the log, it seems the producer was retrying with a backoff around 3 ms. This may exhaust the 10 retries in 30 ms, which may be too aggressive. The default value is actually 100ms. Did you happened to change that?
> flink kafka producer failed with NOT_LEADER_FOR_PARTITION
> ---------------------------------------------------------
>
> Key: FLINK-11736
> URL: https://issues.apache.org/jira/browse/FLINK-11736
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Reporter: Kaicheng Sun
> Priority: Major
>
> my flink program connect kafka as its sinker using "FlinkKafkaProducer011" library. But sometimes the flink will exit abnormally with this kind of error:
> (2019-02-23 11:55:11,656 WARN org.apache.kafka.clients.producer.internals.Sender - Got error produce response with correlation id 55711 on topic-partition tmp_sink_redis-17, retrying (8 attempts left). Error: NOT_LEADER_FOR_PARTITION
> 2019-02-23 11:55:11,658 WARN org.apache.kafka.clients.producer.internals.Sender - Got error produce response with correlation id 55712 on topic-partition tmp_sink_redis-17, retrying (8 attempts left). Error: NOT_LEADER_FOR_PARTITION)
>
> The kafka cluster works properly, so I have no idea why this error will happen
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)