You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "yansuopeng (Jira)" <ji...@apache.org> on 2024/04/03 09:16:00 UTC

[jira] [Created] (FLINK-34995) flink kafka connector source stuck when partition leader invalid

yansuopeng created FLINK-34995:
----------------------------------

             Summary: flink kafka connector source stuck when partition leader invalid
                 Key: FLINK-34995
                 URL: https://issues.apache.org/jira/browse/FLINK-34995
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kafka
    Affects Versions: 1.18.1, 1.19.0, 1.17.0
            Reporter: yansuopeng


when partition leader invalid(leader=-1),  the flink streaming job using KafkaSource can't restart or start a new instance with a new groupid,  it will stuck and got following exception:

"{*}org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition aaa-1 could be determined{*}"

when leader=-1,  kafka api like KafkaConsumer.position() will block until either the position could be determined or an unrecoverable error is encountered 

infact,  leader=-1 not easy to avoid,  even replica=3, three disk offline together will trigger the problem, especially when the cluster size is relatively large.    it rely on kafka administrator to fix in time,  but it take risk when in kafka cluster peak period.

 
 
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)