You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Fabian Paul (Jira)" <ji...@apache.org> on 2021/09/21 11:18:00 UTC
[jira] [Created] (FLINK-24347) KafkaSource cannot checkpoint if the
parallelism is higher than the partition number
Fabian Paul created FLINK-24347:
-----------------------------------
Summary: KafkaSource cannot checkpoint if the parallelism is higher than the partition number
Key: FLINK-24347
URL: https://issues.apache.org/jira/browse/FLINK-24347
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.13.2, 1.14.0
Reporter: Fabian Paul
The KafkaSourceEnumerator signals all the readers if there are no more splits available and the readers shut down. In case the parallelism is higher than the partitions of the consumed topic there are not enough partitions to distribute and reader subtasks go immediately into FINISHED state.
Currently, it is not possible to checkpoint if parts of the job are finished. (this is lifted once FLIP-147 is by default enabled Flink 1.15+)
We should only signal to the readers to go into finished if the job is in bounded execution mode and keep them idling otherwise.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)