You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Ran Tao (Jira)" <ji...@apache.org> on 2023/03/03 17:45:00 UTC
[jira] (FLINK-31006) job is not finished when using pipeline mode to run bounded source like kafka/pulsar
[ https://issues.apache.org/jira/browse/FLINK-31006 ]
Ran Tao deleted comment on FLINK-31006:
---------------------------------
was (Author: lemonjing):
Hi, guys. I think the PR is not work and may cause another bug.
If we set `noMoreNewPartitionSplits = true;` when partitionDiscovery is disabled.
because *context.callAsync to get partitions is a async call.* If call is not finished, then code enter addReader:
!image-2023-03-04-01-07-04-927.png|width=588,height=381!
!image-2023-03-04-01-09-24-199.png|width=581,height=176!
!image-2023-03-04-01-08-29-042.png|width=596,height=636!
it will quit early. it means we can not consume bounded source.
However we expect it should read first partitions and consume and then quit.
WDYT?
> job is not finished when using pipeline mode to run bounded source like kafka/pulsar
> ------------------------------------------------------------------------------------
>
> Key: FLINK-31006
> URL: https://issues.apache.org/jira/browse/FLINK-31006
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Connectors / Pulsar
> Affects Versions: 1.17.0
> Reporter: jackylau
> Assignee: jackylau
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0
>
> Attachments: image-2023-02-10-13-20-52-890.png, image-2023-02-10-13-23-38-430.png, image-2023-02-10-13-24-46-929.png, image-2023-03-04-01-04-18-658.png, image-2023-03-04-01-05-25-335.png, image-2023-03-04-01-07-04-927.png, image-2023-03-04-01-07-36-168.png, image-2023-03-04-01-08-29-042.png, image-2023-03-04-01-09-24-199.png
>
>
> when i do failover works like kill jm/tm when using pipeline mode to run bounded source like kafka, i found job is not finished, when every partition data has consumed.
>
> After dig into code, i found this logical not run when JM recover. the partition infos are not changed. so noMoreNewPartitionSplits is not set to true. then this will not run
>
> !image-2023-02-10-13-23-38-430.png!
>
> !image-2023-02-10-13-24-46-929.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)