You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Flink Jira Bot (Jira)" <ji...@apache.org> on 2022/07/06 10:41:00 UTC
[jira] [Updated] (FLINK-26726) Remove the unregistered task from readersAwaitingSplit
[ https://issues.apache.org/jira/browse/FLINK-26726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flink Jira Bot updated FLINK-26726:
-----------------------------------
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone else may work on it.
> Remove the unregistered task from readersAwaitingSplit
> -------------------------------------------------------
>
> Key: FLINK-26726
> URL: https://issues.apache.org/jira/browse/FLINK-26726
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Ecosystem
> Reporter: zoucao
> Assignee: zoucao
> Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: stack.txt
>
>
> Recently, we faced a problem caused by the unregistered task when using the hive table as a source to do streaming reading.
> I think the problem is that we do not remove the unregistered task from `readersAwaitingSplit` in `ContinuousHiveSplitEnumerator` and `ContinuousFileSplitEnumerator`.
> Assuming that we have two tasks 0 and 1, they all exist in `readersAwaitingSplit`, if there does not exist any new file in the path for a long time. Then, a new split is generated, and it is assigned to task-1. Unfortunately, task-1 can not consume the split successfully, and the exception will be thrown and cause all tasks to restart. The failover will not affect the `readersAwaitingSplit`, but it will clear the `SourceCoordinatorContext#registeredReaders`.
> After restarting, task-0 exists in `readersAwaitingSplit` but not in `registeredReaders`. if task-1 register first and send the request to get split, the SplitEnumerator will assign splits for both task-1 and task-0, but task-0 has not been registered.
> The stack exists in the attachment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)