[jira] [Created] (FLINK-26726) Remove the unregistered task from readersAwaitingSplit

["zoucao (Jira)" <ji...@apache.org>]

zoucao created FLINK-26726:
------------------------------

             Summary: Remove the unregistered  task from readersAwaitingSplit
                 Key: FLINK-26726
                 URL: https://issues.apache.org/jira/browse/FLINK-26726
             Project: Flink
          Issue Type: Improvement
          Components: Table SQL / Ecosystem
            Reporter: zoucao
         Attachments: stack.txt

Recently, we faced a problem caused by the unregistered task when using the hive table as a source to do streaming reading. 
I think the problem is that we do not remove the unregistered  task from `readersAwaitingSplit` in `ContinuousHiveSplitEnumerator` and `ContinuousFileSplitEnumerator`.

Assuming that we have two tasks 0 and 1, they all exist in `readersAwaitingSplit`,  if there does not exist any new file in the path for a long time. Then, a new split is generated, and it is assigned to task-1. Unfortunately, task-1 can not consume the split successfully, and the exception will be thrown and cause all tasks to restart. The failover will not affect the `readersAwaitingSplit`, but it will clear the `SourceCoordinatorContext#registeredReaders`.
After restarting, task-0 exists in `readersAwaitingSplit` but not in `registeredReaders`. if task-1 register first and send the request to get split, the SplitEnumerator will assign splits for both task-1 and task-0, but task-0 has not been registered.


The stack exists in the attachment.








--
This message was sent by Atlassian Jira
(v8.20.1#820001)