[jira] [Created] (FLINK-33360) HybridSource fails to clear the previous round's state when switching sources, leading to data loss

["Feng Jiajie (Jira)" <ji...@apache.org>]

Feng Jiajie created FLINK-33360:
-----------------------------------

             Summary: HybridSource fails to clear the previous round's state when switching sources, leading to data loss
                 Key: FLINK-33360
                 URL: https://issues.apache.org/jira/browse/FLINK-33360
             Project: Flink
          Issue Type: Bug
          Components: Connectors / HybridSource
    Affects Versions: 1.17.1, 1.16.2
            Reporter: Feng Jiajie
             Fix For: 1.7.3


org.apache.flink.connector.base.source.hybrid.HybridSourceSplitEnumerator:
{code:java}
            // track readers that have finished processing for current enumerator
            finishedReaders.add(subtaskId);
            if (finishedReaders.size() == context.currentParallelism()) {
                LOG.debug("All readers finished, ready to switch enumerator!");
                if (currentSourceIndex + 1 < sources.size()) {
                    switchEnumerator();
                    // switch all readers prior to sending split assignments
                    for (int i = 0; i < context.currentParallelism(); i++) {
                        sendSwitchSourceEvent(i, currentSourceIndex);
                    }
                }
            } {code}
I think that *finishedReaders* is used to keep track of all the subTaskIds that have finished reading the current round of the source. Therefore, in the *switchEnumerator* function, *finishedReaders* should be cleared:

If it's not cleared, then in the next source reading, whenever any SourceReader reports a *SourceReaderFinishedEvent* (while other SourceReaders may not have finished processing in parallel), the condition *finishedReaders.size() == context.currentParallelism()* will be satisfied and it will trigger {*}sendSwitchSourceEvent{*}(i, currentSourceIndex), sending a *SwitchSourceEvent* to all SourceReaders.
If a SourceReader receives a SwitchSourceEvent before it finishes reading the previous source, it will execute {*}currentReader.close(){*}, and some data may not be fully read, resulting in a partial data loss in the source.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)