You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Weijie Guo (Jira)" <ji...@apache.org> on 2022/07/27 06:06:00 UTC

[jira] [Created] (FLINK-28701) Minimize the explosion range during failover for hybrid shuffle

Weijie Guo created FLINK-28701:
----------------------------------

             Summary: Minimize the explosion range during failover for hybrid shuffle
                 Key: FLINK-28701
                 URL: https://issues.apache.org/jira/browse/FLINK-28701
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
    Affects Versions: 1.16.0
            Reporter: Weijie Guo


In hybrid shuffle mode, there are currently two strategies to control spilling. For the full spilling strategy, the data is guaranteed to be persistent to the disk after task finished. When a failover occurs, if the upstream has been finished, the data should be recovered directly from the disk file without re-compute. For selective spilling strategy, the entire topology must be restarted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)