[jira] [Updated] (FLINK-27862) FLIP-235: Hybrid Shuffle Mode

["Weijie Guo (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Weijie Guo updated FLINK-27862:
-------------------------------
    Release Note: 
We have introduced a new Hybrid Shuffle Mode for batch executions. It combines the advantages of blocking shuffle and pipelined shuffle (in streaming mode).
- Like blocking shuffle, it does not require upstream and downstream tasks to run simultaneously, which allows executing a job with little resources.
- Like pipelined shuffle, it does not require downstream tasks to be executed after upstream tasks finish, which reduces the overall execution time of the job when given sufficient resources.
- It adapts to custom preferences between persisting less data and restarting less tasks on failures, by providing different spilling strategies.

For more details: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/batch/batch_shuffle

  was:
We have introduced a new Hybrid Shuffle Mode for batch executions. It combines the advantages of blocking shuffle and pipelined shuffle (in streaming mode).
- Like blocking shuffle, it does not require upstream and downstream tasks to run simultaneously, which allows executing a job with little resources.
- Like pipelined shuffle, it does not require downstream tasks to be executed after upstream tasks finish, which reduces the overall execution time of the job when given sufficient resources.
- It adapts to custom preferences between persisting less data and restarting less tasks on failures, by providing different spilling strategies.

For more details: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/batch/batch_shuffle.md


> FLIP-235: Hybrid Shuffle Mode
> -----------------------------
>
>                 Key: FLINK-27862
>                 URL: https://issues.apache.org/jira/browse/FLINK-27862
>             Project: Flink
>          Issue Type: New Feature
>          Components: Runtime / Network
>            Reporter: Weijie Guo
>            Assignee: Weijie Guo
>            Priority: Major
>              Labels: Umbrella
>             Fix For: 1.16.0
>
>
> Introduce a new shuffle mode can overcome some of the problems of Pipelined Shuffle and Blocking Shuffle in batch scenarios, it can make best use of available resources and minimize disk IO load.
> More details see [FLIP-235|https://cwiki.apache.org/confluence/display/FLINK/FLIP-235%3A+Hybrid+Shuffle+Mode]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)