You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yuxin Tan (Jira)" <ji...@apache.org> on 2023/08/01 10:09:00 UTC
[jira] [Updated] (FLINK-31634) FLIP-301: Hybrid Shuffle supports Remote Storage
[ https://issues.apache.org/jira/browse/FLINK-31634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuxin Tan updated FLINK-31634:
------------------------------
Release Note: Hybrid Shuffle supports storing the shuffle data in remote storage. The remote storage path can be configured with the option `taskmanager.network.hybrid-shuffle.remote.path`. Hybrid Shuffle uses less network memory than before by decoupling the memory usage from the number of parallelisms, improving the stability and ease of use. For detailed information, please refer to https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/#hybrid-shuffle (was: This introduces a new Hybrid Shuffle mode to support remote storage. The remote storage path is specified by `taskmanager.network.hybrid-shuffle.remote.path`. The new mode has resolved existing issues in the legacy Hybrid Shuffle mode. Firstly, the new mode uses less required network memory. Secondly, the new mode can store shuffle data in remote storage when the disk space is not enough, which could avoid insufficient disk space errors. The new mode is only enabled when `taskmanager.network.hybrid-shuffle.remote.path` is configured.
Currently, the new Hybrid Shuffle mode is in an experimental phase. In case of unexpected issues, the new mode can fall back to the legacy mode by setting `taskmanager.network.hybrid-shuffle.enable-new-mode` to `false`. Once the new mode is stable, the legacy mode and the fallback option will be removed.)
> FLIP-301: Hybrid Shuffle supports Remote Storage
> ------------------------------------------------
>
> Key: FLINK-31634
> URL: https://issues.apache.org/jira/browse/FLINK-31634
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Affects Versions: 1.18.0
> Reporter: Yuxin Tan
> Assignee: Yuxin Tan
> Priority: Major
> Labels: Umbrella
>
> This is an umbrella ticket for [FLIP-301|https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)