You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yuxin Tan (Jira)" <ji...@apache.org> on 2023/12/05 04:13:00 UTC

[jira] [Created] (FLINK-33742) Hybrid Shuffle should work well with Adaptive Query Execution

Yuxin Tan created FLINK-33742:
---------------------------------

             Summary: Hybrid Shuffle should work well with Adaptive Query Execution
                 Key: FLINK-33742
                 URL: https://issues.apache.org/jira/browse/FLINK-33742
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Coordination, Runtime / Network
    Affects Versions: 1.19.0
            Reporter: Yuxin Tan


At present, Hybrid Shuffle and Adaptive Query Execution (AQE), which includes features such as Dynamic Partition Pruning (DPP), Runtime Filter, and Adaptive Batch Scheduler, are not fully compatible. While they can be used concurrently at the same time, the activation of AQE inhibits the key capability of Hybrid Shuffle to perform simultaneous reading and writing. This limitation arises because AQE dictates that downstream tasks may only initiate once upstream tasks have finished, a requirement that is inconsistent with the simultaneous read-write process facilitated by Hybrid Shuffle.

To harness the full potential of Hybrid Shuffle and AQE, it is essential to refine their integration. By doing so, we can capitalize on each feature's distinct advantages and enhance overall system performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)