[jira] [Closed] (FLINK-25995) Make implicit assumption of SQL local hash explicit

["godfrey he (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-25995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

godfrey he closed FLINK-25995.
------------------------------
    Resolution: Fixed

Fixed in 1.15.0: 546af8424b32ea13cfeec0a12561b6b73e5f8e7d

> Make implicit assumption of SQL local hash explicit
> ---------------------------------------------------
>
>                 Key: FLINK-25995
>                 URL: https://issues.apache.org/jira/browse/FLINK-25995
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table SQL / Planner
>    Affects Versions: 1.15.0
>            Reporter: Zhu Zhu
>            Assignee: godfrey he
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.15.0
>
>
> If there are multiple consecutive and the same hash shuffles, SQL planner will change them except the first one to use forward partitioner, so that these operators can be chained to reduce unnecessary shuffles.
> However, sometimes the local hash operators are not chained (e.g. multiple inputs), and  this kind of forward partitioners will turn into forward job edges. These forward edges still have the local keyBy assumption, so that they cannot be changed into rescale/rebalance edges, otherwise it can lead to incorrect results. This prevents the adaptive batch scheduler from determining parallelism for other forward edge downstream job vertices (see FLINK-25046).
> When SQL planner optimizes the case of multiple consecutive the same groupBy, it should use {{ForwardForConsecutiveHashPartitioner}}, so that the runtime framework can further decide whether the partitioner can be changed to rescale or not.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)