[jira] [Assigned] (FLINK-29425) Hybrid full spilling strategy triggering spilling frequently

["Xintong Song (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-29425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xintong Song reassigned FLINK-29425:
------------------------------------

    Assignee: Weijie Guo

> Hybrid full spilling strategy triggering spilling frequently
> ------------------------------------------------------------
>
>                 Key: FLINK-29425
>                 URL: https://issues.apache.org/jira/browse/FLINK-29425
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Network
>    Affects Versions: 1.16.0
>            Reporter: Weijie Guo
>            Assignee: Weijie Guo
>            Priority: Critical
>              Labels: pull-request-available
>
> In hybrid shuffle mode, we have an internal config option 'DEFAULT_FULL_STRATEGY_NUM_BUFFERS_TRIGGER_SPILLED' to control spilling frequency in the full spilling strategy. Unfortunately, the default value(10) is too small. As a result, frequent disk spilling calls are observed in the TPC-DS test, which seriously affects performance. When we increase the value, the query performance is improved significantly. We should set a more reasonable default value, or adopt an adaptive strategy to determine the spilling frequency.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)