You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Yun Gao (Jira)" <ji...@apache.org> on 2022/01/27 08:06:00 UTC

[jira] [Commented] (FLINK-25781) Adjust the maximum number of buffers can be used per result partition for data read of sort-shuffle

    [ https://issues.apache.org/jira/browse/FLINK-25781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482952#comment-17482952 ] 

Yun Gao commented on FLINK-25781:
---------------------------------

Fix on master via 34d0972655deb33b68768f641b7ca97dec30a768

> Adjust the maximum number of buffers can be used per result partition for data read of sort-shuffle
> ---------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-25781
>                 URL: https://issues.apache.org/jira/browse/FLINK-25781
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Network
>            Reporter: Yingjie Cao
>            Assignee: Yingjie Cao
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.15.0
>
>
> In the current sort-shuffle implementation, the maximum number of buffers can be used per result partition for shuffle data read is 32M. However, for large parallelism jobs, 32M is not enough and for small parallelism jobs, 32M may waste buffers. This ticket aims to adjust the maximum number of buffers can be used per result partition to let it adjust according to parallelism and the selected value is an empirical one based on the TPC-DS test results.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)