You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Yingjie Cao (Jira)" <ji...@apache.org> on 2022/01/24 11:44:00 UTC

[jira] [Created] (FLINK-25781) Adjust the maximum number of buffers can be used per result partition for shuffle data read

Yingjie Cao created FLINK-25781:
-----------------------------------

             Summary: Adjust the maximum number of buffers can be used per result partition for shuffle data read
                 Key: FLINK-25781
                 URL: https://issues.apache.org/jira/browse/FLINK-25781
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Network
            Reporter: Yingjie Cao
             Fix For: 1.15.0


In the current sort-shuffle implementation, the maximum number of buffers can be used per result partition for shuffle data read is 32M. However, for large parallelism jobs, 32M is not enough and for small parallelism jobs, 32M may waste buffers. This ticket aims to adjust the maximum number of buffers can be used per result partition to let it adjust according to parallelism and the selected value is an empirical one based on the TPC-DS test results.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)