[jira] [Created] (FLINK-22153) Manually test the sort-merge blocking shuffle

["Yingjie Cao (Jira)" <ji...@apache.org>]

Yingjie Cao created FLINK-22153:
-----------------------------------

             Summary: Manually test the sort-merge blocking shuffle
                 Key: FLINK-22153
                 URL: https://issues.apache.org/jira/browse/FLINK-22153
             Project: Flink
          Issue Type: Task
          Components: Runtime / Network
    Affects Versions: 1.13.0
            Reporter: Yingjie Cao
             Fix For: 1.13.0


In 1.12, we introduced sort-merge blocking shuffle to Flink and in 1.13, the feature was optimized which improves the usability (fix direct memory OOM issue) and performance (introduce IO scheduling and broadcast optimization).

The sort-merge blocking shuffle can be tested following the bellow process:
 # Write a simple batch job using either sql/table or DataStream API; (Word count should be enough)
 # Enable sort-merge blocking shuffle by setting taskmanager.network.sort-shuffle.min-parallelism to 1 in the Flink configuration file;
 # Submit and run the batch job with different parallelism and data volume;
 # Tune the relevant config options (taskmanager.network.blocking-shuffle.compression.enabled, taskmanager.network.sort-shuffle.min-buffers, taskmanager.memory.framework.off-heap.batch-shuffle.size) and see the influence. 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)