[jira] [Created] (FLINK-28551) Store the number of bytes instead of the number of buffers in index entry for sort-shuffle

["Yingjie Cao (Jira)" <ji...@apache.org>]

Yingjie Cao created FLINK-28551:
-----------------------------------

             Summary: Store the number of bytes instead of the number of buffers in index entry for sort-shuffle
                 Key: FLINK-28551
                 URL: https://issues.apache.org/jira/browse/FLINK-28551
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Network
            Reporter: Yingjie Cao
             Fix For: 1.16.0


Currently, in each index entry of sort-shuffle index file, one filed is the number of buffers in the current data region. The problem is that it is hard to know the data boundary before reading the file, to solve the problem, we can store the number of bytes instead of the number of buffers in index entry. Based on this change, we can do some optimization, for example, read larger size of data than a buffer for better sequential IO like what's mentioned in FLINK-28373.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)