[jira] [Created] (FLINK-28623) Optimize the use of off heap memory by blocking and hybrid shuffle reader

["Weijie Guo (Jira)" <ji...@apache.org>]

Weijie Guo created FLINK-28623:
----------------------------------

             Summary: Optimize the use of off heap memory by blocking and hybrid shuffle reader
                 Key: FLINK-28623
                 URL: https://issues.apache.org/jira/browse/FLINK-28623
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Network
            Reporter: Weijie Guo


Currently, each FileReader(PartitionFileReader or HsSubpartitionFileReaderImpl

) will internally allocate a headerbuffer with the size of 8B. Beside, PartitionFileReader also has a 12B indexEntryBuf. Because FileReader is of subpartition granularity, If the parallelism becomes very big, and there are many slots on each TM, the memory occupation will even reach the MB level. In fact, all FileReader of the same ResultPartition read data in a single thread, so we only need to allocate a headerbuffer to a ResultPartition to optimize this phenomenon.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)