[jira] [Created] (FLINK-15187) Reuse LocalBufferPool for FileBufferReader in blocking partition

["zhijiang (Jira)" <ji...@apache.org>]

zhijiang created FLINK-15187:
--------------------------------

             Summary: Reuse LocalBufferPool for FileBufferReader in blocking partition
                 Key: FLINK-15187
                 URL: https://issues.apache.org/jira/browse/FLINK-15187
             Project: Flink
          Issue Type: Task
          Components: Runtime / Network
            Reporter: zhijiang


If we take the file type via `taskmanager.network.bounded-blocking-subpartition-type` for batch job, while creating the respective view for reading the subpartition persistent data, it would create two unpolled memory segments for every subpartition. This portion of temporary memory is not managed and calculated by framework, so it might cause OOM error concern.

We can also reuse the ResultPartition's `LocalBufferPool` to read subpartition data to avoid this memory overhead. But there are additional two problems for reuse directly. 
 * The current core size of `LocalBufferPool` is `numberOfSubpartitions + 1`, but every subpartition needs two segments for pre-reading atm. We can remove the pre-reading to make the current core pool size suitable for the reading requirements, because the pre-reading function seems has no obvious benefits in practice which is only effecting for the last data.
 * When task finishes, it would destroy the `LocalBufferPool` even though the respective `ResultPartition still alive, so the following subpartition view can not reuse the pool directly. We should adjust the respective logics to either delay destroy the pool or create a new pool for subpartition view.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)