[jira] [Updated] (FLINK-20740) Use managed memory to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool)

["Yingjie Cao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yingjie Cao updated FLINK-20740:
--------------------------------
    Summary: Use managed memory to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool)  (was: Use managed memory (network memory) to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool))

> Use managed memory to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool)
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-20740
>                 URL: https://issues.apache.org/jira/browse/FLINK-20740
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Network
>    Affects Versions: 1.12.0
>            Reporter: Yingjie Cao
>            Priority: Major
>             Fix For: 1.13.0
>
>
> Currently, sort-merge blocking shuffle uses some unmanaged memory for data writing and reading, which means users must increase the size of direct memory, otherwise, one may encounter direct memory OOM error, which is really bad for usability.
> As discussed in the mailing list, a separated buffer pool will be introduced to be used by the shuffle reading phase, this buffer pool is part of Flink framework and has different life cycle with the tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)