[jira] [Comment Edited] (FLINK-18554) Memory exceeds taskmanager.memory.process.size when enabling mmap_read for RocksDB

["Truong Duc Kien (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17155242#comment-17155242 ] 

Truong Duc Kien edited comment on FLINK-18554 at 7/10/20, 7:40 AM:
-------------------------------------------------------------------

According to this issue, mmaping SST files can be a massive memory hog.

[https://github.com/facebook/rocksdb/issues/3216]


was (Author: kien_truong):
According to this issues, mmaping SST files can be a massive memory hog.

https://github.com/facebook/rocksdb/issues/3216

> Memory exceeds taskmanager.memory.process.size when enabling mmap_read for RocksDB
> ----------------------------------------------------------------------------------
>
>                 Key: FLINK-18554
>                 URL: https://issues.apache.org/jira/browse/FLINK-18554
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Configuration
>    Affects Versions: 1.11.0
>            Reporter: Truong Duc Kien
>            Priority: Major
>
> We are testing Flink automatic memory management feature on Flink 1.11. However, YARN kept killing our containers due to the processes' physical memory exceeds the limit, although we have tuned the following configurations:
> {code:java}
> taskmanager.memory.process.size
> taskmanager.memory.managed.fraction
> {code}
> We suspect that it's because we have enabled mmap_read for RocksDB, since turning this options off seems to fix the issue. Maybe Flink automatic memory management is unable to account for the addition memory required when using mmap_read ?
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)