[jira] [Comment Edited] (FLINK-7289) Memory allocation of RocksDB can be problematic in container environments

["Yu Li (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979029#comment-16979029 ] 

Yu Li edited comment on FLINK-7289 at 11/21/19 6:58 AM:
--------------------------------------------------------

Here is a quick update on progress of the work:

Overall we need below actions to fully control the rocksdb memory usage:
 # Enabling {{setWriteBufferManager}} through JNI (frocksdb PR [#4|https://github.com/dataArtisans/frocksdb/pull/4] and [#5|https://github.com/dataArtisans/frocksdb/pull/5]).
 # Enabling {{setCacheIndexAndFilterBlocksWithHighPriority}} and {{setPinTopLevelIndexAndFilter}} through JNI (frocksdb PR [#6|https://github.com/dataArtisans/frocksdb/pull/6]).
 # Build a new frocksdb release to include all required PRs.
 # Supply an option to enable limiting overall memory usage in RocksDB backend.
 # Document how to enable the memory control and the pros. (container won't be killed again due to memory overuse) and cons. (performance impact).

Now all PRs required are ready, and we are working on releasing the new frocksdb release (we decided to stay on rocksdb 5.17.2 due to the [performance issue|https://github.com/facebook/rocksdb/issues/5774] of 5.18.3, more details please refer to FLINK-14483) and preparing the changes for RocksDB backend. Generally we are in good progress and should be able to complete the work before 1.10 feature freeze.


was (Author: carp84):
Here is a quick update on progress of the work:

Overall we need below actions to fully control the rocksdb memory usage:
 # Enabling setWriteBufferManager through JNI (frocksdb PR [#4|https://github.com/dataArtisans/frocksdb/pull/4] and [#5|https://github.com/dataArtisans/frocksdb/pull/5]).
 # Enabling setCacheIndexAndFilterBlocksWithHighPriority and setPinTopLevelIndexAndFilter through JNI (frocksdb PR [#6|https://github.com/dataArtisans/frocksdb/pull/6]).
 # Build a new frocksdb release to include all required PRs.
 # Supply an option to enable limiting overall memory usage in RocksDB backend.
 # Document how to enable the memory control and the pros. (container won't be killed again due to memory overuse) and cons. (performance impact).

Now all PRs required are ready, and we are working on releasing the new frocksdb release (we decided to stay on rocksdb 5.17.2 due to the [performance issue|https://github.com/facebook/rocksdb/issues/5774] of 5.18.3, more details please refer to FLINK-14483) and preparing the changes for RocksDB backend. Generally we are in good progress and should be able to complete the work before 1.10 feature freeze.

> Memory allocation of RocksDB can be problematic in container environments
> -------------------------------------------------------------------------
>
>                 Key: FLINK-7289
>                 URL: https://issues.apache.org/jira/browse/FLINK-7289
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>    Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.7.2, 1.8.2, 1.9.0
>            Reporter: Stefan Richter
>            Priority: Major
>             Fix For: 1.10.0
>
>         Attachments: completeRocksdbConfig.txt
>
>
> Flink's RocksDB based state backend allocates native memory. The amount of allocated memory by RocksDB is not under the control of Flink or the JVM and can (theoretically) grow without limits.
> In container environments, this can be problematic because the process can exceed the memory budget of the container, and the process will get killed. Currently, there is no other option than trusting RocksDB to be well behaved and to follow its memory configurations. However, limiting RocksDB's memory usage is not as easy as setting a single limit parameter. The memory limit is determined by an interplay of several configuration parameters, which is almost impossible to get right for users. Even worse, multiple RocksDB instances can run inside the same process and make reasoning about the configuration also dependent on the Flink job.
> Some information about the memory management in RocksDB can be found here:
> https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB
> https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
> We should try to figure out ways to help users in one or more of the following ways:
> - Some way to autotune or calculate the RocksDB configuration.
> - Conservative default values.
> - Additional documentation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)