[jira] [Updated] (IGNITE-16538) Enable RocksDB to control memory usage better with stalling write if memory usage exceeds limit

["Sergey Chugunov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sergey Chugunov updated IGNITE-16538:
-------------------------------------
    Fix Version/s: 3.0.0-alpha6
                       (was: 3.0.0-alpha5)

> Enable RocksDB to control memory usage better with stalling write if memory usage exceeds limit
> -----------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-16538
>                 URL: https://issues.apache.org/jira/browse/IGNITE-16538
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Yun Tang
>            Priority: Major
>              Labels: iep-74, ignite-3
>             Fix For: 3.0.0-alpha6
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> Currently, Ignite choose RocksDB-6.20.3 and let all RocksDB table storages share same {{WriteBufferManager}} within a data region. This could limit the memory of all storage instances in a way, however, this is still not enough considerring the case that current {{WriteBufferManager}} cannot not limit the memory even memory_usage exceeds buffer_size with multi rocksdb instances.
> We can refer to this feature descriptions:
> "When WriteBufferManager is shared across DBs and column families
> to maintain memory usage under a limit, OOMs have been observed when flush cannot
> finish but writes continuously insert to memtables.
> In order to avoid OOMs, when memory usage goes beyond buffer_limit_ and DBs tries to write,
> this change will stall incoming writers until flush is completed and memory_usage
> drops." (see [the PR summary|https://github.com/facebook/rocksdb/pull/7898])
> RocksDB introduce better memory management and public it to java side from RocksDB-6.27.3 (see [commit-dc00e4b120|https://github.com/facebook/rocksdb/commit/dc00e4b1206cba9995fe042ec976df54d0f6e76f]) Thus, I suggest to bump the RocksDB version to 6.27.3 and enable this feature by default.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)