[GitHub] [flink] Myasuka opened a new pull request #10416: [FLINK-14484] Enable to control memory usage of RocksDB via Cache and WriteBufferManager

[GitBox <gi...@apache.org>]

Myasuka opened a new pull request #10416: [FLINK-14484] Enable to control memory usage of RocksDB via Cache and WriteBufferManager
URL: https://github.com/apache/flink/pull/10416
 
 
   ## What is the purpose of the change
   
   Enable to control memory usage of RocksDB via Cache and WriteBufferManager.
   We would lazy initialize a `LRUCache` and `WriteBufferManager` (which cost memory from the cache) when first keyed state backend created, and share these objects among other RocksDB instances per slot. By doing this, we could limit the memory usage of block cache and write buffers.
   We would also set flag `setCacheIndexAndFilterBlocks` as true to limit the memory usage of index&filter. Moreover, we would also set flags `setCacheIndexAndFilterBlocksWithHighPriority` and `setPinL0FilterAndIndexBlocksInCache` as true to minimize the performance impact when we try to limit the total memory (please refer to [cache-index-filter-doc](https://github.com/facebook/rocksdb/wiki/Block-Cache#caching-index-filter-and-compression-dictionary-blocks) for more details)
   
   
   ## Brief change log
   
     - Introduce a deque of `stateBackendSharedObjects` within `MemoryManager`, which contains objects shared among RocksDB instances per slot.
     - Introduce `state.backend.rocksdb.per-slot.total.memory`, `state.backend.rocksdb.write-buffer.ratio` and `state.backend.rocksdb.high-pri-pool.ratio` to `RocksDBOptions`.
     - If total memory is limited, RocksDB would share cache and writebuffer manager to limit the total memory usage.
   
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
     - Added a new `RocksDBStateBackendBoundedMemoryTest` to verify memory bounded for rocksDB state backend.
     - Extended `RocksDBStateBackendConfigTest` to verify memory control related parameters.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): **yes**, memory control would impact the performance of RocksDB
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? yes
     - If yes, how is the feature documented? docs in FLINK-14495
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services