[GitHub] [flink] StephanEwen opened a new pull request #10517: [FLINK-14926][state-backend-rocksdb] Ensure that RocksObjects are always closed on backend disposal

[GitBox <gi...@apache.org>]

StephanEwen opened a new pull request #10517: [FLINK-14926][state-backend-rocksdb] Ensure that RocksObjects are always closed on backend disposal
URL: https://github.com/apache/flink/pull/10517
 
 
   ## This PR is based PR #10300
   
   ## What is the purpose of the change
   
   This PR ensures that the RocksDB objects are always deallocated when the keyed state backend is closed. These objects consist of
   
     - The RocksDB options objects, with the native handles.
     - Objects with native handles to options that users configure (like bloom filters)
     - Shared RocksDB data structures, Caches
   
   
   ## Brief change log
   
     - The PR introduces one component to hold an collect all these objects: 
     - This `RocksDBResourceContainer`is given to the keyed state backend object upon creation, and disposed when the state backend is disposed.
     - The factories for options are changed to that they can put created objects with native handles into a collection (owned by the resource container) so they will get closed in disposal.
   
   ## Verifying this change
   
     - The effect is better release of native memory, which can be verified by monitoring a TaskManager process with RocksDB state backend that undergoes several restarts.
     - The individual components are unit tested.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): **no**
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
     - The serializers: **no**
     - The runtime per-record code paths (performance sensitive): **no**
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
     - The S3 file system connector: **no**
   
   ## Documentation
   
     - Does this pull request introduce a new feature? **no**
     - If yes, how is the feature documented? **not applicable**
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services