[GitHub] StefanRRichter opened a new pull request #6438: [FLINK-9981] Tune performance of RocksDB implementation

[GitBox <gi...@apache.org>]

StefanRRichter opened a new pull request #6438: [FLINK-9981] Tune performance of RocksDB implementation
URL: https://github.com/apache/flink/pull/6438
 
 
   ## What is the purpose of the change
   
   This PR does some general performance tuning for the RocksDB based implementation of timer state, as well as some polishing and additional tests.
   
   Core performance improvements come from having a cache that is based on a min-max-heap-set instead of an avl-tree. This enables us to drop the fastutil dependency. Furthermore, we track a lower bound of the first element in the RocksDB based store to improve iterator seek performance dramatically for cache refills. This improvement let's us tune down the cache size by an order of magnitude and to drop the need for the potentially dangerous and complex bulk-pull optimization altogether.
   
   Adds an additional test that the configuration of the state implementation is respected by the RocksDB backend and uses RocksDB based timers for one case in the `EventTimeWindowCheckpointingITCase`.
   
   
   ## Verifying this change
   
   Via the new tests and existing coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
     - The serializers: (yes)
     - The runtime per-record code paths (performance sensitive): (yes)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
     - If yes, how is the feature documented? (not applicable)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services