[jira] [Assigned] (FLINK-20496) RocksDB partitioned index filter option

["Yu Li (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yu Li reassigned FLINK-20496:
-----------------------------

        Fix Version/s: 1.13.0
    Affects Version/s: 1.12.0
                       1.10.2
                       1.11.2
             Assignee: YufeiLiu

Thanks for filing the JIRA [~liuyufei] and the proposal LGTM. I've assigned the issue to you and will wait for the PR. Thanks.

btw, there're already some settings for caching index and filters in [RocksDBResourceContainer|https://github.com/apache/flink/blob/59ae84069313ede60cf7ad3a9d2fe1bc07c4e460/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBResourceContainer.java#L147-L149] when using managed memory for RocksDB backend, just in case you didn't notice.

> RocksDB partitioned index filter option
> ---------------------------------------
>
>                 Key: FLINK-20496
>                 URL: https://issues.apache.org/jira/browse/FLINK-20496
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>    Affects Versions: 1.10.2, 1.12.0, 1.11.2
>            Reporter: YufeiLiu
>            Assignee: YufeiLiu
>            Priority: Major
>             Fix For: 1.13.0
>
>
>   When using RocksDBStateBackend and enabling {{state.backend.rocksdb.memory.managed}} and {{state.backend.rocksdb.memory.fixed-per-slot}}, flink will strictly limited rocksdb memory usage which contains "write buffer" and "block cache". With these options rocksdb stores index and filters in block cache, because in default options index/filters can grows unlimited.
>   But it's lead another issue, if high-priority cache(configure by {{state.backend.rocksdb.memory.high-prio-pool-ratio}}) can't fit all index/filters blocks, it will load all metadata from disk when cache missed, and program went extremely slow. According to [Partitioned Index Filters|https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters][1], we can enable two-level index having acceptable performance when index/filters cache missed. 
>   Enable these options can get over 10x faster in my case[2], I think we can add an option {{state.backend.rocksdb.partitioned-index-filters}} and default value is false, so we can use this feature easily.
> [1] Partitioned Index Filters: https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters
> [2] Deduplicate scenario, state.backend.rocksdb.memory.fixed-per-slot=256M, SSD, elapsed time 4.91ms -> 0.33ms.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)