[jira] [Commented] (FLINK-20496) RocksDB partitioned index filter option

["fanrui (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262125#comment-17262125 ] 

fanrui commented on FLINK-20496:
--------------------------------

This feature looks very good. Our production environment encountered the problem that Index was frequently evicted and the CPU was full. The main reason is that Index has only one Block by default, which is relatively large. Frequent reading needs to be decompressed, which consumes a lot of CPU. After using this function, it is completely solved.

> RocksDB partitioned index filter option
> ---------------------------------------
>
>                 Key: FLINK-20496
>                 URL: https://issues.apache.org/jira/browse/FLINK-20496
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>    Affects Versions: 1.10.2, 1.12.0, 1.11.2
>            Reporter: YufeiLiu
>            Assignee: YufeiLiu
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.13.0
>
>
>   When using RocksDBStateBackend and enabling {{state.backend.rocksdb.memory.managed}} and {{state.backend.rocksdb.memory.fixed-per-slot}}, flink will strictly limited rocksdb memory usage which contains "write buffer" and "block cache". With these options rocksdb stores index and filters in block cache, because in default options index/filters can grows unlimited.
>   But it's lead another issue, if high-priority cache(configure by {{state.backend.rocksdb.memory.high-prio-pool-ratio}}) can't fit all index/filters blocks, it will load all metadata from disk when cache missed, and program went extremely slow. According to [Partitioned Index Filters|https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters][1], we can enable two-level index having acceptable performance when index/filters cache missed. 
>   Enable these options can get over 10x faster in my case[2], I think we can add an option {{state.backend.rocksdb.partitioned-index-filters}} and default value is false, so we can use this feature easily.
> [1] Partitioned Index Filters: https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters
> [2] Deduplicate scenario, state.backend.rocksdb.memory.fixed-per-slot=256M, SSD, elapsed time 4.91ms -> 0.33ms.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)