[jira] [Commented] (FLINK-26800) write small data file using share write buffer manager

["Yun Tang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-26800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513176#comment-17513176 ] 

Yun Tang commented on FLINK-26800:
----------------------------------

[~liuyufei], RocksDBJava does not publish too many details about {{WriteBufferManager}}, and we cannot track the stats of each column family under current status. I think you could create such issue or PR to RocksDB community and we could then cherry-pick them back if necessary.

> write small data file using share write buffer manager
> ------------------------------------------------------
>
>                 Key: FLINK-26800
>                 URL: https://issues.apache.org/jira/browse/FLINK-26800
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / State Backends
>    Affects Versions: 1.12.7
>            Reporter: YufeiLiu
>            Priority: Major
>
> When setting config {{state.backend.rocksdb.memory.fixed-per-slot}}, all rocksdb instances in same slot are using shared WriteBufferManager. 
> I meet a extreme circumstances, there are 2 rocksdb using a WriteBufferManager size is 32M, if rocksdb-1 write (32*0.9)M data and won't have more data in a while, it won't trigger flush, then rocksdb-2 start writing data, it will trigger flush every single record and write many small file, spend a lot of time of backgroud compaction. 
> rocksdb-2 only flush current CF data, it won't recovery until rocksdb-1 flush data in memtable.
> I can disable memory managed option to avoid this case, but I can't limit the memory usage.
> Maybe can create a tracker monitor all rocksdb memory usage, trigger a force flush if it's necessary?
> [1] RocksDB Write Buffer Manager https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager



--
This message was sent by Atlassian Jira
(v8.20.1#820001)