[jira] [Commented] (FLINK-28390) Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.

["Yanfei Lei (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626363#comment-17626363 ] 

Yanfei Lei commented on FLINK-28390:
------------------------------------

[~Ming Li] As the bug has been fixed and the changes are minor, I don't think it will conflict with the [fork version of flink.|https://github.com/ververica/frocksdb] I'd like to cherry-pick this to frocksdb.

And I'm +1 to [~yunta]'s comments. Rocksdb Configuration documentation has already been complicated, introducing new TTL setting maybe increase the burden on users to use TTL.

> Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.
> -------------------------------------------------------------------
>
>                 Key: FLINK-28390
>                 URL: https://issues.apache.org/jira/browse/FLINK-28390
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>            Reporter: ming li
>            Priority: Major
>
> We know that the fifo compaction strategy may silently delete data and may lose data for the business. But in some scenarios, FIFO compaction can be a very effective way to reduce CPU usage.
>  
> Flink's Taskmanager is usually some small-scale processes, such as allocating 4 CPUs and 16G memory. When the state size is small, the CPU overhead occupied by RocksDB is not high, and as the state increases, RocksDB may frequently be in the compaction operation, which will occupy a large amount of CPU and affect the computing operation.
>  
> We usually configure a TTL for the state, so when using FIFO we can configure it to be slightly longer than the TTL, so that the upper layer is the same as before. 
>  
> Although the FIFO Compaction strategy may bring space amplification, the disk is cheaper than the CPU after all, so the overall cost is reduced.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)