[jira] [Created] (FLINK-28390) Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.

["ming li (Jira)" <ji...@apache.org>]

ming li created FLINK-28390:
-------------------------------

             Summary: Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.
                 Key: FLINK-28390
                 URL: https://issues.apache.org/jira/browse/FLINK-28390
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / State Backends
            Reporter: ming li


We know that the fifo compaction strategy may silently delete data and may lose data for the business. But in some scenarios, FIFO compaction can be a very effective way to reduce CPU usage.

 

Flink's Taskmanager is usually some small-scale processes, such as allocating 4 CPUs and 16G memory. When the state size is small, the CPU overhead occupied by RocksDB is not high, and as the state increases, RocksDB may frequently be in the compaction operation, which will occupy a large amount of CPU and affect the computing operation.

 

We usually configure a TTL for the state, so when using FIFO we can configure it to be slightly longer than the TTL, so that the upper layer is the same as before. 

 

Although the FIFO Compaction strategy may bring space amplification, the disk is cheaper than the CPU after all, so the overall cost is reduced.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)