[jira] [Resolved] (KAFKA-9450) Decouple inner state flushing from committing

["Guozhang Wang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Guozhang Wang resolved KAFKA-9450.
----------------------------------
    Fix Version/s: 2.7.0
       Resolution: Fixed

> Decouple inner state flushing from committing
> ---------------------------------------------
>
>                 Key: KAFKA-9450
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9450
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Sophie Blee-Goldman
>            Assignee: Guozhang Wang
>            Priority: Major
>             Fix For: 2.7.0
>
>
> When EOS is turned on, the commit interval is set quite low (100ms) and all the store layers are flushed during a commit. This is necessary for forwarding records in the cache to the changelog, but unfortunately also forces rocksdb to flush the current memtable before it's full. The result is a large number of small writes to disk, losing the benefits of batching, and a large number of very small L0 files that are likely to slow compaction.
> Since we have to delete the stores to recreate from scratch anyways during an unclean shutdown with EOS, we may as well skip flushing the innermost StateStore during a commit and only do so during a graceful shutdown, before a rebalance, etc. This is currently blocked on a refactoring of the state store layers to allow decoupling the flush of the caching layer from the actual state store.
> Note that this is especially problematic with EOS due to the necessarily-low commit interval, but still hurts even with at-least-once and a much larger commit interval. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)