[jira] [Updated] (KAFKA-6892) Kafka Streams memory usage grows over the time till OOM

["Dawid Kulig (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dawid Kulig updated KAFKA-6892:
-------------------------------
    Summary: Kafka Streams memory usage grows over the time till OOM  (was: Kafka Streams memory usage grows )

> Kafka Streams memory usage grows over the time till OOM
> -------------------------------------------------------
>
>                 Key: KAFKA-6892
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6892
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 1.1.0
>            Reporter: Dawid Kulig
>            Priority: Minor
>         Attachments: kafka-streams-per-pod-resources-usage.png
>
>
> Hi. I am observing indefinite memory growth of my kafka-streams application. It gets killed by the OS when reaching the memory limit (10gb). 
> It's running two unrelated pipelines (read from 4 source topics - 100 partitions each - aggregate data and write to two destination topics) 
> My environment: 
>  * Kubernetes cluster
>  * 4 app instances
>  * 10GB memory limit per pod (instance)
>  * JRE 8
> JVM:
>  * -Xms2g
>  * -Xmx4g
>  * num.stream.threads = 4
>  
> When my app is running for 24hours it reaches 10GB memory limit. Heap and GC looks good, non-heap avg memory usage is 120MB. I've read it might be related to the RocksDB that works underneath streams app, however I tried to tune it using [confluent doc|https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config] however with no luck. 
> RocksDB config #1:
> {code:java}
> tableConfig.setBlockCacheSize(16 * 1024 * 1024L);
> tableConfig.setBlockSize(16 * 1024L);
> tableConfig.setCacheIndexAndFilterBlocks(true);
> options.setTableFormatConfig(tableConfig);
> options.setMaxWriteBufferNumber(2);{code}
> RocksDB config #2 
> {code:java}
> tableConfig.setBlockCacheSize(1024 * 1024L);
> tableConfig.setBlockSize(16 * 1024L);
> tableConfig.setCacheIndexAndFilterBlocks(true);
> options.setTableFormatConfig(tableConfig);
> options.setMaxWriteBufferNumber(2);
> options.setWriteBufferSize(8 * 1024L);{code}
>  
> This behavior has only been observed with our production traffic, where per topic input message rate is 10msg/sec. I am attaching cluster resources usage from last 24h.
> Any help or advice would be much appreciated. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)