Improving I/O on KafkaStreams

[Murilo Tavares <mu...@gmail.com>]

Hi
I have a KafkaStreams application with a reasonably complex, stateful
topology.
By monitoring it, we can say for sure that it is bounded by writing I/O.
This has become way worse after we upgraded KafkaStreams from 2.4 to 2.8.
(even though we disabled warm-up replicas by setting
"acceptable.recovery.lag" to Long.MAX_VALUE)

So my question is: how can we decrease writing to disk frequency in a
KafkaStreams application? Maybe tweaking memtable in the underlying
Rocksdb? Any other suggestions?

Also, any ideas why upgrading KafkaStreams would affect its i/o
performance? Any changes that might have caused an instance to do more
writes?
Thanks
Murilo

Re: Improving I/O on KafkaStreams

[Bruno Cadonna <ca...@apache.org>]

Hi Murilo,

Have you checked out the following blog post on tuning performance of 
RocksDB state stores [1] especially the section on high disk I/O and 
write stalls [2]?

Do you manage the off-heap memory used by RocksDB as described in the 
Streams docs [3]?

I do not know what may have caused the increased I/O. We upgraded 
RocksDB from 5.18.3 to 5.18.4. And we disabled bulk loading mode for 
RocksDB. But I cannot say if one of these might be the reason.

Have you looked at the RocksDB metrics [4] to get better insight into 
the situation?

Best,
Bruno

[1] 
https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance
[2] 
https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/#write-stalls
[3] 
https://kafka.apache.org/30/documentation/streams/developer-guide/memory-mgmt.html
[4] https://kafka.apache.org/documentation/#kafka_streams_rocksdb_monitoring

On 20.12.21 18:23, Murilo Tavares wrote:
> Hi
> I have a KafkaStreams application with a reasonably complex, stateful
> topology.
> By monitoring it, we can say for sure that it is bounded by writing I/O.
> This has become way worse after we upgraded KafkaStreams from 2.4 to 2.8.
> (even though we disabled warm-up replicas by setting
> "acceptable.recovery.lag" to Long.MAX_VALUE)
> 
> So my question is: how can we decrease writing to disk frequency in a
> KafkaStreams application? Maybe tweaking memtable in the underlying
> Rocksdb? Any other suggestions?
> 
> Also, any ideas why upgrading KafkaStreams would affect its i/o
> performance? Any changes that might have caused an instance to do more
> writes?
> Thanks
> Murilo
>