How to avoid Kafka latency spikes caused by log segment flush

[Jiří Holuša <jh...@azul.com.INVALID>]

Hi,


we're experiencing a big latency spikes (two orders of magnitude) on 99th percentile in our Kafka deployment. We googled a bit and found that this is pretty well documented phenomenon: https://issues.apache.org/jira/browse/KAFKA-9693


In the ticket, suggested "solution" is disabling log flush but that's hardly an acceptable solution if you care about data consistency.


We've tried to tune around log sizes, flush intervals etc. but that's only delaying the log flush doing nothing to the magnitude of the spike. I find it hard to acknowledge that all the Kafka users in the world, the most popular message broker in the world, are OK with such latency spikes.


Question

Is there any real solution/workaround to this problem? To be clear, I'm talking about how to lower the spike down to the minimum.


BTW I apologize for cross-posting but I originally asked on StackOverflow (https://stackoverflow.com/questions/73555649/how-to-avoid-kafka-latency-spikes-caused-by-log-segment-flush) and I'm really trying to get help.


Thanks,

Jiri