Idempotent producer vs. JVM heap usage

[Oleksandr Shulgin <ol...@zalando.de>]

Hello,

We are running Apache Kafka v2.7.1 on a total of 54 brokers distributed
evenly across 3 racks.  All machines are identical (c6g.4xlarge Amazon AWS
EC2) and have 32 GB of RAM, of which 12 GB we dedicate to the JVM heap.

This cluster hosts some thousands of topics (each replicated to all 3
racks) and a total of ~3,600 partitions per broker.  Except for a handful
of log-compacted topics, the retention time is <= 4 days.

Around 3 weeks ago we enabled idempotent producer config for our
application.  Initially this didn't result in any obvious change in cluster
stability, but recently we've got alerted that one of the brokers crashed
with OutOfMemoryError...

We've taken a closer look and found out that the heap usage (as well as G1
old generation usage) was slowly but surely growing on every broker during
the course of ~10 days.  Prior to the changes it was in the range of 3-10
GB and went up to 8-12 GB around the time of problem detection.

As we haven't changed anything else in the middle, we decided to revert the
idempotent config change.  This didn't result in any immediate heap usage
change for the brokers, but now after a few days running with the "new" old
setup we are starting to see the opposite trend, with heap usage going back
to the expected levels.

By examining Kafka source code[1] we've learned that producer snapshots
(which are keeping known producer IDs and other information supplied by
idempotent producers) are stored in special .snapshot files next to the log
segments.  On one of the brokers we've checked, these snapshot files
amounted to 1.7 GB on disk in total.

Finally, here are the questions that we have:

1. Is such a dramatic increase in heap usage expected given the number of
partitions per broker?

2. Is there a way to calculate the extra heap requirements before enabling
the idempotent producer config?

3. Are there any best practices / community experience that we might be
ignoring here?

Thank you.


[1]:
https://github.com/apache/kafka/blob/2.7.1/core/src/main/scala/kafka/log/Log.scala#L2503


Kind regards,
--
Alex