High JVM heap memory usage for Kafka version 1.0.1 and idempotent message production?

[Tianning Zhang <ti...@yahoo.de.INVALID>]

Dear all, 

We have aproblem with high JVM heap memory usage in our Kafka broker nodes - around 9 GBused at each broker.  The problem occurredafter we updated our cluster from the version 0.10.0.0 to 1.0.1. Someapplications also started to use the ‘idempotence’ producer feature to reduce messageduplication. Before the upgrade and the ‘idempotence’ usage the brokers usedtypically 2 - 3GB heap memory. 

Our clusterhas 5 nodes and support throughput around 1 billion message per day (about 1kbytes for each message).  Each node has48GB memory. The max heap size is configured at 28GB. 

The recommendationwe have is that the used heap size should not be bigger than 5G, and the maxheap size should not be bigger than 16G. However, it happened several times that the cluster failed to recoverafter some node failures (blocked by GC) – because the brokers needed more then16G to recover from failures. 
I wouldlike ask if it is true that the new Kafka version or ‘idempotence’ feature requiremuch more heap memory? I understand that  ‘idempotence’ needs more memory – but is 9Ginstead of 3G normal? Or it can be a problem with the new Kafka version?
Thanks in advance, 
Tianning Zhang
|  |
|  
T: +49(30)509691-8301
 M: +49 172 7095686
 E: tianning.zhang@awin.com
  |
|  |
|  
Eichhornstraße 3
 10785 Berlin
 
 www.awin.com
  |