[jira] [Created] (KAFKA-4778) OOM on kafka-streams instances with high numbers of unreaped Record classes

["Dave Thomas (JIRA)" <ji...@apache.org>]

Dave Thomas created KAFKA-4778:
----------------------------------

             Summary: OOM on kafka-streams instances with high numbers of unreaped Record classes
                 Key: KAFKA-4778
                 URL: https://issues.apache.org/jira/browse/KAFKA-4778
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 0.10.1.1
         Environment: AWS m3.large Ubuntu 16.04.1 LTS.  rocksDB on local SSD.  
Kafka has 3 zk, 5 brokers.  
stream processors are run with:
-XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails

Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
Stream processors written in scala 2.11.8
            Reporter: Dave Thomas


We have a stream processing app with ~8 source/sink stages operating roughly at the rate of 500k messages ingested/day (~4M across the 8 stages).

We get OOM eruptions once every ~18 hours. Note it is Linux triggering the OOM-killer, not the JVM terminating itself. 

It may be worth noting that stream processing uses ~50 mbytes while processing normally for hours on end, until the problem surfaces; then in ~20-30 sec memory grows suddenly from under 100 mbytes to >1 gig and does not shrink until the process is killed.

We are using supervisor to restart the instances.  Sometimes, the process dies immediately once stream processing resumes for the same reason, a process which could continue for minutes or hours.  This extended window has enabled us to capture a heap dump using jmap.

jhat's histogram feature reveals the following top objects in memory:

Class	Instance Count	Total Size
class [B	4070487	867857833
class [Ljava.lang.Object;	2066986	268036184
class [C	539519	92010932
class [S	1003290	80263028
class [I	508208	77821516
class java.nio.HeapByteBuffer	1506943	58770777
class org.apache.kafka.common.record.Record	1506783	36162792
class org.apache.kafka.clients.consumer.ConsumerRecord	528652	35948336
class org.apache.kafka.common.record.MemoryRecords$RecordsIterator	501742	32613230
class org.apache.kafka.common.record.LogEntry	2009373	32149968
class org.xerial.snappy.SnappyInputStream	501600	20565600
class java.io.DataInputStream	501742	20069680
class java.io.EOFException	501606	20064240
class java.util.ArrayDeque	501941	8031056
class java.lang.Long	516463	4131704

Note high on the list include org.apache.kafka.common.record.Record, 
org.apache.kafka.clients.consumer.ConsumerRecord,
org.apache.kafka.common.record.MemoryRecords$RecordsIterator,
org.apache.kafka.common.record.LogEntry

All of these contain 500k-1.5M instances.

There is nothing in stream processing logs that is distinctive (log levels are still at default).

Could it be references (weak, phantom, etc) causing these instances to not be garbage-collected?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)