You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 黄川 <bu...@gmail.com> on 2016/08/23 00:49:18 UTC
oom of kafka
Hi, I am using kafka_2.11-0.9.0.1, there are multiple of warnings occur
in kafkaServer.out:
java.lang.OutOfMemoryError: Java heap space.
- the jstat output like this:
# jstat -gc 28591
S0C S1C S0U S1U EC EU OC OU MC
MU CCSC CCSU YGC YGCT FGC FGCT GCT
0.0 0.0 0.0 0.0 18432.0 9216.0 1030144.0 1029128.8 26240.0
25918.9 3456.0 3398.8 34673 1104.062 10772 37219.665 38323.727
- kafkaServer-gc.log:
2016-08-19T08:40:59.352+0800: 84889.380: [Full GC (Allocation Failure)
1017M->1017M(1024M), 3.3727202 secs]
[Eden: 0.0B(51.0M)->0.0B(51.0M) Survivors: 0.0B->0.0B Heap:
1017.7M(1024.0M)->1017.7M(1024.0M)], [Metaspace: 26056K->26056K(1073152K)]
it seems that full gc can not recycle any memory any more.My gc settings is
as follows:
-Xmx1G
-Xms1G
-XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC
-Djava.awt.headless=true
-Xloggc:/var/log/kafka/kafkaServer-gc.log
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dkafka.logs.dir=/var/log/kafka
-Dlog4j.configuration=file:/usr/lib/kafka/bin/../config/log4j.properties
Re: oom of kafka
Posted by 黄川 <bu...@gmail.com>.
thanks for your reply, Kirsch. yes , i have analyzed using MAT, the output
for memory leak is as follows:
[image: 内嵌图片 1]
2016-08-26 14:34 GMT+08:00 Harald Kirsch <ha...@raytion.com>:
> There is hardly any way anyone can guess what happens there from just the
> numbers.
>
> What you should do is start Kafka with -XX:+HeapDumpOnOutOfMemoryError,
> possibly even reduce the Xmx to 500MB and let it bomb out. Then you take a
> look at the generated heap dump with the Eclipse Memory Analyzer (
> https://eclipse.org/mat/downloads.php). Check the dominator tree and see
> where most of the memory is used. From this figure out why it was not
> released.
>
> Harald.
>
>
> On 26.08.2016 07:45, 黄川 wrote:
>
>> is there anybody can help me?
>>
>> 2016-08-23 8:49 GMT+08:00 黄川 <bu...@gmail.com>:
>>
>> Hi, I am using kafka_2.11-0.9.0.1, there are multiple of warnings occur
>>> in kafkaServer.out:
>>> java.lang.OutOfMemoryError: Java heap space.
>>>
>>>
>>> - the jstat output like this:
>>>
>>> # jstat -gc 28591
>>> S0C S1C S0U S1U EC EU OC OU MC
>>> MU CCSC CCSU YGC YGCT FGC FGCT GCT
>>> 0.0 0.0 0.0 0.0 18432.0 9216.0 1030144.0 1029128.8
>>> 26240.0 25918.9 3456.0 3398.8 34673 1104.062 10772 37219.665 38323.727
>>>
>>>
>>> - kafkaServer-gc.log:
>>>
>>> 2016-08-19T08:40:59.352+0800: 84889.380: [Full GC (Allocation Failure)
>>> 1017M->1017M(1024M), 3.3727202 secs]
>>> [Eden: 0.0B(51.0M)->0.0B(51.0M) Survivors: 0.0B->0.0B Heap:
>>> 1017.7M(1024.0M)->1017.7M(1024.0M)], [Metaspace:
>>> 26056K->26056K(1073152K)]
>>>
>>> it seems that full gc can not recycle any memory any more.My gc settings
>>> is as follows:
>>> -Xmx1G
>>> -Xms1G
>>> -XX:+UseG1GC
>>> -XX:MaxGCPauseMillis=20
>>> -XX:InitiatingHeapOccupancyPercent=35
>>> -XX:+DisableExplicitGC
>>> -Djava.awt.headless=true
>>> -Xloggc:/var/log/kafka/kafkaServer-gc.log
>>> -verbose:gc
>>> -XX:+PrintGCDetails
>>> -XX:+PrintGCDateStamps
>>> -XX:+PrintGCTimeStamps
>>> -Dcom.sun.management.jmxremote
>>> -Dcom.sun.management.jmxremote.authenticate=false
>>> -Dcom.sun.management.jmxremote.ssl=false
>>> -Dkafka.logs.dir=/var/log/kafka
>>> -Dlog4j.configuration=file:/usr/lib/kafka/bin/../config/log4j.properties
>>>
>>>
>>
Re: oom of kafka
Posted by Harald Kirsch <ha...@raytion.com>.
There is hardly any way anyone can guess what happens there from just
the numbers.
What you should do is start Kafka with -XX:+HeapDumpOnOutOfMemoryError,
possibly even reduce the Xmx to 500MB and let it bomb out. Then you take
a look at the generated heap dump with the Eclipse Memory Analyzer
(https://eclipse.org/mat/downloads.php). Check the dominator tree and
see where most of the memory is used. From this figure out why it was
not released.
Harald.
On 26.08.2016 07:45, \u9ec4\u5ddd wrote:
> is there anybody can help me?
>
> 2016-08-23 8:49 GMT+08:00 \u9ec4\u5ddd <bu...@gmail.com>:
>
>> Hi, I am using kafka_2.11-0.9.0.1, there are multiple of warnings occur
>> in kafkaServer.out:
>> java.lang.OutOfMemoryError: Java heap space.
>>
>>
>> - the jstat output like this:
>>
>> # jstat -gc 28591
>> S0C S1C S0U S1U EC EU OC OU MC
>> MU CCSC CCSU YGC YGCT FGC FGCT GCT
>> 0.0 0.0 0.0 0.0 18432.0 9216.0 1030144.0 1029128.8
>> 26240.0 25918.9 3456.0 3398.8 34673 1104.062 10772 37219.665 38323.727
>>
>>
>> - kafkaServer-gc.log:
>>
>> 2016-08-19T08:40:59.352+0800: 84889.380: [Full GC (Allocation Failure)
>> 1017M->1017M(1024M), 3.3727202 secs]
>> [Eden: 0.0B(51.0M)->0.0B(51.0M) Survivors: 0.0B->0.0B Heap:
>> 1017.7M(1024.0M)->1017.7M(1024.0M)], [Metaspace: 26056K->26056K(1073152K)]
>>
>> it seems that full gc can not recycle any memory any more.My gc settings
>> is as follows:
>> -Xmx1G
>> -Xms1G
>> -XX:+UseG1GC
>> -XX:MaxGCPauseMillis=20
>> -XX:InitiatingHeapOccupancyPercent=35
>> -XX:+DisableExplicitGC
>> -Djava.awt.headless=true
>> -Xloggc:/var/log/kafka/kafkaServer-gc.log
>> -verbose:gc
>> -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps
>> -XX:+PrintGCTimeStamps
>> -Dcom.sun.management.jmxremote
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dkafka.logs.dir=/var/log/kafka
>> -Dlog4j.configuration=file:/usr/lib/kafka/bin/../config/log4j.properties
>>
>
Re: oom of kafka
Posted by 黄川 <bu...@gmail.com>.
is there anybody can help me?
2016-08-23 8:49 GMT+08:00 黄川 <bu...@gmail.com>:
> Hi, I am using kafka_2.11-0.9.0.1, there are multiple of warnings occur
> in kafkaServer.out:
> java.lang.OutOfMemoryError: Java heap space.
>
>
> - the jstat output like this:
>
> # jstat -gc 28591
> S0C S1C S0U S1U EC EU OC OU MC
> MU CCSC CCSU YGC YGCT FGC FGCT GCT
> 0.0 0.0 0.0 0.0 18432.0 9216.0 1030144.0 1029128.8
> 26240.0 25918.9 3456.0 3398.8 34673 1104.062 10772 37219.665 38323.727
>
>
> - kafkaServer-gc.log:
>
> 2016-08-19T08:40:59.352+0800: 84889.380: [Full GC (Allocation Failure)
> 1017M->1017M(1024M), 3.3727202 secs]
> [Eden: 0.0B(51.0M)->0.0B(51.0M) Survivors: 0.0B->0.0B Heap:
> 1017.7M(1024.0M)->1017.7M(1024.0M)], [Metaspace: 26056K->26056K(1073152K)]
>
> it seems that full gc can not recycle any memory any more.My gc settings
> is as follows:
> -Xmx1G
> -Xms1G
> -XX:+UseG1GC
> -XX:MaxGCPauseMillis=20
> -XX:InitiatingHeapOccupancyPercent=35
> -XX:+DisableExplicitGC
> -Djava.awt.headless=true
> -Xloggc:/var/log/kafka/kafkaServer-gc.log
> -verbose:gc
> -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> -XX:+PrintGCTimeStamps
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dkafka.logs.dir=/var/log/kafka
> -Dlog4j.configuration=file:/usr/lib/kafka/bin/../config/log4j.properties
>