You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by John Fang <xi...@alibaba-inc.com> on 2015/12/18 03:12:50 UTC
答复: Re: java.lang.OutOfMemoryError: Direct buffer memory
You can add -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation for nimbus and supervisor, so it can produce .gc log. Then you check your supervisor's gc log and nimbus's gc log. At the same time, you can check jstack files of nimbus and supervisor.
-----邮件原件-----
发件人: ght230 [mailto:ght230@163.com]
发送时间: 2015年12月18日 9:24
收件人: dev
抄送: user
主题: Re: Re: java.lang.OutOfMemoryError: Direct buffer memory
Hi
I have checked supervisor log and nimbus log, it seems nothing abnormal.
ght230
From: Manu Zhang
Date: 2015-12-18 08:27
To: dev
CC: user
Subject: Re: java.lang.OutOfMemoryError: Direct buffer memory It could be other issues that trigger these netty connection problems and finally OOM. Could you check your supervisor log and nimbus log as well ?
On Thu, Dec 17, 2015 at 3:46 PM, ght230 <gh...@163.com> wrote:
> Hi :
>
> I ran a simple wordcount topology on storm0.9.6, found 2 abnormal
> behaviors 1. Netty reconnect so many time.
> 2. direct memory continuously growing. and finally throw
> java.lang.OutOfMemoryError.
>
> following is the unexpected information:
>
> 2015-12-15T21:30:18.546+0800 b.s.d.executor [INFO] Activating spout
> emitSpout:(1)
> 2015-12-15T21:30:18.578+0800 b.s.d.executor [INFO] Preparing bolt
> __system:(-1)
> 2015-12-15T21:30:18.614+0800 b.s.d.executor [INFO] Prepared bolt
> __system:(-1)
> 2015-12-15T21:38:37.453+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.lang.OutOfMemoryError: Direct buffer memory
> 2015-12-15T21:38:37.457+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.464+0800 b.s.m.n.Client [ERROR] failed to send 945
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.472+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.477+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.479+0800 b.s.m.n.Client [ERROR] failed to send 938
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.481+0800 b.s.m.n.Client [ERROR] failed to send 946
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.482+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.484+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.485+0800 b.s.m.n.Client [ERROR] failed to send 941
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.490+0800 b.s.m.n.Client [ERROR] failed to send 942
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.504+0800 b.s.m.n.Client [ERROR] failed to send 941
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.505+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.507+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.510+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.516+0800 b.s.m.n.StormClientHandler [INFO]
> Connection failed Netty-Client-eda.local/192.168.137.2:6708
> java.lang.OutOfMemoryError: Direct buffer memory at
> java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_67] at
> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> ~[na:1.7.0_67]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> ~[na:1.7.0_67] at
> org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffe
> rs.java:167)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffe
> rs.java:151)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.messaging.netty.MessageBatch.buffer(MessageBatch.java:8
> 4)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.messaging.netty.MessageEncoder.encode(MessageEncoder.ja
> va:32)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.doEncode(O
> neToOneEncoder.java:66)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.handleDown
> stream(OneToOneEncoder.java:59)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(D
> efaultChannelPipeline.java:591)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(D
> efaultChannelPipeline.java:582)
> ~[storm-core-0.9.6.jar:0.9.6]
> at org.apache.storm.netty.channel.Channels.write(Channels.java:704)
> ~[storm-core-0.9.6.jar:0.9.6]
> at org.apache.storm.netty.channel.Channels.write(Channels.java:671)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.channel.AbstractChannel.write(AbstractChannel.j
> ava:248)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.messaging.netty.Client.flushMessages(Client.java:323)
> ~[storm-core-0.9.6.jar:0.9.6]
> at backtype.storm.messaging.netty.Client.send(Client.java:250)
> ~[storm-core-0.9.6.jar:0.9.6]
> at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610$fn__4
> 611.invoke(worker.clj:338)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610.invok
> e(worker.clj:336)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.disruptor$clojure_handler$reify__1445.onEvent(disruptor
> .clj:58)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueu
> e.java:132)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(Disrupto
> rQueue.java:106)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor
> .clj:80)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.
> clj:94)
> ~[storm-core-0.9.6.jar:0.9.6]
> at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
> ~[storm-core-0.9.6.jar:0.9.6]
> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] at
> java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> 2015-12-15T21:38:37.516+0800 b.s.m.n.Client [ERROR] failed to send 945
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.521+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
>
> and my configure Fragment in storm.yaml as following:
>
> #topology.skip.missing.kryo.registrations: false
> storm.messaging.netty.flush.check.interval.ms: 100
> storm.messaging.transport: "backtype.storm.messaging.netty.Context"
> storm.messaging.netty.server_worker_threads: 1
> storm.messaging.netty.client_worker_threads: 1
> #storm.messaging.netty.buffer_size: 5242880
> storm.messaging.netty.transfer.batch.size: 462144
> storm.messaging.netty.buffer_size: 5242880
> storm.messaging.netty.max_retries: 100
> storm.messaging.netty.max_wait_ms: 1000
> storm.messaging.netty.min_wait_ms: 100
> topology.backpressure.enable: false
> #storm.scheduler: "backtype.storm.scheduler.DefaultScheduler"
> worker.childopts: "-Xmx768m -XX:NewSize=128m -XX:MaxNewSize=128m
> -XX:PermSize=64m -XX:MaxPermSize=64m -XX:MaxDirectMemorySize=128m
> -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps -verbose:gc
> -Xloggc:/ffcs/gc/gc-storm-worker-%ID%.log
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/ffcs/gc/heapdump"
> #worker.childopts: "-Xmx1024m"
> supervisor.childopts: "-Xmx256m"
> nimbus.childopts: "-Xmx768m"
> ui.childopts: "-Xmx768m"
> nimbus.thrift.threads: 256
>
>
> Anyone knows how to solve these 2 problems?
>
>
>
> ght230
>
Re: 答复: java.lang.OutOfMemoryError: Direct buffer memory
Posted by ght230 <gh...@163.com>.
Hi John :
Thank you for your advises.
I will check it.
ght230
发件人： John Fang
发送时间： 2015-12-18 10:12
收件人： dev@storm.apache.org
抄送： 'user'
主题： 答复: Re: java.lang.OutOfMemoryError: Direct buffer memory
You can add -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation for nimbus and supervisor, so it can produce .gc log. Then you check your supervisor's gc log and nimbus's gc log. At the same time, you can check jstack files of nimbus and supervisor.
-----邮件原件-----
发件人: ght230 [mailto:ght230@163.com]
发送时间: 2015年12月18日 9:24
收件人: dev
抄送: user
主题: Re: Re: java.lang.OutOfMemoryError: Direct buffer memory
Hi
I have checked supervisor log and nimbus log, it seems nothing abnormal.
ght230
From: Manu Zhang
Date: 2015-12-18 08:27
To: dev
CC: user
Subject: Re: java.lang.OutOfMemoryError: Direct buffer memory It could be other issues that trigger these netty connection problems and finally OOM. Could you check your supervisor log and nimbus log as well ?
On Thu, Dec 17, 2015 at 3:46 PM, ght230 <gh...@163.com> wrote:
> Hi :
>
> I ran a simple wordcount topology on storm0.9.6, found 2 abnormal
> behaviors 1. Netty reconnect so many time.
> 2. direct memory continuously growing. and finally throw
> java.lang.OutOfMemoryError.
>
> following is the unexpected information:
>
> 2015-12-15T21:30:18.546+0800 b.s.d.executor [INFO] Activating spout
> emitSpout:(1)
> 2015-12-15T21:30:18.578+0800 b.s.d.executor [INFO] Preparing bolt
> __system:(-1)
> 2015-12-15T21:30:18.614+0800 b.s.d.executor [INFO] Prepared bolt
> __system:(-1)
> 2015-12-15T21:38:37.453+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.lang.OutOfMemoryError: Direct buffer memory
> 2015-12-15T21:38:37.457+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.464+0800 b.s.m.n.Client [ERROR] failed to send 945
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.472+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.477+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.479+0800 b.s.m.n.Client [ERROR] failed to send 938
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.481+0800 b.s.m.n.Client [ERROR] failed to send 946
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.482+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.484+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.485+0800 b.s.m.n.Client [ERROR] failed to send 941
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.490+0800 b.s.m.n.Client [ERROR] failed to send 942
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.504+0800 b.s.m.n.Client [ERROR] failed to send 941
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.505+0800 b.s.m.n.Client [ERROR] failed to send 943
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.507+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.510+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.516+0800 b.s.m.n.StormClientHandler [INFO]
> Connection failed Netty-Client-eda.local/192.168.137.2:6708
> java.lang.OutOfMemoryError: Direct buffer memory at
> java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_67] at
> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> ~[na:1.7.0_67]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> ~[na:1.7.0_67] at
> org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffe
> rs.java:167)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffe
> rs.java:151)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.messaging.netty.MessageBatch.buffer(MessageBatch.java:8
> 4)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.messaging.netty.MessageEncoder.encode(MessageEncoder.ja
> va:32)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.doEncode(O
> neToOneEncoder.java:66)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.handleDown
> stream(OneToOneEncoder.java:59)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(D
> efaultChannelPipeline.java:591)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(D
> efaultChannelPipeline.java:582)
> ~[storm-core-0.9.6.jar:0.9.6]
> at org.apache.storm.netty.channel.Channels.write(Channels.java:704)
> ~[storm-core-0.9.6.jar:0.9.6]
> at org.apache.storm.netty.channel.Channels.write(Channels.java:671)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> org.apache.storm.netty.channel.AbstractChannel.write(AbstractChannel.j
> ava:248)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.messaging.netty.Client.flushMessages(Client.java:323)
> ~[storm-core-0.9.6.jar:0.9.6]
> at backtype.storm.messaging.netty.Client.send(Client.java:250)
> ~[storm-core-0.9.6.jar:0.9.6]
> at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610$fn__4
> 611.invoke(worker.clj:338)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610.invok
> e(worker.clj:336)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.disruptor$clojure_handler$reify__1445.onEvent(disruptor
> .clj:58)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueu
> e.java:132)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(Disrupto
> rQueue.java:106)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor
> .clj:80)
> ~[storm-core-0.9.6.jar:0.9.6]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.
> clj:94)
> ~[storm-core-0.9.6.jar:0.9.6]
> at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
> ~[storm-core-0.9.6.jar:0.9.6]
> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] at
> java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> 2015-12-15T21:38:37.516+0800 b.s.m.n.Client [ERROR] failed to send 945
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
> 2015-12-15T21:38:37.521+0800 b.s.m.n.Client [ERROR] failed to send 944
> messages to Netty-Client-eda.local/192.168.137.2:6708:
> java.nio.channels.ClosedChannelException
>
> and my configure Fragment in storm.yaml as following:
>
> #topology.skip.missing.kryo.registrations: false
> storm.messaging.netty.flush.check.interval.ms: 100
> storm.messaging.transport: "backtype.storm.messaging.netty.Context"
> storm.messaging.netty.server_worker_threads: 1
> storm.messaging.netty.client_worker_threads: 1
> #storm.messaging.netty.buffer_size: 5242880
> storm.messaging.netty.transfer.batch.size: 462144
> storm.messaging.netty.buffer_size: 5242880
> storm.messaging.netty.max_retries: 100
> storm.messaging.netty.max_wait_ms: 1000
> storm.messaging.netty.min_wait_ms: 100
> topology.backpressure.enable: false
> #storm.scheduler: "backtype.storm.scheduler.DefaultScheduler"
> worker.childopts: "-Xmx768m -XX:NewSize=128m -XX:MaxNewSize=128m
> -XX:PermSize=64m -XX:MaxPermSize=64m -XX:MaxDirectMemorySize=128m
> -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps -verbose:gc
> -Xloggc:/ffcs/gc/gc-storm-worker-%ID%.log
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/ffcs/gc/heapdump"
> #worker.childopts: "-Xmx1024m"
> supervisor.childopts: "-Xmx256m"
> nimbus.childopts: "-Xmx768m"
> ui.childopts: "-Xmx768m"
> nimbus.thrift.threads: 256
>
>
> Anyone knows how to solve these 2 problems?
>
>
>
> ght230
>