You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by shahab <sh...@gmail.com> on 2014/03/10 20:19:32 UTC
java.lang.OutOfMemoryError: Java heap space in Nimbus
Hi,
I am facing "OutOfMemoryError: Java heap space" exception in Nimbus while
running in cluster mode. I just wonder what are the possible JVM or Storm
options that I can set to overcome this problem?
I am running a storm topology in Cluster mode where all servers (zookeeper,
nimbus, supervisor and worker) are in one machine. The toplogy that I use
is as follows:
conf.setMaxSpoutPending(2000); // maximum number of pending messages at
spout
conf.setNumWorkers(4);
conf.put(Config.STORM_ZOOKEEPER_CONNECTION_TIMEOUT, "120000");
conf.setMaxTaskParallelism(2);
but I get the following Exception in Nimbus log file:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:271)
at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:219)
at
org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1136)
at
backtype.storm.daemon.nimbus$read_storm_topology.invoke(nimbus.clj:305)
at
backtype.storm.daemon.nimbus$compute_executors.invoke(nimbus.clj:407)
at
backtype.storm.daemon.nimbus$compute_executor__GT_component.invoke(nimbus.clj:420)
at
backtype.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:315)
at
backtype.storm.daemon.nimbus$mk_assignments$iter__3416__3420$fn__3421.invoke(nimbus.clj:636)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
at clojure.lang.RT.seq(RT.java:473)
at clojure.core$seq.invoke(core.clj:133)
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
at clojure.core.protocols$fn__5875.invoke(protocols.clj:54)
at
clojure.core.protocols$fn__5828$G__5823__5841.invoke(protocols.clj:13)
at clojure.core$reduce.invoke(core.clj:6030)
at clojure.core$into.invoke(core.clj:6077)
at
backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:635)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at
backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto____3593$fn__3598$fn__3599.invoke(nimbus.clj:872)
at
backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto____3593$fn__3598.invoke(nimbus.clj:871)
at
backtype.storm.timer$schedule_recurring$this__1776.invoke(timer.clj:69)
at
backtype.storm.timer$mk_timer$fn__1759$fn__1760.invoke(timer.clj:33)
at backtype.storm.timer$mk_timer$fn__1759.invoke(timer.clj:26)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
2014-03-10 20:10:02 NIOServerCnxn [ERROR] Thread
Thread[pool-4-thread-40,5,main] died
java.lang.OutOfMemoryError: Java heap space
at
org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:271)
at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:219)
at
org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1136)
at
backtype.storm.daemon.nimbus$read_storm_topology.invoke(nimbus.clj:305)
at
backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__$reify__3605.getTopologyInfo(nimbus.clj:1066)
at
backtype.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:1481)
at
backtype.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:1469)
at
org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
at
org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-03-10 20:10:02 util [INFO] Halting process: ("Error when processing an
event")
best ,
/Shahab
Re: java.lang.OutOfMemoryError: Java heap space in Nimbus
Posted by shahab <sh...@gmail.com>.
Thanks Derek for the tips. quite useful.
best,
/Shahab
On Mon, Mar 10, 2014 at 8:49 PM, Derek Dagit <de...@yahoo-inc.com> wrote:
> Yes, set 'nimbus.childopts: "-Xmx?????"' in your storm.yaml, and restart
> nimbus. If unset, I believe the default is "-Xmx1024m", for a max of 1024
> MB heap.
>
> You can set it to "-Xmx2048m", for example, to have a max heap size of
> 2048 MB.
>
> Set this on the node that runs nimbus, not in your topology conf.
>
> --
> Derek
>
>
> On 3/10/14, 14:19, shahab wrote:
>
>> Hi,
>>
>> I am facing "OutOfMemoryError: Java heap space" exception in Nimbus while
>> running in cluster mode. I just wonder what are the possible JVM or Storm
>> options that I can set to overcome this problem?
>>
>> I am running a storm topology in Cluster mode where all servers
>> (zookeeper, nimbus, supervisor and worker) are in one machine. The toplogy
>> that I use is as follows:
>>
>>
>> conf.setMaxSpoutPending(2000); // maximum number of pending messages at
>> spout
>> conf.setNumWorkers(4);
>> conf.put(Config.STORM_ZOOKEEPER_CONNECTION_TIMEOUT, "120000");
>> conf.setMaxTaskParallelism(2);
>>
>>
>> but I get the following Exception in Nimbus log file:
>> java.lang.OutOfMemoryError: Java heap space
>> at org.apache.commons.io.output.ByteArrayOutputStream.
>> toByteArray(ByteArrayOutputStream.java:271)
>> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:219)
>> at org.apache.commons.io.FileUtils.readFileToByteArray(
>> FileUtils.java:1136)
>> at backtype.storm.daemon.nimbus$read_storm_topology.invoke(
>> nimbus.clj:305)
>> at backtype.storm.daemon.nimbus$compute_executors.invoke(
>> nimbus.clj:407)
>> at backtype.storm.daemon.nimbus$compute_executor__GT_
>> component.invoke(nimbus.clj:420)
>> at backtype.storm.daemon.nimbus$read_topology_details.invoke(
>> nimbus.clj:315)
>> at backtype.storm.daemon.nimbus$mk_assignments$iter__3416__
>> 3420$fn__3421.invoke(nimbus.clj:636)
>> at clojure.lang.LazySeq.sval(LazySeq.java:42)
>> at clojure.lang.LazySeq.seq(LazySeq.java:60)
>> at clojure.lang.RT.seq(RT.java:473)
>> at clojure.core$seq.invoke(core.clj:133)
>> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
>> at clojure.core.protocols$fn__5875.invoke(protocols.clj:54)
>> at clojure.core.protocols$fn__5828$G__5823__5841.invoke(
>> protocols.clj:13)
>> at clojure.core$reduce.invoke(core.clj:6030)
>> at clojure.core$into.invoke(core.clj:6077)
>> at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(
>> nimbus.clj:635)
>> at clojure.lang.RestFn.invoke(RestFn.java:410)
>> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__
>> __3593$fn__3598$fn__3599.invoke(nimbus.clj:872)
>> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__
>> __3593$fn__3598.invoke(nimbus.clj:871)
>> at backtype.storm.timer$schedule_recurring$this__1776.invoke(
>> timer.clj:69)
>> at backtype.storm.timer$mk_timer$fn__1759$fn__1760.invoke(
>> timer.clj:33)
>> at backtype.storm.timer$mk_timer$fn__1759.invoke(timer.clj:26)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.lang.Thread.run(Thread.java:744)
>> 2014-03-10 20:10:02 NIOServerCnxn [ERROR] Thread
>> Thread[pool-4-thread-40,5,main] died
>> java.lang.OutOfMemoryError: Java heap space
>> at org.apache.commons.io.output.ByteArrayOutputStream.
>> toByteArray(ByteArrayOutputStream.java:271)
>> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:219)
>> at org.apache.commons.io.FileUtils.readFileToByteArray(
>> FileUtils.java:1136)
>> at backtype.storm.daemon.nimbus$read_storm_topology.invoke(
>> nimbus.clj:305)
>> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__
>> $reify__3605.getTopologyInfo(nimbus.clj:1066)
>> at backtype.storm.generated.Nimbus$Processor$
>> getTopologyInfo.getResult(Nimbus.java:1481)
>> at backtype.storm.generated.Nimbus$Processor$
>> getTopologyInfo.getResult(Nimbus.java:1469)
>> at org.apache.thrift7.ProcessFunction.process(
>> ProcessFunction.java:32)
>> at org.apache.thrift7.TBaseProcessor.process(
>> TBaseProcessor.java:34)
>> at org.apache.thrift7.server.TNonblockingServer$
>> FrameBuffer.invoke(TNonblockingServer.java:632)
>> at org.apache.thrift7.server.THsHaServer$Invocation.run(
>> THsHaServer.java:201)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> 2014-03-10 20:10:02 util [INFO] Halting process: ("Error when processing
>> an event")
>>
>>
>> best ,
>> /Shahab
>>
>>
Re: java.lang.OutOfMemoryError: Java heap space in Nimbus
Posted by Derek Dagit <de...@yahoo-inc.com>.
Yes, set 'nimbus.childopts: "-Xmx?????"' in your storm.yaml, and restart nimbus. If unset, I believe the default is "-Xmx1024m", for a max of 1024 MB heap.
You can set it to "-Xmx2048m", for example, to have a max heap size of 2048 MB.
Set this on the node that runs nimbus, not in your topology conf.
--
Derek
On 3/10/14, 14:19, shahab wrote:
> Hi,
>
> I am facing "OutOfMemoryError: Java heap space" exception in Nimbus while running in cluster mode. I just wonder what are the possible JVM or Storm options that I can set to overcome this problem?
>
> I am running a storm topology in Cluster mode where all servers (zookeeper, nimbus, supervisor and worker) are in one machine. The toplogy that I use is as follows:
>
>
> conf.setMaxSpoutPending(2000); // maximum number of pending messages at spout
> conf.setNumWorkers(4);
> conf.put(Config.STORM_ZOOKEEPER_CONNECTION_TIMEOUT, "120000");
> conf.setMaxTaskParallelism(2);
>
>
> but I get the following Exception in Nimbus log file:
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:271)
> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:219)
> at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1136)
> at backtype.storm.daemon.nimbus$read_storm_topology.invoke(nimbus.clj:305)
> at backtype.storm.daemon.nimbus$compute_executors.invoke(nimbus.clj:407)
> at backtype.storm.daemon.nimbus$compute_executor__GT_component.invoke(nimbus.clj:420)
> at backtype.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:315)
> at backtype.storm.daemon.nimbus$mk_assignments$iter__3416__3420$fn__3421.invoke(nimbus.clj:636)
> at clojure.lang.LazySeq.sval(LazySeq.java:42)
> at clojure.lang.LazySeq.seq(LazySeq.java:60)
> at clojure.lang.RT.seq(RT.java:473)
> at clojure.core$seq.invoke(core.clj:133)
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
> at clojure.core.protocols$fn__5875.invoke(protocols.clj:54)
> at clojure.core.protocols$fn__5828$G__5823__5841.invoke(protocols.clj:13)
> at clojure.core$reduce.invoke(core.clj:6030)
> at clojure.core$into.invoke(core.clj:6077)
> at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:635)
> at clojure.lang.RestFn.invoke(RestFn.java:410)
> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto____3593$fn__3598$fn__3599.invoke(nimbus.clj:872)
> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto____3593$fn__3598.invoke(nimbus.clj:871)
> at backtype.storm.timer$schedule_recurring$this__1776.invoke(timer.clj:69)
> at backtype.storm.timer$mk_timer$fn__1759$fn__1760.invoke(timer.clj:33)
> at backtype.storm.timer$mk_timer$fn__1759.invoke(timer.clj:26)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.lang.Thread.run(Thread.java:744)
> 2014-03-10 20:10:02 NIOServerCnxn [ERROR] Thread Thread[pool-4-thread-40,5,main] died
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:271)
> at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:219)
> at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1136)
> at backtype.storm.daemon.nimbus$read_storm_topology.invoke(nimbus.clj:305)
> at backtype.storm.daemon.nimbus$fn__3592$exec_fn__1228__auto__$reify__3605.getTopologyInfo(nimbus.clj:1066)
> at backtype.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:1481)
> at backtype.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:1469)
> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
> at org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-03-10 20:10:02 util [INFO] Halting process: ("Error when processing an event")
>
>
> best ,
> /Shahab
>