You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Prakash Ramesh Dayaramani <Pr...@infosys.com> on 2015/06/05 17:11:31 UTC
Storm out of memory error while deploying jar
Hi All,
I am trying to deploy jar on storm cluster which loads volume of data at startup and stores it into memory. However, while deploying that jar itself its throwing outofmemory error.
When I tried this option on local topology it was working fine. And it takes command line arguments of minmax settings java. (-Xmx4096m)
But when I deploy it on cluster it gives me error. I tried different options to configure memory using -Xmx4096m in storm.bat itself but in vein.
Can you please point out how this configuration works ? and how can I fix this problem?
Regards,
Prakash Dayaramani
VoIP: +91-20- 42521220
Chat<si...@infosys.com> Mail<ma...@infosys.com>
[cid:_4_0E30F19C0E30EE3C004CAAD265257D2A]<http://www.infosys.com/bte>
Re: Storm out of memory error while deploying jar
Posted by Fang Chen <fc...@gmail.com>.
better enable gc logging and jmx port, and watch how gc behaves.
Fang
On Fri, Jun 5, 2015 at 8:11 AM, Prakash Ramesh Dayaramani <
Prakash_Dayaramani@infosys.com> wrote:
> Hi All,
>
> I am trying to deploy jar on storm cluster which loads
> volume of data at startup and stores it into memory. However, while
> deploying that jar itself its throwing outofmemory error.
>
> When I tried this option on local topology it was working fine. And it
> takes command line arguments of minmax settings java. (–Xmx4096m)
>
> But when I deploy it on cluster it gives me error. I tried different
> options to configure memory using –Xmx4096m in storm.bat itself but in
> vein.
>
> Can you please point out how this configuration works ? and how can I fix
> this problem?
>
>
>
>
>
> Regards,
>
> *Prakash Dayaramani*
>
> VoIP: +91-20- 42521220
>
> Chat Mail <pr...@infosys.com>
>
> [image: cid:_4_0E30F19C0E30EE3C004CAAD265257D2A]
> <http://www.infosys.com/bte>
>
>
>
Re: Storm out of memory error while deploying jar
Posted by Javier Gonzalez <ja...@gmail.com>.
Yes, this is the way to increase memory for storm. I would add one caveat:
this is the per worker memory allocation. So one has to dimension the
memory available in the machine and plan for *number of workers*
accordingly, e.g. a server with 64GB RAM will struggle if you start on it
ten workers with -Xms10GB :)
On Fri, Jun 5, 2015 at 12:23 PM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:
> I would set the needed JVM arguments in storm.yaml file. This must be
> done on every worker node.
>
> worker.childopts: "-Xmx4096m"
> ( or maybe
> supervisor.childopts: "-Xmx496m" )
>
> -Matthias
>
> On 06/05/2015 05:11 PM, Prakash Ramesh Dayaramani wrote:
> > Hi All,
> >
> > I am trying to deploy jar on storm cluster which loads
> > volume of data at startup and stores it into memory. However, while
> > deploying that jar itself its throwing outofmemory error.
> >
> > When I tried this option on local topology it was working fine. And it
> > takes command line arguments of minmax settings java. (–Xmx4096m)
> >
> > But when I deploy it on cluster it gives me error. I tried different
> > options to configure memory using –Xmx4096m in storm.bat itself but in
> > vein.
> >
> > Can you please point out how this configuration works ? and how can I
> > fix this problem?
> >
> >
> >
> >
> >
> > Regards,
> >
> > *Prakash Dayaramani*
> >
> > VoIP: +91-20-42521220
> >
> > Chat <si...@infosys.com> Mail
> > <ma...@infosys.com>
> >
> > cid:_4_0E30F19C0E30EE3C004CAAD265257D2A <http://www.infosys.com/bte>
> >
> >
> >
>
>
--
Javier González Nicolini
RE: Storm out of memory error while deploying jar
Posted by Ra...@DellTeam.com.
Dell - Internal Use - Confidential
What logs are you seeing the out of memory error, can you post some stack traces.
I would turn the -Xmx argument on the local machine to a much lower value after enabling heapdumps on OOM. Then analyze the heapdump to see who is holding on to the memory and if that is an issue.
Eventually you may also want to turn on verbose gc logging and JMX on the worker.childopts below are some excerpts
-Xmx2048m -Xms2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/storm/ -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/var/log/storm/worker-%ID%-jvm-gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -,process=Worker_%ID%_JVM -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1%ID% -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UnlockCommercialFeatures -XX:+FlightRecorder
The above should give you
1) JMX access via JConsole or Java Mission Control (found in the bin folder of your JDK)
2) Verbose GC logs
3) Heap dumps
Also it is better to try this on a single worker first.
-----Original Message-----
From: Matthias J. Sax [mailto:mjsax@informatik.hu-berlin.de]
Sent: Friday, June 05, 2015 11:24 AM
To: user@storm.apache.org
Subject: Re: Storm out of memory error while deploying jar
I would set the needed JVM arguments in storm.yaml file. This must be done on every worker node.
worker.childopts: "-Xmx4096m"
( or maybe
supervisor.childopts: "-Xmx496m" )
-Matthias
On 06/05/2015 05:11 PM, Prakash Ramesh Dayaramani wrote:
> Hi All,
>
> I am trying to deploy jar on storm cluster which loads
> volume of data at startup and stores it into memory. However, while
> deploying that jar itself its throwing outofmemory error.
>
> When I tried this option on local topology it was working fine. And it
> takes command line arguments of minmax settings java. (-Xmx4096m)
>
> But when I deploy it on cluster it gives me error. I tried different
> options to configure memory using -Xmx4096m in storm.bat itself but in
> vein.
>
> Can you please point out how this configuration works ? and how can I
> fix this problem?
>
>
>
>
>
> Regards,
>
> *Prakash Dayaramani*
>
> VoIP: +91-20-42521220
>
> Chat Mail
>
>
> cid:_4_0E30F19C0E30EE3C004CAAD265257D2A
>
>
>
Re: Storm out of memory error while deploying jar
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
I would set the needed JVM arguments in storm.yaml file. This must be
done on every worker node.
worker.childopts: "-Xmx4096m"
( or maybe
supervisor.childopts: "-Xmx496m" )
-Matthias
On 06/05/2015 05:11 PM, Prakash Ramesh Dayaramani wrote:
> Hi All,
>
> I am trying to deploy jar on storm cluster which loads
> volume of data at startup and stores it into memory. However, while
> deploying that jar itself its throwing outofmemory error.
>
> When I tried this option on local topology it was working fine. And it
> takes command line arguments of minmax settings java. (–Xmx4096m)
>
> But when I deploy it on cluster it gives me error. I tried different
> options to configure memory using –Xmx4096m in storm.bat itself but in
> vein.
>
> Can you please point out how this configuration works ? and how can I
> fix this problem?
>
>
>
>
>
> Regards,
>
> *Prakash Dayaramani*
>
> VoIP: +91-20-42521220
>
> Chat <si...@infosys.com> Mail
> <ma...@infosys.com>
>
> cid:_4_0E30F19C0E30EE3C004CAAD265257D2A <http://www.infosys.com/bte>
>
>
>