You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Omar Atia <om...@its.ws> on 2011/12/11 09:01:09 UTC
Garbage collection Execuation ...
Dears ,
I have 3 camel routes deployed as bundle on serviceMix ESB , I have increased the heap memory to 2 GB in order to perform stress and performance testing .
The top command is as below :
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
4780 ocs 168 47 4 2033M 1883M sleep 29:44 3.49% java
1377 ocs 18 47 4 109M 54M sleep 0:01 0.08% java
1916 ocs 18 47 4 109M 54M sleep 0:02 0.05% java
1829 ocs 18 47 4 113M 54M sleep 0:01 0.04% java
1670 ocs 18 47 4 113M 54M sleep 0:01 0.03% java
1374 ocs 18 47 4 113M 54M sleep 0:01 0.02% java
2301 root 57 59 0 155M 94M sleep 172:13 0.02% java
1387 ocs 18 47 4 113M 54M sleep 0:01 0.02% java
1924 ocs 1 59 0 3712K 2296K cpu/11 0:01 0.02% top
1524 ocs 18 47 4 109M 50M sleep 0:01 0.02% java
68 root 3 59 0 4015M 2348M sleep 214:07 0.02% vxconfigd
395 tabsmed 1 59 0 6277M 6258M sleep 82:20 0.02% oracle
402 tabsmed 1 59 0 6277M 6258M sleep 82:17 0.02% oracle
4437 oracle 1 59 0 6277M 6214M sleep 53:48 0.01% oracle
3487 noaccess 18 59 0 158M 73M sleep 42:44 0.01% java
Can you tell me how can I make garbage collection ? the camel context is attached , I didn't use any java code all in XML configuration .
How can I minimize the memory usage to continue making stress testing ? should I use garbage collection from jconsole ? can you comment in the above command.
Seems with Huge load u need independent Hardware , I 'm having SUN OS above .
Camel context is attached .
The camel routes is explained :
>From (TCP port) ---> Java Object--> Call WS --> return XML to TCP point as string .
thanks
RE: Garbage collection Execuation ...
Posted by Omar Atia <om...@its.ws>.
Attached Graph for Old Gen behavior .
[cid:image001.png@01CCB9D3.EFAC3460]
-----Original Message-----
From: deckerego [mailto:deckerego@gmail.com]
Sent: Monday, December 12, 2011 6:10 PM
To: users@camel.apache.org
Subject: Re: Garbage collection Execuation ...
If you're seeing objects repeatedly get tenured into old gen, that often is a
symptom of object references never being freed up. In that case it's not a
garbage collection issue but more of a memory leak in your code. Collections
that grow indefinitely or repeatedly creating prototype beans can often be
the cause.
If a leak isn't immediately obvious in code, probably the next best step is
profiling the application and watching where the memory growth is happening.
If you see large byte[] arrays or the like then you may be holding far too
many documents in memory or caching strings at some point.
atiato wrote
>
> The issue in the old gen heap , it keeps growing while running the stress
> testing it reaches 1.5 GB and keeps the used heap that size even after
> finishing the testing , it keeps the same size I tried GC from JMX nothing
> changed .
>
> Can you advise if I should pass any parameter to JVM for GC ?
>
> How can I manage to free the old gen heap to back to original state before
> doing the stress testing after directly finishing the testing I need to
> restore the old heap .
>
--
View this message in context: http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5068773.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Garbage collection Execuation ...
Posted by Omar Atia <om...@its.ws>.
Finally , after digging a lot with GCs logging etc monitoring etc changing Xmx and Xms ....it appears that mock uri added in the code was the issue to make old gen FULL .
Mock endpoints keep received Exchanges in memory indefinitely
Remember that Mock is designed for testing. When you add Mock endpoints to a route, each Exchange<http://camel.apache.org/exchange.html> sent to the endpoint will be stored (to allow for later validation) in memory until explicitly reset or the JVM is restarted. If you are sending high volume and/or large messages, this may cause excessive memory use. If your goal is to test deployable routes inline, consider using NotifyBuilder<http://camel.apache.org/notifybuilder.html> or AdviceWith<http://camel.apache.org/advicewith.html> in your tests instead of adding Mock endpoints to routes directly.
From: Omar Atia
Sent: Tuesday, December 13, 2011 8:15 PM
To: users@camel.apache.org
Subject: RE: Garbage collection Execuation ...
Attached Graph for Old Gen behavior .
[cid:image001.png@01CCBB24.C273FDA0]
-----Original Message-----
From: deckerego [mailto:deckerego@gmail.com]
Sent: Monday, December 12, 2011 6:10 PM
To: users@camel.apache.org
Subject: Re: Garbage collection Execuation ...
If you're seeing objects repeatedly get tenured into old gen, that often is a
symptom of object references never being freed up. In that case it's not a
garbage collection issue but more of a memory leak in your code. Collections
that grow indefinitely or repeatedly creating prototype beans can often be
the cause.
If a leak isn't immediately obvious in code, probably the next best step is
profiling the application and watching where the memory growth is happening.
If you see large byte[] arrays or the like then you may be holding far too
many documents in memory or caching strings at some point.
atiato wrote
>
> The issue in the old gen heap , it keeps growing while running the stress
> testing it reaches 1.5 GB and keeps the used heap that size even after
> finishing the testing , it keeps the same size I tried GC from JMX nothing
> changed .
>
> Can you advise if I should pass any parameter to JVM for GC ?
>
> How can I manage to free the old gen heap to back to original state before
> doing the stress testing after directly finishing the testing I need to
> restore the old heap .
>
--
View this message in context: http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5068773.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Garbage collection Execuation ...
Posted by Omar Atia <om...@its.ws>.
Dear Deckerego,
Thanks for your help , the memory is being garbage collected when the it reaches its heap peak , I found that it is doing Full Garbage collection .
By the way old gen is being allocated and as per DBA he inform me that no issue if the old gen increased but it should reach a limit which is heap peak (maximum memory and it should garbage collected ).
Thanks,
Omar Atia
-----Original Message-----
From: deckerego [mailto:deckerego@gmail.com]
Sent: Monday, December 12, 2011 6:10 PM
To: users@camel.apache.org
Subject: Re: Garbage collection Execuation ...
If you're seeing objects repeatedly get tenured into old gen, that often is a
symptom of object references never being freed up. In that case it's not a
garbage collection issue but more of a memory leak in your code. Collections
that grow indefinitely or repeatedly creating prototype beans can often be
the cause.
If a leak isn't immediately obvious in code, probably the next best step is
profiling the application and watching where the memory growth is happening.
If you see large byte[] arrays or the like then you may be holding far too
many documents in memory or caching strings at some point.
atiato wrote
>
> The issue in the old gen heap , it keeps growing while running the stress
> testing it reaches 1.5 GB and keeps the used heap that size even after
> finishing the testing , it keeps the same size I tried GC from JMX nothing
> changed .
>
> Can you advise if I should pass any parameter to JVM for GC ?
>
> How can I manage to free the old gen heap to back to original state before
> doing the stress testing after directly finishing the testing I need to
> restore the old heap .
>
--
View this message in context: http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5068773.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Garbage collection Execuation ...
Posted by Omar Atia <om...@its.ws>.
Dear Deckerego,
I didn't write any code it is all XML configuration as spring and I have attached xml came context.
Thanks,
Omar Atia
-----Original Message-----
From: deckerego [mailto:deckerego@gmail.com]
Sent: Monday, December 12, 2011 6:10 PM
To: users@camel.apache.org
Subject: Re: Garbage collection Execuation ...
If you're seeing objects repeatedly get tenured into old gen, that often is a
symptom of object references never being freed up. In that case it's not a
garbage collection issue but more of a memory leak in your code. Collections
that grow indefinitely or repeatedly creating prototype beans can often be
the cause.
If a leak isn't immediately obvious in code, probably the next best step is
profiling the application and watching where the memory growth is happening.
If you see large byte[] arrays or the like then you may be holding far too
many documents in memory or caching strings at some point.
atiato wrote
>
> The issue in the old gen heap , it keeps growing while running the stress
> testing it reaches 1.5 GB and keeps the used heap that size even after
> finishing the testing , it keeps the same size I tried GC from JMX nothing
> changed .
>
> Can you advise if I should pass any parameter to JVM for GC ?
>
> How can I manage to free the old gen heap to back to original state before
> doing the stress testing after directly finishing the testing I need to
> restore the old heap .
>
--
View this message in context: http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5068773.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Garbage collection Execuation ...
Posted by deckerego <de...@gmail.com>.
If you're seeing objects repeatedly get tenured into old gen, that often is a
symptom of object references never being freed up. In that case it's not a
garbage collection issue but more of a memory leak in your code. Collections
that grow indefinitely or repeatedly creating prototype beans can often be
the cause.
If a leak isn't immediately obvious in code, probably the next best step is
profiling the application and watching where the memory growth is happening.
If you see large byte[] arrays or the like then you may be holding far too
many documents in memory or caching strings at some point.
atiato wrote
>
> The issue in the old gen heap , it keeps growing while running the stress
> testing it reaches 1.5 GB and keeps the used heap that size even after
> finishing the testing , it keeps the same size I tried GC from JMX nothing
> changed .
>
> Can you advise if I should pass any parameter to JVM for GC ?
>
> How can I manage to free the old gen heap to back to original state before
> doing the stress testing after directly finishing the testing I need to
> restore the old heap .
>
--
View this message in context: http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5068773.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Garbage collection Execuation ...
Posted by Omar Atia <om...@its.ws>.
Thanks Raul ,
After doing what you suggest from dumping heap memory and reading the object and classes which has been loaded I figured out that all traffic messages are loaded into Old Gen memory , which was because of mock uri added inside the route .
I really appreciate your help and pointing me to correct path to fix the issue .
Thanks,
Omar Atia
From: Raul Kripalani [mailto:raul@fusesource.com]
Sent: Monday, December 12, 2011 12:34 PM
To: users@camel.apache.org; Omar Atia
Subject: Re: Garbage collection Execuation ...
Hi Omar,
The Sun JDK provides a number of tools which can help you pinpoint the issue.
* I would start monitoring the Java Heap Sizes and Garbage Collection execution as your stress test runs. You can do this with the "jstat -gcutil" command.
* If you observe continued heap growth, and no garbage collection happening, then you will want to fine-tune heap sizes and garbage collection policies [1].
* Conversely, if you do see garbage collection occurring but heap size not diminishing, then your objects are being retained in memory. You can use the "jmap" command to obtain stats with the -histo parameters, or save a dump to later analyse it with "jmap".
* If you want to dig deeper (and you will probably want to), VisualVM is a great graphical tool to do so. You can see what's taking over your heap and even inspect object by object. Perhaps the second half of this article can help you [2].
Hope this helps. Please get back to us with your findings!
[1] http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html.
[2] http://marxsoftware.blogspot.com/2009/06/heap-dump-and-analysis-with-visualvm.html
Regards,
Raúl.
On 12 December 2011 04:10, Omar Atia <om...@its.ws>> wrote:
Dear Deckergo,
The issue in the old gen heap , it keeps growing while running the stress testing it reaches 1.5 GB and keeps the used heap that size even after finishing the testing , it keeps the same size I tried GC from JMX nothing changed .
Can you advise if I should pass any parameter to JVM for GC ?
How can I manage to free the old gen heap to back to original state before doing the stress testing after directly finishing the testing I need to restore the old heap .
Thanks,
Omar atia
Sent via BlackBerry® from mtc touch
-----Original Message-----
From: deckerego <de...@gmail.com>>
Date: Sun, 11 Dec 2011 18:00:32
To: <us...@camel.apache.org>>
Reply-To: <us...@camel.apache.org>>
Subject: Re: Garbage collection Execuation ...
You can't force garbage collection to happen. You can /request/ that the JVM
invoke it's configured garbage collector when it's next possible to do so,
but a System garbage collection call is just a prompt and not a forced
invocation.
Since JMX is enabled on your map, your best bet would be to use the garbage
collection profiling plugin of VisualVM. Watch which portions of the heap
grow - PermGen, Eden, Tenured, etc. That will tell you if you have leaks or
if you are constantly generating new objects and destroying them. It sounds
like you're rapidly marshalling objects, so I'm guessing it's the latter
situation.
If you have a large number of messages being concurrently marshalled than 2
GB isn't out of the realm of possibility. You may want to find the average
size of each message being marshalled - both its XML representation as well
as the size of the TCP packet. It's not uncommon for an XML message to be
10x the size of its bytecode representation.
It doesn't sound like you have a Camel issue per se, but more of an
application profiling issue.
atiato wrote
>
> Can you tell me how can I make garbage collection ? the camel context is
> attached , I didn't use any java code all in XML configuration .
>
> How can I minimize the memory usage to continue making stress testing ?
> should I use garbage collection from jconsole ? can you comment in the
> above command.
>
> Seems with Huge load u need independent Hardware , I 'm having SUN OS
> above .
>
> Camel context is attached .
>
> The camel routes is explained :
>
> From (TCP port) ---> Java Object--> Call WS --> return XML to TCP point as
> string .
>
--
View this message in context: http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5066978.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Garbage collection Execuation ...
Posted by Raul Kripalani <ra...@fusesource.com>.
Hi Omar,
The Sun JDK provides a number of tools which can help you pinpoint the
issue.
- I would start monitoring the Java Heap Sizes and Garbage Collection
execution as your stress test runs. You can do this with the "jstat
-gcutil" command.
- If you observe continued heap growth, and no garbage collection
happening, then you will want to fine-tune heap sizes and garbage
collection policies [1].
- Conversely, if you do see garbage collection occurring but heap size
not diminishing, then your objects are being retained in memory. You can
use the "jmap" command to obtain stats with the -histo parameters, or save
a dump to later analyse it with "jmap".
- If you want to dig deeper (and you will probably want to), VisualVM is
a great graphical tool to do so. You can see what's taking over your heap
and even inspect object by object. Perhaps the second half of this article
can help you [2].
Hope this helps. Please get back to us with your findings!
[1] http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html.
[2]
http://marxsoftware.blogspot.com/2009/06/heap-dump-and-analysis-with-visualvm.html
Regards,
Raúl.
On 12 December 2011 04:10, Omar Atia <om...@its.ws> wrote:
> Dear Deckergo,
>
> The issue in the old gen heap , it keeps growing while running the stress
> testing it reaches 1.5 GB and keeps the used heap that size even after
> finishing the testing , it keeps the same size I tried GC from JMX nothing
> changed .
>
> Can you advise if I should pass any parameter to JVM for GC ?
>
> How can I manage to free the old gen heap to back to original state before
> doing the stress testing after directly finishing the testing I need to
> restore the old heap .
>
> Thanks,
> Omar atia
> Sent via BlackBerry® from mtc touch
>
> -----Original Message-----
> From: deckerego <de...@gmail.com>
> Date: Sun, 11 Dec 2011 18:00:32
> To: <us...@camel.apache.org>
> Reply-To: <us...@camel.apache.org>
> Subject: Re: Garbage collection Execuation ...
>
> You can't force garbage collection to happen. You can /request/ that the
> JVM
> invoke it's configured garbage collector when it's next possible to do so,
> but a System garbage collection call is just a prompt and not a forced
> invocation.
>
> Since JMX is enabled on your map, your best bet would be to use the garbage
> collection profiling plugin of VisualVM. Watch which portions of the heap
> grow - PermGen, Eden, Tenured, etc. That will tell you if you have leaks or
> if you are constantly generating new objects and destroying them. It sounds
> like you're rapidly marshalling objects, so I'm guessing it's the latter
> situation.
>
> If you have a large number of messages being concurrently marshalled than 2
> GB isn't out of the realm of possibility. You may want to find the average
> size of each message being marshalled - both its XML representation as well
> as the size of the TCP packet. It's not uncommon for an XML message to be
> 10x the size of its bytecode representation.
>
> It doesn't sound like you have a Camel issue per se, but more of an
> application profiling issue.
>
>
> atiato wrote
> >
> > Can you tell me how can I make garbage collection ? the camel context is
> > attached , I didn't use any java code all in XML configuration .
> >
> > How can I minimize the memory usage to continue making stress testing ?
> > should I use garbage collection from jconsole ? can you comment in the
> > above command.
> >
> > Seems with Huge load u need independent Hardware , I 'm having SUN OS
> > above .
> >
> > Camel context is attached .
> >
> > The camel routes is explained :
> >
> > From (TCP port) ---> Java Object--> Call WS --> return XML to TCP point
> as
> > string .
> >
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5066978.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Garbage collection Execuation ...
Posted by Omar Atia <om...@its.ws>.
Dear Deckergo,
The issue in the old gen heap , it keeps growing while running the stress testing it reaches 1.5 GB and keeps the used heap that size even after finishing the testing , it keeps the same size I tried GC from JMX nothing changed .
Can you advise if I should pass any parameter to JVM for GC ?
How can I manage to free the old gen heap to back to original state before doing the stress testing after directly finishing the testing I need to restore the old heap .
Thanks,
Omar atia
Sent via BlackBerry® from mtc touch
-----Original Message-----
From: deckerego <de...@gmail.com>
Date: Sun, 11 Dec 2011 18:00:32
To: <us...@camel.apache.org>
Reply-To: <us...@camel.apache.org>
Subject: Re: Garbage collection Execuation ...
You can't force garbage collection to happen. You can /request/ that the JVM
invoke it's configured garbage collector when it's next possible to do so,
but a System garbage collection call is just a prompt and not a forced
invocation.
Since JMX is enabled on your map, your best bet would be to use the garbage
collection profiling plugin of VisualVM. Watch which portions of the heap
grow - PermGen, Eden, Tenured, etc. That will tell you if you have leaks or
if you are constantly generating new objects and destroying them. It sounds
like you're rapidly marshalling objects, so I'm guessing it's the latter
situation.
If you have a large number of messages being concurrently marshalled than 2
GB isn't out of the realm of possibility. You may want to find the average
size of each message being marshalled - both its XML representation as well
as the size of the TCP packet. It's not uncommon for an XML message to be
10x the size of its bytecode representation.
It doesn't sound like you have a Camel issue per se, but more of an
application profiling issue.
atiato wrote
>
> Can you tell me how can I make garbage collection ? the camel context is
> attached , I didn't use any java code all in XML configuration .
>
> How can I minimize the memory usage to continue making stress testing ?
> should I use garbage collection from jconsole ? can you comment in the
> above command.
>
> Seems with Huge load u need independent Hardware , I 'm having SUN OS
> above .
>
> Camel context is attached .
>
> The camel routes is explained :
>
> From (TCP port) ---> Java Object--> Call WS --> return XML to TCP point as
> string .
>
--
View this message in context: http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5066978.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Garbage collection Execuation ...
Posted by deckerego <de...@gmail.com>.
You can't force garbage collection to happen. You can /request/ that the JVM
invoke it's configured garbage collector when it's next possible to do so,
but a System garbage collection call is just a prompt and not a forced
invocation.
Since JMX is enabled on your map, your best bet would be to use the garbage
collection profiling plugin of VisualVM. Watch which portions of the heap
grow - PermGen, Eden, Tenured, etc. That will tell you if you have leaks or
if you are constantly generating new objects and destroying them. It sounds
like you're rapidly marshalling objects, so I'm guessing it's the latter
situation.
If you have a large number of messages being concurrently marshalled than 2
GB isn't out of the realm of possibility. You may want to find the average
size of each message being marshalled - both its XML representation as well
as the size of the TCP packet. It's not uncommon for an XML message to be
10x the size of its bytecode representation.
It doesn't sound like you have a Camel issue per se, but more of an
application profiling issue.
atiato wrote
>
> Can you tell me how can I make garbage collection ? the camel context is
> attached , I didn't use any java code all in XML configuration .
>
> How can I minimize the memory usage to continue making stress testing ?
> should I use garbage collection from jconsole ? can you comment in the
> above command.
>
> Seems with Huge load u need independent Hardware , I 'm having SUN OS
> above .
>
> Camel context is attached .
>
> The camel routes is explained :
>
> From (TCP port) ---> Java Object--> Call WS --> return XML to TCP point as
> string .
>
--
View this message in context: http://camel.465427.n5.nabble.com/Garbage-collection-Execuation-tp5065511p5066978.html
Sent from the Camel - Users mailing list archive at Nabble.com.