You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Eric Robinson <er...@psmnv.com> on 2015/03/19 01:44:45 UTC
Java Heap Space / Thread Dump Numbers
When we see these errors in the tomcat logs...
java.lang.OutOfMemoryError: Java heap space
... we do a thread dump and examine the memory numbers, resulting in output such as the following...
PSYoungGen total 95808K, used 22323K [0x00002aaac3770000, 0x00002aaaca210000, 0x00002aaaca210000)
eden space 82432K, 10% used [0x00002aaac3770000,0x00002aaac402f7f8,0x00002aaac87f0000)
from space 13376K, 99% used [0x00002aaac87f0000,0x00002aaac94fd5a8,0x00002aaac9500000)
to space 13376K, 0% used [0x00002aaac9500000,0x00002aaac9500000,0x00002aaaca210000)
PSOldGen total 177792K, used 144800K [0x00002aaab6210000, 0x00002aaac0fb0000, 0x00002aaac3770000)
object space 177792K, 81% used [0x00002aaab6210000,0x00002aaabef78110,0x00002aaac0fb0000)
PSPermGen total 120960K, used 65149K [0x00002aaaae210000, 0x00002aaab5830000, 0x00002aaab6210000)
object space 120960K, 53% used [0x00002aaaae210000,0x00002aaab21af440,0x00002aaab5830000)
Which of those lines represents the heap space?
--Eric
RE: Java Heap Space / Thread Dump Numbers
Posted by Eric Robinson <er...@psmnv.com>.
//
One correction, option -XX:-HeapDumpOnOutOfMemoryError disables heapdumps on OOM, you should use -XX:+HeapDumpOnOutOfMemoryError - with plus sign, to enable it.
//
Thanks for the follow-up.
--Eric
Re: Java Heap Space / Thread Dump Numbers
Posted by Антон Мацюк <de...@gmail.com>.
2015-03-21 12:37 GMT+02:00 Антон Мацюк <de...@gmail.com>:
> 2015-03-21 2:06 GMT+02:00 Eric Robinson <er...@psmnv.com>:
>>> Set this options to JVM, and it will make heapdumps automatically.
>> Will these heap dumps be the same size as the current tomcat memory utilization?
> Yeap, and OOM will guarantee that there is no garbage already in memory.
> But there is one trick: for example, original file with heapdump have
> 2031 MB in it, but .tar.gz "weights" only 202 MB. So just compress
> heapdump, and it will be much smaller.
One correction, option -XX:-HeapDumpOnOutOfMemoryError disables
heapdumps on OOM, you should use -XX:+HeapDumpOnOutOfMemoryError -
with plus sign, to enable it.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Java Heap Space / Thread Dump Numbers
Posted by Антон Мацюк <de...@gmail.com>.
2015-03-21 2:06 GMT+02:00 Eric Robinson <er...@psmnv.com>:
>> I think if you have vendor-locked app in vendor-locked environment (am I right?)
> Yes indeed.
So, worth a try, at least.
>> Set this options to JVM, and it will make heapdumps automatically.
> Will these heap dumps be the same size as the current tomcat memory utilization?
Yeap, and OOM will guarantee that there is no garbage already in memory.
But there is one trick: for example, original file with heapdump have
2031 MB in it, but .tar.gz "weights" only 202 MB. So just compress
heapdump, and it will be much smaller.
>> From heapdump you can look what it was.
> That sounds extremely promising. Now I'm starting to get excited.
As I said above - it's worth a try.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by Eric Robinson <er...@psmnv.com>.
//
I think if you have vendor-locked app in vendor-locked environment (am I right?)
//
Yes indeed.
//
As I said above, there is an options for JVM:
-XX:-HeapDumpOnOutOfMemoryError - it will make heapdump on OOM.
-XX:HeapDumpPath=./java_pid<pid>.hprof - give it an reasonable path to file.
Set this options to JVM, and it will make heapdumps automatically.
//
Will these heap dumps be the same size as the current tomcat memory utilization?
//
After you got an heapdump - you can look for biggest object in it, or give it to a vendor.
What you are doind now - is a temperature measurement using something, that's not measure temperature.
You are collect threaddump, but it's a "where it was"-state of app, but not "what was in it"-state.
Imagine that something already occupies some size in memory. And some thread doing it's small job and tries to allocate a little size of memory. A small size, but there is no more free memory for allocation
- this is a reason of OOM.
From heapdump you can look what it was.
//
That sounds extremely promising. Now I'm starting to get excited.
--Eric
Re: Java Heap Space / Thread Dump Numbers
Posted by Антон Мацюк <de...@gmail.com>.
2015-03-20 22:29 GMT+02:00 Eric Robinson <er...@psmnv.com>:
>> Very good information. I much prefer finding the actual root causes of things rather than just bumping the memory, but I'm not sure how much that would help because the best I can do is report the issue to the vendor. Changing the code is off the table for me. If we can zero in on a problem, we may be able to get them to fix it.
I think if you have vendor-locked app in vendor-locked environment (am
I right?) - the bast way of fighting with OOMs will be looking into
heaps.
As I said above, there is an options for JVM:
-XX:-HeapDumpOnOutOfMemoryError - it will make heapdump on OOM.
-XX:HeapDumpPath=./java_pid<pid>.hprof - give it an reasonable path to file.
Set this options to JVM, and it will make heapdumps automatically.
After you got an heapdump - you can look for biggest object in it, or
give it to a vendor.
What you are doind now - is a temperature measurement using something,
that's not measure temperature.
You are collect threaddump, but it's a "where it was"-state of app,
but not "what was in it"-state.
Imagine that something already occupies some size in memory. And some
thread doing it's small job and tries to allocate a little size of
memory. A small size, but there is no more free memory for allocation
- this is a reason of OOM.
>From heapdump you can look what it was.
Threaddump on OOM is showing you what threads was not able to allocate
some more memory in heap, and this may not be the thread which ate all
memory.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by Eric Robinson <er...@psmnv.com>.
//
You can look for biggest objects in heap (using MAT, Leak Suspects report, Dominators Tree report).
This way you can try to find what was the exact reason of OOM instead of just thinking "eh, I need to give instances more memory".
MAT does things good. I've already found using MAT+JVVM the reasons why my instances of two different apps die with OOM (and only one of that reasons was 3rd party library, others was "not well written code").
Now I'm looking to make optimisations and shrink memory for instances a lot.
That's my sucess story :)
//
Very good information. I much prefer finding the actual root causes of things rather than just bumping the memory, but I'm not sure how much that would help because the best I can do is report the issue to the vendor. Changing the code is off the table for me. If we can zero in on a problem, we may be able to get them to fix it.
--Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Java Heap Space / Thread Dump Numbers
Posted by Антон Мацюк <de...@gmail.com>.
2015-03-20 22:09 GMT+02:00 Eric Robinson <er...@psmnv.com>:
>> I've posted above some howtos, have you looked at them?
> No, I'm not sure how useful I would find them. I think the heap summary is probably all I need, but I may be wrong. Would the heap dump provide more actionable intel as far as tuning my memory parameters?
You can look for biggest objects in heap (using MAT, Leak Suspects
report, Dominators Tree report).
This way you can try to find what was the exact reason of OOM instead
of just thinking "eh, I need to give instances more memory".
MAT does things good. I've already found using MAT+JVVM the reasons
why my instances of two different apps die with OOM (and only one of
that reasons was 3rd party library, others was "not well written
code").
Now I'm looking to make optimisations and shrink memory for instances a lot.
That's my sucess story :)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Eric Robinson [mailto:eric.robinson@psmnv.com]
> Subject: RE: Java Heap Space / Thread Dump Numbers
> Would the heap dump provide more actionable intel as far as tuning my memory parameters?
It would provide information about what types of objects are consuming the heap space. From that, you may be able to adjust your webapp's settings to avoid running into trouble (or modify the webapp, if it's abusing the heap).
If you monitor the heap usage on a regular basis (e.g., once an hour), you might be able to see if the webapp is leaking memory. Also, by comparing the heap dumps with the current request load, you might get an idea of just how large you need to make the heap to satisfy your peak periods.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by Eric Robinson <er...@psmnv.com>.
//
Yeah, heapdumps.
I've posted above some howtos, have you looked at them?
//
No, I'm not sure how useful I would find them. I think the heap summary is probably all I need, but I may be wrong. Would the heap dump provide more actionable intel as far as tuning my memory parameters?
--Eric
Re: Java Heap Space / Thread Dump Numbers
Posted by Антон Мацюк <de...@gmail.com>.
2015-03-20 1:15 GMT+02:00 Eric Robinson <er...@psmnv.com>:
> Heap dumps?
> What we do is called a thread dump, as far as I know. We use kill -3 on Linux, which dumps the thread activity. The memory data shows up at the bottom of that. See: http://producthelp.sdl.com/WorldServer/10.2/en/GUID-4F09CD10-BC4F-46A8-AE83-599A5C8C7740.html
Yeah, heapdumps.
I've posted above some howtos, have you looked at them?
//Sorry for previous top-posting, I still can't get used to
bottom-posting into mailing lists :(
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by Eric Robinson <er...@psmnv.com>.
//
Mostly leaner - a 32-bit JVM uses 32-bit pointers, so object references consume less heap and stack space. Whether or not the code runs faster or slower depends on what you're doing, since the tradeoff is fewer registers available in 32-bit mode, which can lead to more register spills and reloads. Testing of the specific webapps is required.
//
Worth trying it anyway. It's an easy test.
--Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Eric Robinson [mailto:eric.robinson@psmnv.com]
> Subject: RE: Java Heap Space / Thread Dump Numbers
> > If you have the option, you might want to run a 32-bit JVM; it will probably run leaner
> > and faster than a 64-bit JVM will.
> What do you mean my faster and leaner?
Mostly leaner - a 32-bit JVM uses 32-bit pointers, so object references consume less heap and stack space. Whether or not the code runs faster or slower depends on what you're doing, since the tradeoff is fewer registers available in 32-bit mode, which can lead to more register spills and reloads. Testing of the specific webapps is required.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by Eric Robinson <er...@psmnv.com>.
//
Recent 64-bit JVMs will automatically use "-XX:+UseCompressedOops".
I'm not sure about your version, specifically. If you have the option, you might want to run a 32-bit JVM; it will probably run leaner and faster than a 64-bit JVM will.
//
Interesting. What do you mean my faster and leaner? We're currently running 60-90 instances of tomcat on 8-core servers with 32-64GB RAM, and they all run fast. Swapping is negligible, CPU is around 20% average utilization, iowait is low. The only time we have trouble is when an instance occasionally runs out of memory and throws an OOME. When that happens, we allocate another 64MB to that instance and restart it.
//
If you are issuing a "kill -3" and then looking at stdout, I'd encourage you to take a look at "jstack" which can do the same thing (dump the thread status) but it can be redirected to some other file.
That is, it doesn't dump to the target JVM's stdout, but the jstack tool's stdout. It's much more convenient.
If you are looking for heap information, you want "jmap" instead:
$ jmap -heap [pid]
Unfortunately, each of these tools produces output in a slightly different format, so you may have to adjust some of your automated tools to suit.
//
Great info, thanks.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Java Heap Space / Thread Dump Numbers
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Eric,
On 3/19/15 7:15 PM, Eric Robinson wrote:
> Christopher Shultz wrote: // Time to upgrade. Tomcat is hideously
> out of date (probably because you are using RedHat's Tomcat
> package), at least by version number. I'm not sure what RedHat
> does (if anything) about security fixes, etc. but a vanilla 6.0.18
> is probably vulnerable and has been for a very long time.
>
> Java 1.6 is EOL and Java 1.7 is getting close.
>
> I strongly advise you to move up to Tomcat 8.0.20 and Java 1.8 and
> start developing and testing your application against those
> versions. //
>
> Would if I could. Vendor restriction. I can only run what they
> will support.
Bah. I hate "Vendors". Tell them to get with the program. What you are
getting isn't support: it's complacency. If someone hacks your
environment, they would certainly start paying attention. You should
hire another vendor to do penetration tests and let the sparks fly.
> Christopher Shultz wrote: //
>> Typical startup options looks like this...
>>
>> JAVA_OPTS="-Xms192M -Xmx384M \ -XX:MaxPermSize=128M \
>
> Those seem reasonable, except maybe not PermGen. Are you
> increasing it or reducing it from the default on your platform? //
>
> Increased from default of 64 MB, which ran fine for years until a
> recent software upgrade.
Okay.
> Christopher Shultz wrote: //
>> The memory allocation seems low to you because we run many
>> instances of tomcat on the same server. Although the servers
>> have 32-64GB of RAM, the individual tomcat/java instances are
>> kept fairly low.
>
> We ran for years in a 64MiB heap. Then we got enough traffic that
> sessions required us to expand our heaps. If you can run in a
> small heap, great. But keep your eye on what's happening with your
> users; you may find that you have outgrown your old heap settings.
> //
>
> That's what this question is about. We ran 64 MB heaps for years,
> but with the latest application software version we find that
> 192-384 is required per instance. Others throw errors unless they
> get 768. We cannot set them all the same, so we need to configure
> them very carefully and closely.
Recent 64-bit JVMs will automatically use "-XX:+UseCompressedOops".
I'm not sure about your version, specifically. If you have the option,
you might want to run a 32-bit JVM; it will probably run leaner and
faster than a 64-bit JVM will.
> Christopher Shultz wrote: //
>> We are trying to be proactive by watching the memory usage
>> numbers in the thread dumps.
>
> Heap dumps? //
>
> What we do is called a thread dump, as far as I know. We use kill
> -3 on Linux, which dumps the thread activity. The memory data
> shows up at the bottom of that. See:
> http://producthelp.sdl.com/WorldServer/10.2/en/GUID-4F09CD10-BC4F-46A8-AE83-599A5C8C7740.html
Okay.
>
>
A heap dump produces a file roughly the size of the heap space.
You are getting a heap summary.
If you are issuing a "kill -3" and then looking at stdout, I'd
encourage you to take a look at "jstack" which can do the same thing
(dump the thread status) but it can be redirected to some other file.
That is, it doesn't dump to the target JVM's stdout, but the jstack
tool's stdout. It's much more convenient.
If you are looking for heap information, you want "jmap" instead:
$ jmap -heap [pid]
Unfortunately, each of these tools produces output in a slightly
different format, so you may have to adjust some of your automated
tools to suit.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVDEVSAAoJEBzwKT+lPKRYmNMP/is3AWqp7ZaHccFFQ2olE9su
51SmM1iLsQz9d0KhpXZtC7JK0l7pzE0roxhiAZEHmhg+nmWNRY8nCzcNvT4BM3P2
WUJnakmGve2gvziHXOH/bhtWdsODYCHm/rdxMHUYf+9i2i4aKwpBG0KtVHzK3f3r
29oC17hqzE2IqRSLaWKSM3xLY4smXVBb29srKnhjL86k4pPtRisdI9beCRTRagoo
D4tcWK/vre6r+0Hq50bj2oM7ECCpNnWVv/TloKy18DsuyU1ODsTE9XWoBKuMVSry
HYiStjGB0wHuiltwF5Q7PtVH+gAMJTFYExUekMFOt4nESllcZw6bezZnbPm2Q2Qd
AJkLvA2SRgRBEzXUFkbWozlSCIDz/kKNAYYd9mzAJQDwRrvaoumACYbOdyBWApZM
xC+bIb2rgVvXKHZr0cbzEmfoXRq3bHSfI3rMCpj2lY7PCjLw9jFukuc+ogiuiPbu
HgkfyFofJKOAYp2aimbzH4RBGzddDh0gm3VD7kOUHudyWngH2UmUtwHcdOmH/Jtw
MFK99lhQ+5Mdxg1Damb4nZxmfWWmQkHOV8pvNBw/pyWFIEo73QO3L4T0ZTAZM3Ll
CpqAPiWKTWzEBGE3hL1PWB9DKbWTTeZjDFOwJ/gKW8Gcb8y8RKq41EL4yzRc8lWu
QCoidw6zb75NIrXYuXd3
=8EKS
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by Eric Robinson <er...@psmnv.com>.
Christopher Shultz wrote:
//
Time to upgrade. Tomcat is hideously out of date (probably because you are using RedHat's Tomcat package), at least by version number. I'm not sure what RedHat does (if anything) about security fixes, etc. but a vanilla 6.0.18 is probably vulnerable and has been for a very long time.
Java 1.6 is EOL and Java 1.7 is getting close.
I strongly advise you to move up to Tomcat 8.0.20 and Java 1.8 and start developing and testing your application against those versions.
//
Would if I could. Vendor restriction. I can only run what they will support.
Christopher Shultz wrote:
//
> Typical startup options looks like this...
>
> JAVA_OPTS="-Xms192M -Xmx384M \ -XX:MaxPermSize=128M \
Those seem reasonable, except maybe not PermGen. Are you increasing it or reducing it from the default on your platform?
//
Increased from default of 64 MB, which ran fine for years until a recent software upgrade.
Christopher Shultz wrote:
//
> The memory allocation seems low to you because we run many instances
> of tomcat on the same server. Although the servers have 32-64GB of
> RAM, the individual tomcat/java instances are kept fairly low.
We ran for years in a 64MiB heap. Then we got enough traffic that sessions required us to expand our heaps. If you can run in a small heap, great. But keep your eye on what's happening with your users; you may find that you have outgrown your old heap settings.
//
That's what this question is about. We ran 64 MB heaps for years, but with the latest application software version we find that 192-384 is required per instance. Others throw errors unless they get 768. We cannot set them all the same, so we need to configure them very carefully and closely.
Christopher Shultz wrote:
//
> We are trying to be proactive by watching the memory usage numbers in
> the thread dumps.
Heap dumps?
//
What we do is called a thread dump, as far as I know. We use kill -3 on Linux, which dumps the thread activity. The memory data shows up at the bottom of that. See: http://producthelp.sdl.com/WorldServer/10.2/en/GUID-4F09CD10-BC4F-46A8-AE83-599A5C8C7740.html
--Eric
Re: Java Heap Space / Thread Dump Numbers
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Eric,
On 3/19/15 11:15 AM, Eric Robinson wrote:
> Thanks for the feedback. We run various versions of RHEL (5.5 to
> 6.3) x64. Tomcat is mostly 6.0.18. Java is mostly 6u21 x 64.
Time to upgrade. Tomcat is hideously out of date (probably because you
are using RedHat's Tomcat package), at least by version number. I'm
not sure what RedHat does (if anything) about security fixes, etc. but
a vanilla 6.0.18 is probably vulnerable and has been for a very long time.
Java 1.6 is EOL and Java 1.7 is getting close.
I strongly advise you to move up to Tomcat 8.0.20 and Java 1.8 and
start developing and testing your application against those versions.
> Typical startup options looks like this...
>
> JAVA_OPTS="-Xms192M -Xmx384M \ -XX:MaxPermSize=128M \
Those seem reasonable, except maybe not PermGen. Are you increasing it
or reducing it from the default on your platform?
> The memory allocation seems low to you because we run many
> instances of tomcat on the same server. Although the servers have
> 32-64GB of RAM, the individual tomcat/java instances are kept
> fairly low.
We ran for years in a 64MiB heap. Then we got enough traffic that
sessions required us to expand our heaps. If you can run in a small
heap, great. But keep your eye on what's happening with your users;
you may find that you have outgrown your old heap settings.
> We are trying to be proactive by watching the memory usage numbers
> in the thread dumps.
Heap dumps?
> When we see a number that could be getting too close for comfort,
> we increase the memory allocation slightly and restart the tomcat.
> (We can get away with this in production because of the way our
> load balancing works it does not cause any downtime for clients.)
>
> Any thoughts on what numbers we should watch closest? The from
> space is always mostly used, but that does not appear to cause a
> problem.
http://wiki.apache.org/tomcat/FAQ/Monitoring
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVCxEOAAoJEBzwKT+lPKRYqI0P/iIGZ5NfK5yAxdSlSFoIMQcS
alKJqPBCPhEe6mECyBN+KB24Ci8YvJDtdWzMe56BgJ0uUvCKmlfRxO4VTyUhJH+X
WxkkRoZ2PylZNtwj9mvs/IFat09nBTVCeUy3e4ZSE4bEN30HHe9MgS3bz+Tyvmbt
UCONb+JrjHAFvwcSTACdgJHK62TIACLkJkZ534JshPkaTVIpRqGSgx2OimoAk4k8
Z7K7E8uQlJg65XG+uR7bXHZ+mHGdwL6yT574OGb+FreLW5fraF2LmUzlEsP62ynI
50oUXGEoOUM1LLGNDxmKXrCuKXiuxd2yqC+TbHtfdBlfXwvS3AUMldZ3Q+2ZtDUe
xF4v5y25OO4pYoGJp7vQNQGmfWTI6DJ+Ru0vnULnqsMLuBKsIfCsbnB9hV2Fx9e4
8rzIgc1YyuIACFi78Pr1Ot95mBw14C/802a2ypJl+6UCvWZfbU+/CkdhcJ44Mnf7
VyD5beBJJy2Dt0m8lcIxI6vw2Pe2msGLHLLPlu3yhewOc4ii1DIE5gJ2IrXxncLY
YaZqG7rq8TDR5y88xdBFkM3gLugZR/v30S82dnt+sTlIOMDyA750poKY7L8BFK+z
a7dSRTKpJd1GYqKo0pMsq0YV/PwVmODiCQp6XNLkvyj68Cp3N4fRF4fOZLkhPjJ/
aeVTxYJX97p946/DlHMh
=p3WY
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by Eric Robinson <er...@psmnv.com>.
> PSYoungGen total 95808K, used 22323K [0x00002aaac3770000, 0x00002aaaca210000, 0x00002aaaca210000)
> eden space 82432K, 10% used [0x00002aaac3770000,0x00002aaac402f7f8,0x00002aaac87f0000)
> from space 13376K, 99% used [0x00002aaac87f0000,0x00002aaac94fd5a8,0x00002aaac9500000)
> to space 13376K, 0% used [0x00002aaac9500000,0x00002aaac9500000,0x00002aaaca210000)
> PSOldGen total 177792K, used 144800K [0x00002aaab6210000, 0x00002aaac0fb0000, 0x00002aaac3770000)
> object space 177792K, 81% used [0x00002aaab6210000,0x00002aaabef78110,0x00002aaac0fb0000)
> PSPermGen total 120960K, used 65149K [0x00002aaaae210000, 0x00002aaab5830000, 0x00002aaab6210000)
> object space 120960K, 53% used [0x00002aaaae210000,0x00002aaab21af440,0x00002aaab5830000)
> Which of those lines represents the heap space?
Chuck Caldarale wrote:
//
Actually, all of them. The JVM's heap is not just a simple collection, but is instead a somewhat complex structure designed to facilitate garbage collection. A brief description of the components:
eden space - where most objects are born (and most of them die)
from-space - where live objects exist after a minor garbage collection
to-space - where objects will be copied to during a minor garbage collection
old gen object space - where long-lived objects reside
perm gen object space - where instances of java.lang.Class reside
Normally sized objects are initially created in eden. When eden fills, a minor GC copies most live objects from eden and the from-space into the to-space; objects that have been around a while are instead put into old gen. At the end of the minor GC, the from- (now empty) and to-spaces swap roles. When things get too full, a major GC is initiated, which squeezes out any dead space from the old gen. One exception to the above is allocation of large objects, which are placed immediately into old gen. Your OOME may be due to attempting to allocate something big.
Since you didn't provide any information about the JVM version, GC settings, platform you're running on, or what your applications do, it's hard to say whether or not your heap is sized appropriately. However, your total of around 262M is rather small in this age of 64-bit machines with multiple gigabytes of memory.
More information (not all of it up to date):
http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html
//
Thanks for the feedback. We run various versions of RHEL (5.5 to 6.3) x64. Tomcat is mostly 6.0.18. Java is mostly 6u21 x 64. Typical startup options looks like this...
JAVA_OPTS="-Xms192M -Xmx384M \
-XX:MaxPermSize=128M \
-Djvm=$JVM_ID \
-Djava.awt.headless=true \
-Djava.net.preferIPv4Stack=true \
-Duser.timezone=US/Pacific \
-Xloggc:/alley/site214/tomcat6/logs/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails \
$JPDA_OPTS $JMX_OPTS"
The memory allocation seems low to you because we run many instances of tomcat on the same server. Although the servers have 32-64GB of RAM, the individual tomcat/java instances are kept fairly low.
We are trying to be proactive by watching the memory usage numbers in the thread dumps. When we see a number that could be getting too close for comfort, we increase the memory allocation slightly and restart the tomcat. (We can get away with this in production because of the way our load balancing works it does not cause any downtime for clients.)
Any thoughts on what numbers we should watch closest? The from space is always mostly used, but that does not appear to cause a problem.
--Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Java Heap Space / Thread Dump Numbers
Posted by Антон Мацюк <de...@gmail.com>.
Eric, if you are looking for a reasons and think, what can be done
with OOMs, take a look at this page:
http://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html
There is an options:
-XX:-HeapDumpOnOutOfMemoryError - it will make heapdump on OOM.
-XX:HeapDumpPath=./java_pid<pid>.hprof - give it an reasonable path to file.
After you get an heapdump - use Eclipse Memory Analyzing Tool, there
is an effective Find Leak Suspects report, and also Dominator Tree.
https://www.dropbox.com/s/m2kgvel22vjjrzv/screenshot_000122.png?dl=0 -
this is an example.
This way I've found an memory leak in BoneCP, for example.
https://www.dropbox.com/s/3v0tb56b5uu7ub6/screenshot_000123.png?dl=0 -
look, 634M of SybConnections (Sybase ASE connection).
https://www.dropbox.com/s/64qsm9wd5ls1h5v/screenshot_000124.png?dl=0 -
Each SybConnection occupy from 14M to 45M of heap.
https://www.dropbox.com/s/p2qpqmn9r65481i/screenshot_000125.png?dl=0 -
Dominator Tree will show, what is the reason, in my case it was a
bunch of SQLWarnings in each connection.
Also you can monitor a server instance with JVisualVM. Use VisualGC
plugin to visualize, what's happening with your Tomcat's heap in
detail: https://www.dropbox.com/s/za36q46v765gltf/screenshot_000126_cut.png?dl=0
(VisualGC is a downloadable plugin to JVisualVM, JVisualVM can be
found in the bin directory of installed JDK). You may also need to
launch jstatd with the same user, which launched Tomcat to be able to
monitor Tomcat remotely.
Good luck! :)
2015-03-19 3:35 GMT+02:00 Caldarale, Charles R <Ch...@unisys.com>:
>> From: Eric Robinson [mailto:eric.robinson@psmnv.com]
>> Subject: Java Heap Space / Thread Dump Numbers
>
>> PSYoungGen total 95808K, used 22323K [0x00002aaac3770000, 0x00002aaaca210000, 0x00002aaaca210000)
>> eden space 82432K, 10% used [0x00002aaac3770000,0x00002aaac402f7f8,0x00002aaac87f0000)
>> from space 13376K, 99% used [0x00002aaac87f0000,0x00002aaac94fd5a8,0x00002aaac9500000)
>> to space 13376K, 0% used [0x00002aaac9500000,0x00002aaac9500000,0x00002aaaca210000)
>> PSOldGen total 177792K, used 144800K [0x00002aaab6210000, 0x00002aaac0fb0000, 0x00002aaac3770000)
>> object space 177792K, 81% used [0x00002aaab6210000,0x00002aaabef78110,0x00002aaac0fb0000)
>> PSPermGen total 120960K, used 65149K [0x00002aaaae210000, 0x00002aaab5830000, 0x00002aaab6210000)
>> object space 120960K, 53% used [0x00002aaaae210000,0x00002aaab21af440,0x00002aaab5830000)
>
>> Which of those lines represents the heap space?
>
> Actually, all of them. The JVM's heap is not just a simple collection, but is instead a somewhat complex structure designed to facilitate garbage collection. A brief description of the components:
> eden space - where most objects are born (and most of them die)
> from-space - where live objects exist after a minor garbage collection
> to-space - where objects will be copied to during a minor garbage collection
> old gen object space - where long-lived objects reside
> perm gen object space - where instances of java.lang.Class reside
>
> Normally sized objects are initially created in eden. When eden fills, a minor GC copies most live objects from eden and the from-space into the to-space; objects that have been around a while are instead put into old gen. At the end of the minor GC, the from- (now empty) and to-spaces swap roles. When things get too full, a major GC is initiated, which squeezes out any dead space from the old gen. One exception to the above is allocation of large objects, which are placed immediately into old gen. Your OOME may be due to attempting to allocate something big.
>
> Since you didn't provide any information about the JVM version, GC settings, platform you're running on, or what your applications do, it's hard to say whether or not your heap is sized appropriately. However, your total of around 262M is rather small in this age of 64-bit machines with multiple gigabytes of memory.
>
> More information (not all of it up to date):
> http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java Heap Space / Thread Dump Numbers
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Eric Robinson [mailto:eric.robinson@psmnv.com]
> Subject: Java Heap Space / Thread Dump Numbers
> PSYoungGen total 95808K, used 22323K [0x00002aaac3770000, 0x00002aaaca210000, 0x00002aaaca210000)
> eden space 82432K, 10% used [0x00002aaac3770000,0x00002aaac402f7f8,0x00002aaac87f0000)
> from space 13376K, 99% used [0x00002aaac87f0000,0x00002aaac94fd5a8,0x00002aaac9500000)
> to space 13376K, 0% used [0x00002aaac9500000,0x00002aaac9500000,0x00002aaaca210000)
> PSOldGen total 177792K, used 144800K [0x00002aaab6210000, 0x00002aaac0fb0000, 0x00002aaac3770000)
> object space 177792K, 81% used [0x00002aaab6210000,0x00002aaabef78110,0x00002aaac0fb0000)
> PSPermGen total 120960K, used 65149K [0x00002aaaae210000, 0x00002aaab5830000, 0x00002aaab6210000)
> object space 120960K, 53% used [0x00002aaaae210000,0x00002aaab21af440,0x00002aaab5830000)
> Which of those lines represents the heap space?
Actually, all of them. The JVM's heap is not just a simple collection, but is instead a somewhat complex structure designed to facilitate garbage collection. A brief description of the components:
eden space - where most objects are born (and most of them die)
from-space - where live objects exist after a minor garbage collection
to-space - where objects will be copied to during a minor garbage collection
old gen object space - where long-lived objects reside
perm gen object space - where instances of java.lang.Class reside
Normally sized objects are initially created in eden. When eden fills, a minor GC copies most live objects from eden and the from-space into the to-space; objects that have been around a while are instead put into old gen. At the end of the minor GC, the from- (now empty) and to-spaces swap roles. When things get too full, a major GC is initiated, which squeezes out any dead space from the old gen. One exception to the above is allocation of large objects, which are placed immediately into old gen. Your OOME may be due to attempting to allocate something big.
Since you didn't provide any information about the JVM version, GC settings, platform you're running on, or what your applications do, it's hard to say whether or not your heap is sized appropriately. However, your total of around 262M is rather small in this age of 64-bit machines with multiple gigabytes of memory.
More information (not all of it up to date):
http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org