You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Stephanie Wullbieter <sw...@gmx.de> on 2008/04/22 09:18:53 UTC
Java VM slowly eats all memory up - without JVM heap OutOfMemory
failures!
Dear list members,
a tomcat machine has a problem because java eats the memory up. Here some data:
1.555.828 KiB host RAM
java version "1.4.2_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_06-b03)
Java HotSpot(TM) Client VM (build 1.4.2_06-b03, mixed mode)
Tomcat Version 4.1.31
java ist started with the command /home/tomcatp/prod/java2/bin/java -Dtomcat.worker=2 -server -Djava.library.path=/home/tomcatp/prod/dctm5/dfc -Xmx768m -verbose:gc -Dcom.sun.jndi.ldap.connect.pool.timeout=60000 -Dcom.sun.jndi.ldap.connect.pool.maxsize=5000 -Djava.awt.headless=true -Djava.endorsed.dirs=/home/tomcatp/prod/tomcat4/common/endorsed -classpath /home/tomcatp/prod/dctm5/dfc/dfc.jar:/home/tomcatp/prod/dctm5/dfc/bsf.jar:/home/tomcatp/prod/dctm5/dfc/log4j.jar:/home/tomcatp/prod/dctm5/dfc/xalan.jar:/home/tomcatp/prod/dctm5/dfc/xml-apis.jar:/home/tomcatp/prod/dctm5/dfc/workflow.jar:/home/tomcatp/prod/dctm5/dfc/subscription.jar:/home/tomcatp/prod/dctm5/dfc/xforms.jar:/home/tomcatp/prod/dctm5/dfc/All-MB.jar:/home/tomcatp/prod/dctm5/dfc/ci.jar:/home/tomcatp/prod/dctm5/dfc/bpmutil.jar:/home/tomcatp/prod/dctm5/dfc/mail.jar:/home/tomcatp/prod/dctm5/dfc/activation.jar:/home/tomcatp/prod/dctm5/config:/home/tomcatp/prod/java2/lib/tools.jar:/home/tomcatp/prod/tomcat4/bin/bootstrap.
jar -Dcatalina.base=/home/tomcatp/prod/tomcat4 -Dcatalina.home=/home/tomcatp/prod/tomcat4 -Djava.io.tmpdir=/home/tomcatp/prod/tomcat4/temp org.apache.catalina.startup.Bootstrap start
After e.g. 14 days running (200 application users), the java applications on the tomcat get instable, i think due to memory problems because the memory of the host gets eaten slowly but constantly up. In the end it looks like this:
free -k
total used free shared buffers cached
Mem: 1555828 1522108 33720 0 71984 78676
-/+ buffers/cache: 1371448 184380
Swap: 265064 265064 0
So the host seems to be under memory pressure and it's a question of time that it comes to instability.
ps aux
tomcatp 19814 0.7 85.5 1860356 1330700 ? Sl Apr08 152:48 /home/tomcatp/prod/java2/bin/java -Dtomcat
The Java VM has no OutOfMemory failures, the JVM usage is about 600 MiB, and does not go over 700 MiB (768 MiB JVM Heap).
So it looks like Java VM is using about 700 MiB for itself. Is that normal?
Yesterday the tomcat got problems with one application. So I switched to a fallback server and did not restart the tomcat but only reloaded the little application that did not work properly anymore. So there were no users anymore on that tomcat. And now, after 12h, the memory usage of java is still the same, it has not decreased. So what's up there? Why is there no memory freed if the applications are not used anymore? When the tomcat is started, java only uses 300 MiB overall, then it increases constantly.
It would be fine, if someone could help me or give me a tip how to analyze the tomcat to get more info what could be the problem.
Could it be a solution to decrease the JVM heap?
Many Thanks!
Stephanie
--
Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games!
http://games.entertainment.gmx.net/de/entertainment/games/free
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Java VM slowly eats all memory up - without JVM heap OutOfMemory
failures!
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Stephanie,
Stephanie Wullbieter wrote:
| -Xmx768m
| -verbose:gc
Do you see GC activity in stdout? You should be able to get some heap
metrics easily by looking at that. What does the heap use curve look like?
| total used free shared buffers cached
| Mem: 1555828 1522108 33720 0 71984 78676
| -/+ buffers/cache: 1371448 184380
| Swap: 265064 265064 0
|
| So the host seems to be under memory pressure and it's a question of
time that it comes to instability.
To quote a wise newsgroup (or blog?) sage: "what good memory if you're
not using it?". Seriously. Why are you worried that your machine's
memory is not free? 'free' reports that no wrap is being used. It looks
like your kernel is managing your memory quite well.
| ps aux
| tomcatp 19814 0.7 85.5 1860356 1330700 ? Sl Apr08 152:48
/home/tomcatp/prod/java2/bin/java -Dtomcat
Okay, this is kinda weird. You started the JVM with a 768MiB heap, and
the process appears to be using a little less than double that. It seems
to me that your heap has reached its maximum size (768MiB), which
accounts for a bit over half of the allocation of memory. How many
connectors do you have enabled in Tomcat? Per connector, how many
connections are you allowing it to accept? Are you using any special
connector like NIO, APR, or Comet?
If you have a lot of threads allocated through connectors, they all need
lots of memory to stay running.
| The Java VM has no OutOfMemory failures, the JVM usage is about 600
MiB, and does not go over 700 MiB (768 MiB JVM Heap).
| So it looks like Java VM is using about 700 MiB for itself. Is that
normal?
Yes, if you have lots of threads.
|
| When the tomcat is started, java only uses 300 MiB overall, then it
increases constantly.
Your JVM will never release heap memory back to the operating system, so
a process memory size that only increases is not cause for alarm. If you
are getting no OOMEs and everything runs properly, why are you so concerned?
You never mentioned what type of "problem" occurred to your application.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkgOEYMACgkQ9CaO5/Lv0PCjUQCfTTGM9YSq7VS9l+IgEMr/osYN
iJ0An18KjNs8KBpjz5TnX8FsF2vD0T/P
=ZnOF
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: RE: Java VM slowly eats all memory up - without JVM heap
OutOfMemory failures!
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Stephanie Wullbieter [mailto:swullb@gmx.de]
> SUSE LINUX Enterprise Server 9 (i586)
> Linux-Kernel 2.6.5-7
man strace
man ltrace
Good luck! I'd use strace as it gives you more information, but ltrace may be a faster way of finding open/close mismatches. Annoyingly, you'll have to start Tomcat with strace - you might want to do that on your test system, assuming you have one.
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: RE: Java VM slowly eats all memory up - without JVM heap
OutOfMemory failures!
Posted by Stephanie Wullbieter <sw...@gmx.de>.
Thank You for the tip!
The OS is
SUSE LINUX Enterprise Server 9 (i586)
Linux-Kernel 2.6.5-7
> Can you monitor the system calls made by the application over a period of
> time, looking for systematic mismatches between resources that are opened
> and resources that are closed? I'm not sure which variant of UNIX you're
> running, but most have some way of monitoring a process' system calls. If
> you have the /proc filesystem on your system, looking through /proc for your
> process may also help.
--
GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen!
Jetzt dabei sein: http://www.shortview.de/?mc=sv_ext_mf@gmx
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Java VM slowly eats all memory up - without JVM heap
OutOfMemory failures!
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Stephanie Wullbieter [mailto:swullb@gmx.de]
> a tomcat machine has a problem because java eats the memory
> up. Here some data:
>
> 1.555.828 KiB host RAM
>
> java version "1.4.2_06"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_06-b03)
> Java HotSpot(TM) Client VM (build 1.4.2_06-b03, mixed mode)
>
> Tomcat Version 4.1.31
[...]
> The Java VM has no OutOfMemory failures, the JVM usage is
> about 600 MiB, and does not go over 700 MiB (768 MiB JVM Heap).
> So it looks like Java VM is using about 700 MiB for itself.
> Is that normal?
Sounds like your application is not correctly releasing some operating system resource, so the JVM is keeping hold of the resource. This would cause memory to be reserved outside the Java heap but inside the process. Good candidates are: files, threads (which have both a Java stack and a host stack, and it might be the host stack not being released), almost anything JNI. I've no doubt there are other possibilities as well!
Can you monitor the system calls made by the application over a period of time, looking for systematic mismatches between resources that are opened and resources that are closed? I'm not sure which variant of UNIX you're running, but most have some way of monitoring a process' system calls. If you have the /proc filesystem on your system, looking through /proc for your process may also help.
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Java VM slowly eats all memory up - without JVM heap OutOfMemory
failures!
Posted by Rainer Jung <ra...@kippdata.de>.
Stephanie Wullbieter schrieb:
> Dear list members,
>
> a tomcat machine has a problem because java eats the memory up. Here some data:
>
> 1.555.828 KiB host RAM
>
> java version "1.4.2_06"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_06-b03)
> Java HotSpot(TM) Client VM (build 1.4.2_06-b03, mixed mode)
>
> Tomcat Version 4.1.31
>
> java ist started with the command /home/tomcatp/prod/java2/bin/java -Dtomcat.worker=2 -server -Djava.library.path=/home/tomcatp/prod/dctm5/dfc -Xmx768m -verbose:gc -Dcom.sun.jndi.ldap.connect.pool.timeout=60000 -Dcom.sun.jndi.ldap.connect.pool.maxsize=5000 -Djava.awt.headless=true -Djava.endorsed.dirs=/home/tomcatp/prod/tomcat4/common/endorsed -classpath /home/tomcatp/prod/dctm5/dfc/dfc.jar:/home/tomcatp/prod/dctm5/dfc/bsf.jar:/home/tomcatp/prod/dctm5/dfc/log4j.jar:/home/tomcatp/prod/dctm5/dfc/xalan.jar:/home/tomcatp/prod/dctm5/dfc/xml-apis.jar:/home/tomcatp/prod/dctm5/dfc/workflow.jar:/home/tomcatp/prod/dctm5/dfc/subscription.jar:/home/tomcatp/prod/dctm5/dfc/xforms.jar:/home/tomcatp/prod/dctm5/dfc/All-MB.jar:/home/tomcatp/prod/dctm5/dfc/ci.jar:/home/tomcatp/prod/dctm5/dfc/bpmutil.jar:/home/tomcatp/prod/dctm5/dfc/mail.jar:/home/tomcatp/prod/dctm5/dfc/activation.jar:/home/tomcatp/prod/dctm5/config:/home/tomcatp/prod/java2/lib/tools.jar:/home/tomcatp/prod/tomcat4/bin/bootstr
ap.
> jar -Dcatalina.base=/home/tomcatp/prod/tomcat4 -Dcatalina.home=/home/tomcatp/prod/tomcat4 -Djava.io.tmpdir=/home/tomcatp/prod/tomcat4/temp org.apache.catalina.startup.Bootstrap start
>
> After e.g. 14 days running (200 application users), the java applications on the tomcat get instable, i think due to memory problems because the memory of the host gets eaten slowly but constantly up. In the end it looks like this:
>
> free -k
> total used free shared buffers cached
> Mem: 1555828 1522108 33720 0 71984 78676
> -/+ buffers/cache: 1371448 184380
> Swap: 265064 265064 0
>
> So the host seems to be under memory pressure and it's a question of time that it comes to instability.
>
> ps aux
> tomcatp 19814 0.7 85.5 1860356 1330700 ? Sl Apr08 152:48 /home/tomcatp/prod/java2/bin/java -Dtomcat
>
> The Java VM has no OutOfMemory failures, the JVM usage is about 600 MiB, and does not go over 700 MiB (768 MiB JVM Heap).
> So it looks like Java VM is using about 700 MiB for itself. Is that normal?
>
> Yesterday the tomcat got problems with one application. So I switched to a fallback server and did not restart the tomcat but only reloaded the little application that did not work properly anymore. So there were no users anymore on that tomcat. And now, after 12h, the memory usage of java is still the same, it has not decreased. So what's up there? Why is there no memory freed if the applications are not used anymore? When the tomcat is started, java only uses 300 MiB overall, then it increases constantly.
>
> It would be fine, if someone could help me or give me a tip how to analyze the tomcat to get more info what could be the problem.
>
> Could it be a solution to decrease the JVM heap?
Just a shot in the dark (since you are using somewhat old versions of
everything): there was a bug in java.io.File.deleteOnExit() that lead to
a leak in native memory. This problem hit a lot of code using
File.deleteOnExit(), e.g. commons-fileupload and others.
See:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4513817
https://issues.apache.org/jira/browse/FILEUPLOAD-95
http://blogs.sun.com/chegar/#diagnosing_deleteonexit_issues
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org