You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jacob Bunk Nielsen <jb...@bunk.cc> on 2007/06/15 10:47:24 UTC
Problem with JVM, and Tomcat tuning
Hi
I'm running Tomcat 6.0.13 on Sun Java6 and Linux. I upgraded from
Tomcat 5.5.17 and Java5 update 9 a couple of weeks ago.
I experience random crashes of the JVM on a daily basis. It simply
fails with segmentation faults, double frees and similar. Are any of
you experiencing similar problems? I had the same problems with Java5
and Tomcat 5.5, but less often.
If I submit a bug report with Sun I get a reply that I should contact
Apache because the problem seems to be triggered by Tomcat. Is anyone
at Apache interested in those hs_err_PID.log files generated by the
JVM?
So, how would I go about avoiding to trigger these sorts of problems?
It is a production system and I'm not very happy about it being down
almost daily due to bugs in the JVM.
I should add that I run my fairly memory consuming application on a 32
bit server with 4 GB of physical memory and 8 GB of swap space and
with CATALINA_OPTS set as:
CATALINA_OPTS=-Djava.awt.headless=true -server -Xms512m -Xmx2g
-verbose:gc -XX:+PrintGCTimeStamps -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+UseTLAB
-XX:+UsePerfData -XX:NewSize=128m -XX:MaxNewSize=128m
-XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=1024 -Xincgc
I tried some even more agressive garbage collector settings a while
back because I was experiencing long pauses while the garbage
collector was running, but I kept triggering loads of segmentation
faults. This is the settings that around a year ago gave the best
balance between crashes and halts from the garbage collector.
But are these settings optimal for Tomcat 6 and Java6?
--
Jacob
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Jacob Bunk Nielsen <jb...@bunk.cc>.
Hi
Mark Thomas <ma...@apache.org> writes:
> Jacob Bunk Nielsen wrote:
>> If I submit a bug report with Sun I get a reply that I should contact
>> Apache because the problem seems to be triggered by Tomcat. Is anyone
>> at Apache interested in those hs_err_PID.log files generated by the
>> JVM?
> If you are using the native connector, it is possible that Tomcat is
> the cause. In this case open a bugzilla item and add the log files and
> any other relevant information. Do you have a url for a Sun bug report
> we can look at?
I'm using the Java connector as far as I can figure. I'm more or less
using the standard configuration that comes with Tomcat.
I don't have a Sun bug to link at because they won't accept my bug. In
stead they say:
,----[ From the email that I got after Sun reviewed my lastest bug report ]
| Thank you for reporting this issue.
|
| Please ask Apache to investigate this issue further. Since Tomcat
| appears to trigger this failure, the developers at Apache will be able
| to examine their product, isolate the cause of the behavior, and, if
| necessary, provide us with a detailed report.
|
| We greatly appreciate your efforts in identifying areas in the Java
| Standard Edition where we can improve upon and I would request you to
| continue doing so.
`----
I have gotten this standard reply several times by now.
> If you are using the pure Java configuration, I don't see what we can
> do - it would be an issue for Sun in that case.
I agree!
>> But are these settings optimal for Tomcat 6 and Java6?
> Your application is usually the biggest factor.
I know.
However, I also noticed that Tomcat 6 features "Memory usage
optimizations". That's why I asked.
> If you have the cycles, you might want to repeat the testing you did
> that arrived at these settings but using your new TC6/Java6
> configuration.
It took a lot of time and a lot of segmentation faults (resulting in
my application being unavailable) to find these settings. The upside
was that I got the script that automatically restarts Tomcat when the
JVM dies perfected to a degree I had hoped wasn't necessary.
--
Jacob
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Mark Thomas <ma...@apache.org>.
Jacob Bunk Nielsen wrote:
> If I submit a bug report with Sun I get a reply that I should contact
> Apache because the problem seems to be triggered by Tomcat. Is anyone
> at Apache interested in those hs_err_PID.log files generated by the
> JVM?
If you are using the native connector, it is possible that Tomcat is
the cause. In this case open a bugzilla item and add the log files and
any other relevant information. Do you have a url for a Sun bug report
we can look at?
If you are using the pure Java configuration, I don't see what we can
do - it would be an issue for Sun in that case.
> So, how would I go about avoiding to trigger these sorts of problems?
> It is a production system and I'm not very happy about it being down
> almost daily due to bugs in the JVM.
If using the native connector, switch to the pure Java one in production.
> But are these settings optimal for Tomcat 6 and Java6?
Your application is usually the biggest factor. If you have the
cycles, you might want to repeat the testing you did that arrived at
these settings but using your new TC6/Java6 configuration.
Mark
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Jacob Bunk Nielsen <jb...@bunk.cc>.
Andrew Miehs <an...@2sheds.de> writes:
> What messages are you getting 'EXACTLY'?
>From one of the hs_err_pid.log:
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xb7ec6e9c, pid=13753, tid=719874992
#
# Java VM: Java HotSpot(TM) Server VM (1.6.0-b105 mixed mode)
# Problematic frame:
# C [libc.so.6+0x72e9c]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
> You may also want to look in /var/log/messages and dmesg and see if
> you are
> seeing any strange messages in there...
No weird things in /var/log/messages - the JVM is just a regurlar
userland application that crashes, and I don't thing it logs anything
via syslog.
> PS: The server isn't a SuperMicro server is it? (or rebadged like
> Transtec)
No, they are both Dell PowerEdge. However, I have also seen the same
thing happen on "my" development server that is a HP DL 585.
--
Jacob
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Andrew Miehs <an...@2sheds.de>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 15/06/2007, at 4:36 PM, Peter Crowther wrote:
>> From: Hassan Schroeder [mailto:hassan.schroeder@gmail.com]
>> On 6/15/07, Jacob Bunk Nielsen <jb...@bunk.cc> wrote:
>>> I experience random crashes of the JVM on a daily basis. It simply
>>> fails with segmentation faults, double frees and similar. Are any of
>>> you experiencing similar problems? I had the same problems
>> with Java5
>>> and Tomcat 5.5, but less often.
>>
>> I've had no such problems with either of the combinations you mention
>> on SuSE10/2.6.13-15-default or RHES3/2.4.21-9.EL. ...
>
> Jacob, are you *absolutely* sure the hardware is solid? Does anything
> else ever crash? Does the same setup work on another machine (I
> appreciate this is not always easy to test)?
I have to agree with Peter. This definitely sounds like a hardware
problem.
'Random' errors sound very much like faulty RAM. As you were also having
this problem before, it is obviously not a Java 6 problem.
What messages are you getting 'EXACTLY'?
You may also want to look in /var/log/messages and dmesg and see if
you are
seeing any strange messages in there...
Cheers
Andrew
PS: The server isn't a SuperMicro server is it? (or rebadged like
Transtec)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)
iD8DBQFGcqZCW126qUNSzvURAsdTAJ9Mh3TQo8WuX+1+cs1LpYC1bxGQHwCfYTUR
+iw9g74h0B8SrSJEvRqmI5c=
=Hxyk
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Jacob Bunk Nielsen <jb...@bunk.cc>.
"Peter Crowther" <Pe...@melandra.com> writes:
> Jacob, are you *absolutely* sure the hardware is solid? Does anything
> else ever crash? Does the same setup work on another machine (I
> appreciate this is not always easy to test)?
Yes, I'm quite sure the hardware is solid.
Actually I'm running the application on two almost identical servers -
one is a Dell PowerEdge 6600 and the other one is a Dell PowerEdge
6650 purchased roughly a year later than the 6600, so it has newer
RAID controllers and such.
> Also, which Linux?
Currently Debian Sarge. However, I saw the same problems when it was
running Potato. I should get around to upgrading to Etch sometime
soon.
I'm currently running on kernels that I compiled from kernel.org, and
the two servers are running separate kernelversions at the moment. I
see the same JVM-issues on both.
Also I don't see anything else crashing on those two systems, and they
both keep high uptimes and basically seem rock-solid.
--
Jacob
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Problem with JVM, and Tomcat tuning
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Hassan Schroeder [mailto:hassan.schroeder@gmail.com]
> On 6/15/07, Jacob Bunk Nielsen <jb...@bunk.cc> wrote:
>
> > I'm running Tomcat 6.0.13 on Sun Java6 and Linux.
>
> > I experience random crashes of the JVM on a daily basis. It simply
> > fails with segmentation faults, double frees and similar. Are any of
> > you experiencing similar problems? I had the same problems
> with Java5
> > and Tomcat 5.5, but less often.
>
> I've had no such problems with either of the combinations you mention
> on SuSE10/2.6.13-15-default or RHES3/2.4.21-9.EL. ...
Jacob, are you *absolutely* sure the hardware is solid? Does anything
else ever crash? Does the same setup work on another machine (I
appreciate this is not always easy to test)?
Also, which Linux? Not all kernels or distros are created equal. Linux
is now where Windows was about a decade ago*, with the equivalent of DLL
hell but with the added bonus of some buggy kernels as well as all the
usual issues of buggy drivers and libraries. It used to be a small,
cute system back when I started with it in 1995; now there can be some
unpleasant stability issues that can only show up with some combinations
of kernel, modules and applications.
- Peter
* Some would argue it still is, although IMHO that's more an issue with
application developers than Microsoft now.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Hassan Schroeder <ha...@gmail.com>.
On 6/15/07, Jacob Bunk Nielsen <jb...@bunk.cc> wrote:
> I'm running Tomcat 6.0.13 on Sun Java6 and Linux.
> I experience random crashes of the JVM on a daily basis. It simply
> fails with segmentation faults, double frees and similar. Are any of
> you experiencing similar problems? I had the same problems with Java5
> and Tomcat 5.5, but less often.
I've had no such problems with either of the combinations you mention
on SuSE10/2.6.13-15-default or RHES3/2.4.21-9.EL. ...
> I should add that I run my fairly memory consuming application on a 32
> bit server with 4 GB of physical memory and 8 GB of swap space and
> with CATALINA_OPTS set as:
.. but my systems' OPTS are pretty simple: -Xms256M -Xmx256M for
one and -Xms512m -Xmx512m for the other.
In any case I've never had a JVM crash.
HTH,
--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Rainer Jung <ra...@kippdata.de>.
No, sorry I misread you mail telling us that you upgraded from 1.5.0_09.
Java 6 should be OK, there is a 1.6.0_01 out for a couple of weeks now.
Still interested in your hotspot error files next week.
Regards,
Rainer
Jacob Bunk Nielsen schrieb:
> Rainer Jung <ra...@kippdata.de> writes:
>
>> Give us a look at your hotspot error files.
>
> I'll put a few online when I get back to work on Monday.
>
>> If double free is really the reason it won't be caused by a pure Java
>> Tomcat, instead it could be native libs integrated into your webapps by
>> JNI or a bug between the JVM and your platform.
>
> I think the "double free" error was only found in catalina.out right
> after a crash, no hs_err_pid.log file was generated, but still my Java
> process ended.
>
>> A second thing: since some of your JVM options are very advanced,
>> consider upgrading to Java 1.5.0_12. JVM updates are usually low risk
>> and easy to do and most of the patch releases enhance stability of the
>> more advanced GC options.
>
> Currently I'm running Java 1.6.0 - are you suggesting that I
> downgrade?
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Jacob Bunk Nielsen <jb...@bunk.cc>.
Rainer Jung <ra...@kippdata.de> writes:
> Give us a look at your hotspot error files.
I'll put a few online when I get back to work on Monday.
> If double free is really the reason it won't be caused by a pure Java
> Tomcat, instead it could be native libs integrated into your webapps by
> JNI or a bug between the JVM and your platform.
I think the "double free" error was only found in catalina.out right
after a crash, no hs_err_pid.log file was generated, but still my Java
process ended.
> A second thing: since some of your JVM options are very advanced,
> consider upgrading to Java 1.5.0_12. JVM updates are usually low risk
> and easy to do and most of the patch releases enhance stability of the
> more advanced GC options.
Currently I'm running Java 1.6.0 - are you suggesting that I
downgrade?
--
Jacob
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Jacob Bunk Nielsen <jb...@bunk.cc>.
Mark Thomas <ma...@apache.org> writes:
> Jacob Bunk Nielsen wrote:
>> Rainer Jung <ra...@kippdata.de> writes:
>>
>>> Give us a look at your hotspot error files.
>>
>> I have put a few online at http://flof.dk/~jbn/hs_err/ [...]
>
> The first 2 are Java2D problems, not Tomcat issues. Sun need to take a
> look at these. Feel free to quote me when you try and get them to open
> a bug.
Thank you for you support. I have replied to their standard answer and
said that you believe (just as I also do) that it is a Java problem,
not a Tomcat problem.
I hope they will accept my bug report, so the error can be fixed soon.
--
Jacob
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Mark Thomas <ma...@apache.org>.
Jacob Bunk Nielsen wrote:
> Rainer Jung <ra...@kippdata.de> writes:
>
>> Give us a look at your hotspot error files.
>
> I have put a few online at http://flof.dk/~jbn/hs_err/ - now that I
> look at it two of them are OutOfMemory-errors.
The first 2 are Java2D problems, not Tomcat issues. Sun need to take a
look at these. Feel free to quote me when you try and get them to open
a bug.
Mark
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Jacob Bunk Nielsen <jb...@bunk.cc>.
Rainer Jung <ra...@kippdata.de> writes:
> I agree with Mark, two of those look like coming from the Java 2D
> disposer thread, and the other two - as you already observed - come from
> OutOfMemory related to growableArray. A quick look into the bug database
> shows, that both areas unfortunately seem to be notorious for JVM
> crashes. There seems to be not bug fix in the 1.6.0_01 release notes
> related to those bugs.
Too bad - they are really hitting us hard, sometimes with several
crashes each day. I have been experiencing these types of bugs for
years now and submitted numerous bug reports, but because I am unable
to provide a minimal example triggering the bug(s) none of my bug
reports has been accepted.
> Sorry, but there seems to be nothing, we can do from the Tomcat side :(
Sort of already what I expected.
I was just hoping for some real life experiences from people who have
fought the same problems and perhaps found some workarounds.
--
Jacob
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Rainer Jung <ra...@kippdata.de>.
I agree with Mark, two of those look like coming from the Java 2D
disposer thread, and the other two - as you already observed - come from
OutOfMemory related to growableArray. A quick look into the bug database
shows, that both areas unfortunately seem to be notorious for JVM
crashes. There seems to be not bug fix in the 1.6.0_01 release notes
related to those bugs.
Sorry, but there seems to be nothing, we can do from the Tomcat side :(
Regards,
Rainer
Jacob Bunk Nielsen wrote:
> Rainer Jung <ra...@kippdata.de> writes:
>
>> Give us a look at your hotspot error files.
>
> I have put a few online at http://flof.dk/~jbn/hs_err/ - now that I
> look at it two of them are OutOfMemory-errors. I newer had those with
> Java5, but Sun has accepted a bugreport about it, and we are
> communicating about the issue, see:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6568433 ... for
> some of the discussion, I think it's quite a bit delayed, at least we
> have exchanged several emails that are not visible (yet?).
>
> However, the other two error files are segmentation faults. Not
> something I expect my JVM to be doing.
>
> Luckily the files I have put online is about a week old, meaning that
> I haven't seen crashes causing hs_err_pid.log files to be written
> within the past week. Not that I changed anything, so I'm still just
> waiting for the next crash.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Jacob Bunk Nielsen <jb...@bunk.cc>.
Rainer Jung <ra...@kippdata.de> writes:
> Give us a look at your hotspot error files.
I have put a few online at http://flof.dk/~jbn/hs_err/ - now that I
look at it two of them are OutOfMemory-errors. I newer had those with
Java5, but Sun has accepted a bugreport about it, and we are
communicating about the issue, see:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6568433 ... for
some of the discussion, I think it's quite a bit delayed, at least we
have exchanged several emails that are not visible (yet?).
However, the other two error files are segmentation faults. Not
something I expect my JVM to be doing.
Luckily the files I have put online is about a week old, meaning that
I haven't seen crashes causing hs_err_pid.log files to be written
within the past week. Not that I changed anything, so I'm still just
waiting for the next crash.
--
Jacob
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with JVM, and Tomcat tuning
Posted by Rainer Jung <ra...@kippdata.de>.
Give us a look at your hotspot error files.
Do they look very close to each other, or do they show very different
stacks as the reason for the crash?
If double free is really the reason it won't be caused by a pure Java
Tomcat, instead it could be native libs integrated into your webapps by
JNI or a bug between the JVM and your platform.
Nevertheless a view at your hotspot error files should give a hint.
A second thing: since some of your JVM options are very advanced,
consider upgrading to Java 1.5.0_12. JVM updates are usually low risk
and easy to do and most of the patch releases enhance stability of the
more advanced GC options.
Regards,
Rainer
Jacob Bunk Nielsen wrote:
> Hi
>
> I'm running Tomcat 6.0.13 on Sun Java6 and Linux. I upgraded from
> Tomcat 5.5.17 and Java5 update 9 a couple of weeks ago.
>
> I experience random crashes of the JVM on a daily basis. It simply
> fails with segmentation faults, double frees and similar. Are any of
> you experiencing similar problems? I had the same problems with Java5
> and Tomcat 5.5, but less often.
>
> If I submit a bug report with Sun I get a reply that I should contact
> Apache because the problem seems to be triggered by Tomcat. Is anyone
> at Apache interested in those hs_err_PID.log files generated by the
> JVM?
>
> So, how would I go about avoiding to trigger these sorts of problems?
> It is a production system and I'm not very happy about it being down
> almost daily due to bugs in the JVM.
>
> I should add that I run my fairly memory consuming application on a 32
> bit server with 4 GB of physical memory and 8 GB of swap space and
> with CATALINA_OPTS set as:
>
> CATALINA_OPTS=-Djava.awt.headless=true -server -Xms512m -Xmx2g
> -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseConcMarkSweepGC
> -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+UseTLAB
> -XX:+UsePerfData -XX:NewSize=128m -XX:MaxNewSize=128m
> -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=1024 -Xincgc
>
> I tried some even more agressive garbage collector settings a while
> back because I was experiencing long pauses while the garbage
> collector was running, but I kept triggering loads of segmentation
> faults. This is the settings that around a year ago gave the best
> balance between crashes and halts from the garbage collector.
>
> But are these settings optimal for Tomcat 6 and Java6?
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org