You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Carl Kabbe <ca...@etrak-plus.com> on 2012/02/28 02:32:07 UTC

Tomcat suddenly dies

I have run Tomcat in two environments and both produced the same results.

Server 1:

OS - Slackware Linux 13.x 64 bit
JVM - 1.6.0_31
Tomcat - 6.0.24

Server memory 16GB
Heap 2GB, PermGen 300M

Server 2:

OS - Slackware 13.x 64 bit
JVM - 1.6.0_31
Tomcat - 7.0.14

Server memory 8GB
Heap 2GB, PermGen 300M

The history:

Two years ago, I was running the application on JVM 1.6.0_18 and having a similar or the same problem and Charles suggested I go back to 1.6.0_7.  I did and the app ran pretty flawlessly (except for screwups in the code changes.)  We have been adding a number of clients so I am moving toward a clustered environment and trying to update the JVM, Tomcat, etc.  

The problem: 

Starting about a month ago, Tomcat would suddenly fail, no heap dump, no indication of trouble in the log, no indication the JVM crashed (JVisualVM continues to hum along), nothing (that I can find.)  At first, we thought it was load related as our heaviest loads are in the early afternoon and the crashes were consistently 1-2:00PM.  Then, last Friday, it crashed around 4:00PM when there was relatively light load.  As nearly as we can tell, there is no slowness or other indicator the failure is coming… the system just hums along and bam, gone.

The garbage collection and cpu utilization look pretty normal (garbage collection is the typical saw edge and cpu utilization is 0-20% with an occasional short duration spike.)

I am going back to 1.6.0_7 on server 1 tomorrow.

I was thinking the only way to potentially get to the bottom of this would be to take frequent heap dumps between 1:00 and 2:00PM but, while they may be interesting, I am not certain they will show anything useful as the system shows no signs of strain before dying.

Anyone have any ideas?

Thanks,

Carl


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat suddenly dies

Posted by Carl Kabbe <ca...@etrak-plus.com>.
Dan,

Thanks for the response.

> Are you accessing any native code from Java?  For example using JNI /
> JNA or a third party module which requires you to set
> "java.library.path"?


No.

Thanks,

Carl


On Feb 28, 2012, at 2:35 PM, Daniel Mikusa wrote:

> On Tue, 2012-02-28 at 11:09 -0800, Carl Kabbe wrote:
>> Ranier,
>> 
>> Thanks for your response and thoughts.
>> 
>>> Are there normal shutdown messages in the Tomcat logs?
>> 
>> 
>> No, the catalina.out log has our messages but none of the normal shutdown stuff (we see the shutdown messaging every day when we push changes and restart Tomcat at 4:00AM.)
> 
> Are you accessing any native code from Java?  For example using JNI /
> JNA or a third party module which requires you to set
> "java.library.path"?
> 
> Dan
> 
> 
>> 
>> Thanks,
>> 
>> Carl
>> 
>> On Feb 28, 2012, at 2:00 PM, Rainer Jung wrote:
>> 
>>> On 28.02.2012 19:47, Carl Kabbe wrote:
>>>> Chuck and Chris,
>>>> 
>>>> Thanks for your replies.  Below is some information to your questions/suggestions:
>>>> 
>>>>>> Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not
>>>>>> just the Tomcat ones.  Also, look for a JVM dump file
>>>>>> (hs_err_pid*.log)
>>>> 
>>>> 
>>>> I have and there is nothing in the messages file except accesses granted to specific workstations coming in on ssh and sync'ing to a time server.  Neither of these have times that correspond to the crashes.
>>>> 
>>>> There are no hs_err_* files anywhere on the servers.
>>>> 
>>>>> Smells a lot like OOM killer.
>>>>> 
>>>>> Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit
>>>>> JVM? What about other large-memory processes on the same boxes? Do you
>>>>> have other JVMs running or a database, etc.? Does the JVM die on any
>>>>> kind of schedule?
>>>> 
>>>> 
>>>> We are running 64 bit OS's (Slackware 13.x, the latest version.)
>>>> 
>>>> There are two other applications running on each of the boxes: 1) the Apache James email server (localhost SMTP only) and 2) a small application that serves reports.  They are both very small (the current server shows 11GB+ free memory) and always survive theTomcat crashes.
>>>> 
>>>> These servers are only used for Tomcat (and the related James and report serving app.)
>>>> 
>>>> Not on a timed schedule but usually during high traffic periods (usually, but not always, as with last Friday.)
>>> 
>>> Are there normal shutdown messages in the Tomcat logs?
>>> 
>>> Regards,
>>> 
>>> Rainer
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>> 
>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat suddenly dies

Posted by Daniel Mikusa <dm...@vmware.com>.
On Tue, 2012-02-28 at 11:09 -0800, Carl Kabbe wrote:
> Ranier,
> 
> Thanks for your response and thoughts.
> 
> > Are there normal shutdown messages in the Tomcat logs?
> 
> 
> No, the catalina.out log has our messages but none of the normal shutdown stuff (we see the shutdown messaging every day when we push changes and restart Tomcat at 4:00AM.)

Are you accessing any native code from Java?  For example using JNI /
JNA or a third party module which requires you to set
"java.library.path"?

Dan


> 
> Thanks,
> 
> Carl
> 
> On Feb 28, 2012, at 2:00 PM, Rainer Jung wrote:
> 
> > On 28.02.2012 19:47, Carl Kabbe wrote:
> >> Chuck and Chris,
> >> 
> >> Thanks for your replies.  Below is some information to your questions/suggestions:
> >> 
> >>>> Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not
> >>>> just the Tomcat ones.  Also, look for a JVM dump file
> >>>> (hs_err_pid*.log)
> >> 
> >> 
> >> I have and there is nothing in the messages file except accesses granted to specific workstations coming in on ssh and sync'ing to a time server.  Neither of these have times that correspond to the crashes.
> >> 
> >> There are no hs_err_* files anywhere on the servers.
> >> 
> >>> Smells a lot like OOM killer.
> >>> 
> >>> Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit
> >>> JVM? What about other large-memory processes on the same boxes? Do you
> >>> have other JVMs running or a database, etc.? Does the JVM die on any
> >>> kind of schedule?
> >> 
> >> 
> >> We are running 64 bit OS's (Slackware 13.x, the latest version.)
> >> 
> >> There are two other applications running on each of the boxes: 1) the Apache James email server (localhost SMTP only) and 2) a small application that serves reports.  They are both very small (the current server shows 11GB+ free memory) and always survive theTomcat crashes.
> >> 
> >> These servers are only used for Tomcat (and the related James and report serving app.)
> >> 
> >> Not on a timed schedule but usually during high traffic periods (usually, but not always, as with last Friday.)
> > 
> > Are there normal shutdown messages in the Tomcat logs?
> > 
> > Regards,
> > 
> > Rainer
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> > 
> > 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>

Re: Tomcat suddenly dies

Posted by Carl Kabbe <ca...@etrak-plus.com>.
Ranier,

Thanks for your response and thoughts.

> Are there normal shutdown messages in the Tomcat logs?


No, the catalina.out log has our messages but none of the normal shutdown stuff (we see the shutdown messaging every day when we push changes and restart Tomcat at 4:00AM.)

Thanks,

Carl

On Feb 28, 2012, at 2:00 PM, Rainer Jung wrote:

> On 28.02.2012 19:47, Carl Kabbe wrote:
>> Chuck and Chris,
>> 
>> Thanks for your replies.  Below is some information to your questions/suggestions:
>> 
>>>> Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not
>>>> just the Tomcat ones.  Also, look for a JVM dump file
>>>> (hs_err_pid*.log)
>> 
>> 
>> I have and there is nothing in the messages file except accesses granted to specific workstations coming in on ssh and sync'ing to a time server.  Neither of these have times that correspond to the crashes.
>> 
>> There are no hs_err_* files anywhere on the servers.
>> 
>>> Smells a lot like OOM killer.
>>> 
>>> Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit
>>> JVM? What about other large-memory processes on the same boxes? Do you
>>> have other JVMs running or a database, etc.? Does the JVM die on any
>>> kind of schedule?
>> 
>> 
>> We are running 64 bit OS's (Slackware 13.x, the latest version.)
>> 
>> There are two other applications running on each of the boxes: 1) the Apache James email server (localhost SMTP only) and 2) a small application that serves reports.  They are both very small (the current server shows 11GB+ free memory) and always survive theTomcat crashes.
>> 
>> These servers are only used for Tomcat (and the related James and report serving app.)
>> 
>> Not on a timed schedule but usually during high traffic periods (usually, but not always, as with last Friday.)
> 
> Are there normal shutdown messages in the Tomcat logs?
> 
> Regards,
> 
> Rainer
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat suddenly dies

Posted by Rainer Jung <ra...@kippdata.de>.
On 28.02.2012 19:47, Carl Kabbe wrote:
> Chuck and Chris,
>
> Thanks for your replies.  Below is some information to your questions/suggestions:
>
>>> Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not
>>> just the Tomcat ones.  Also, look for a JVM dump file
>>> (hs_err_pid*.log)
>
>
> I have and there is nothing in the messages file except accesses granted to specific workstations coming in on ssh and sync'ing to a time server.  Neither of these have times that correspond to the crashes.
>
> There are no hs_err_* files anywhere on the servers.
>
>> Smells a lot like OOM killer.
>>
>> Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit
>> JVM? What about other large-memory processes on the same boxes? Do you
>> have other JVMs running or a database, etc.? Does the JVM die on any
>> kind of schedule?
>
>
> We are running 64 bit OS's (Slackware 13.x, the latest version.)
>
> There are two other applications running on each of the boxes: 1) the Apache James email server (localhost SMTP only) and 2) a small application that serves reports.  They are both very small (the current server shows 11GB+ free memory) and always survive theTomcat crashes.
>
> These servers are only used for Tomcat (and the related James and report serving app.)
>
> Not on a timed schedule but usually during high traffic periods (usually, but not always, as with last Friday.)

Are there normal shutdown messages in the Tomcat logs?

Regards,

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat suddenly dies

Posted by Carl Kabbe <ca...@etrak-plus.com>.
Chuck and Chris,

Thanks for your replies.  Below is some information to your questions/suggestions:

>> Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not
>> just the Tomcat ones.  Also, look for a JVM dump file
>> (hs_err_pid*.log)


I have and there is nothing in the messages file except accesses granted to specific workstations coming in on ssh and sync'ing to a time server.  Neither of these have times that correspond to the crashes.

There are no hs_err_* files anywhere on the servers.

> Smells a lot like OOM killer.
> 
> Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit
> JVM? What about other large-memory processes on the same boxes? Do you
> have other JVMs running or a database, etc.? Does the JVM die on any
> kind of schedule?


We are running 64 bit OS's (Slackware 13.x, the latest version.)

There are two other applications running on each of the boxes: 1) the Apache James email server (localhost SMTP only) and 2) a small application that serves reports.  They are both very small (the current server shows 11GB+ free memory) and always survive theTomcat crashes.

These servers are only used for Tomcat (and the related James and report serving app.)

Not on a timed schedule but usually during high traffic periods (usually, but not always, as with last Friday.)

Thanks,

Carl


On Feb 28, 2012, at 11:13 AM, Christopher Schultz wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Chuck,
> 
> On 2/27/12 9:01 PM, Caldarale, Charles R wrote:
>>> From: Carl Kabbe [mailto:carl@etrak-plus.com] Subject: Tomcat
>>> suddenly dies
>> 
>>> Starting about a month ago, Tomcat would suddenly fail, no heap
>>> dump, no indication of trouble in the log, no indication the JVM
>>> crashed
>> 
>> Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not
>> just the Tomcat ones.  Also, look for a JVM dump file
>> (hs_err_pid*.log), possibly in the current directory of what was
>> the Tomcat process.
> 
> +1
> 
> Smells a lot like OOM killer.
> 
> Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit
> JVM? What about other large-memory processes on the same boxes? Do you
> have other JVMs running or a database, etc.? Does the JVM die on any
> kind of schedule?
> 
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAk9M/RcACgkQ9CaO5/Lv0PDtmQCgmnkl/KExQjSradWMoIvxEM+Y
> AWsAmgJudFWslkrIZLv7uID+uWUu9HZs
> =lUVS
> -----END PGP SIGNATURE-----
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat suddenly dies

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chuck,

On 2/27/12 9:01 PM, Caldarale, Charles R wrote:
>> From: Carl Kabbe [mailto:carl@etrak-plus.com] Subject: Tomcat
>> suddenly dies
> 
>> Starting about a month ago, Tomcat would suddenly fail, no heap
>> dump, no indication of trouble in the log, no indication the JVM
>> crashed
> 
> Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not
> just the Tomcat ones.  Also, look for a JVM dump file
> (hs_err_pid*.log), possibly in the current directory of what was
> the Tomcat process.

+1

Smells a lot like OOM killer.

Carl, you say you have a 2GiB heap. Are you using 32-bit or 64-bit
JVM? What about other large-memory processes on the same boxes? Do you
have other JVMs running or a database, etc.? Does the JVM die on any
kind of schedule?

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9M/RcACgkQ9CaO5/Lv0PDtmQCgmnkl/KExQjSradWMoIvxEM+Y
AWsAmgJudFWslkrIZLv7uID+uWUu9HZs
=lUVS
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: Tomcat suddenly dies

Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Carl Kabbe [mailto:carl@etrak-plus.com] 
> Subject: Tomcat suddenly dies

> Starting about a month ago, Tomcat would suddenly fail, no heap dump, 
> no indication of trouble in the log, no indication the JVM crashed

Check the kernel logs (e.g., /var/log/messages, /var/log/warn), not just the Tomcat ones.  Also, look for a JVM dump file (hs_err_pid*.log), possibly in the current directory of what was the Tomcat process.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat suddenly dies

Posted by "Au, Leon" <le...@amazon.com>.
On 2/27/12 5:32 PM, "Carl Kabbe" <ca...@etrak-plus.com> wrote:

>I have run Tomcat in two environments and both produced the same results.
>
>Server 1:
>
>OS - Slackware Linux 13.x 64 bit
>JVM - 1.6.0_31
>Tomcat - 6.0.24
>
>Server memory 16GB
>Heap 2GB, PermGen 300M
>
>Server 2:
>
>OS - Slackware 13.x 64 bit
>JVM - 1.6.0_31
>Tomcat - 7.0.14
>
>Server memory 8GB
>Heap 2GB, PermGen 300M
>
>The history:
>
>Two years ago, I was running the application on JVM 1.6.0_18 and having a
>similar or the same problem and Charles suggested I go back to 1.6.0_7.
>I did and the app ran pretty flawlessly (except for screwups in the code
>changes.)  We have been adding a number of clients so I am moving toward
>a clustered environment and trying to update the JVM, Tomcat, etc.
>
>The problem: 
>
>Starting about a month ago, Tomcat would suddenly fail, no heap dump, no
>indication of trouble in the log, no indication the JVM crashed
>(JVisualVM continues to hum along), nothing (that I can find.)  At first,
>we thought it was load related as our heaviest loads are in the early
>afternoon and the crashes were consistently 1-2:00PM.  Then, last Friday,
>it crashed around 4:00PM when there was relatively light load.  As nearly
>as we can tell, there is no slowness or other indicator the failure is
>comingŠ the system just hums along and bam, gone.
>
>The garbage collection and cpu utilization look pretty normal (garbage
>collection is the typical saw edge and cpu utilization is 0-20% with an
>occasional short duration spike.)
>
>I am going back to 1.6.0_7 on server 1 tomorrow.
>
>I was thinking the only way to potentially get to the bottom of this
>would be to take frequent heap dumps between 1:00 and 2:00PM but, while
>they may be interesting, I am not certain they will show anything useful
>as the system shows no signs of strain before dying.
>
>Anyone have any ideas?
>
>Thanks,
>
>Carl
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>For additional commands, e-mail: users-help@tomcat.apache.org
>

This may be a silly question, but do you have the -XX:ErrorFile flag
turned on for your JVM so that it would record the error data to a file?
I don't think heap dumps are recorded by default.

http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.htm
l#DebuggingOptions

Leon


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org