You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Carl <ca...@etrak-plus.com> on 2010/03/03 02:38:43 UTC
Re: Tomcat dies suddenly
This is an update.
Tried Sun JVM 1.6.0_16 (I had already tried 1.6.0_17 and 1.6.0_18... Taylan
thought 1.6.0_16 worked properly in his environment.) Failed with the same
indicator as the other tests: brought it up on Sunday, failed about 11:00AM
this morning (the T105 AMD 64 bit Slackware, etc.)
Switched to the T110 (64 bit Intel Slackware, etc.) and it failed at about
2:30PM today. The load today has been steady but not overly heavy.
Have had problems getting any information from strace. Konstantin was
correct that using start.sh caused a fork which meant strace had to be
started with the '-f' option. However, my first attempts caused strace to
crash:
root@tomcat2:/usr/local/tomcat_wars# *** glibc detected *** strace: free():
invalid next size (normal): 0x0000000000657dd0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x7f1f530347e8]
/lib64/libc.so.6(cfree+0x76)[0x7f1f53036da6]
strace[0x40b700]
strace[0x4061a5]
strace[0x40442e]
strace[0x405434]
/lib64/libc.so.6(__libc_start_main+0xe6)[0x7f1f52fdb526]
strace[0x402cf9]
======= Memory map: ========
00400000-0042c000 r-xp 00000000 08:03 55356
/usr/bin/strace
0062b000-00648000 rw-p 0002b000 08:03 55356
/usr/bin/strace
00648000-00678000 rw-p 00648000 00:00 0
[heap]
7f1f4c000000-7f1f4c021000 rw-p 7f1f4c000000 00:00 0
7f1f4c021000-7f1f50000000 ---p 7f1f4c021000 00:00 0
7f1f52da6000-7f1f52dbc000 r-xp 00000000 08:03 105
/usr/lib64/libgcc_s.so.1
7f1f52dbc000-7f1f52fbc000 ---p 00016000 08:03 105
/usr/lib64/libgcc_s.so.1
7f1f52fbc000-7f1f52fbd000 rw-p 00016000 08:03 105
/usr/lib64/libgcc_s.so.1
7f1f52fbd000-7f1f53124000 r-xp 00000000 08:01 10709
/lib64/libc-2.9.so
7f1f53124000-7f1f53323000 ---p 00167000 08:01 10709
/lib64/libc-2.9.so
7f1f53323000-7f1f53327000 r--p 00166000 08:01 10709
/lib64/libc-2.9.so
7f1f53327000-7f1f53328000 rw-p 0016a000 08:01 10709
/lib64/libc-2.9.so
7f1f53328000-7f1f5332d000 rw-p 7f1f53328000 00:00 0
7f1f5332d000-7f1f5334d000 r-xp 00000000 08:01 10751
/lib64/ld-2.9.so
7f1f53525000-7f1f53527000 rw-p 7f1f53525000 00:00 0
7f1f53549000-7f1f5354c000 rw-p 7f1f53549000 00:00 0
7f1f5354c000-7f1f5354d000 r--p 0001f000 08:01 10751
/lib64/ld-2.9.so
7f1f5354d000-7f1f5354e000 rw-p 00020000 08:01 10751
/lib64/ld-2.9.so
7fffa9d8a000-7fffa9d9f000 rw-p 7ffffffea000 00:00 0
[stack]
7fffa9df8000-7fffa9df9000 r-xp 7fffa9df8000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
If I started it as a background process:
root@tomcat2:/usr/local/tomcat/bin# strace -f -o
/usr/local/tomcat/logs/trace_log.txt.`date '+%Y-%m-%d:%H:%M:%S'`
/usr/local/tomcat/bin/catalina.sh start &
it quickly crashed:
root@tomcat2:/usr/local/tomcat/bin# #Using CATALINA_BASE:
/usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/local/java
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar
*** glibc detected *** strace: munmap_chunk(): invalid pointer:
0x0000000000657dd0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x7f2ff7aa57e8]
strace[0x40b700]
strace[0x4061a5]
strace[0x40442e]
strace[0x405434]
/lib64/libc.so.6(__libc_start_main+0xe6)[0x7f2ff7a4c526]
strace[0x402cf9]
======= Memory map: ========
00400000-0042c000 r-xp 00000000 08:03 55356
/usr/bin/strace
0062b000-00648000 rw-p 0002b000 08:03 55356
/usr/bin/strace
00648000-00678000 rw-p 00648000 00:00 0
[heap]
7f2ff7817000-7f2ff782d000 r-xp 00000000 08:03 105
/usr/lib64/libgcc_s.so.1
7f2ff782d000-7f2ff7a2d000 ---p 00016000 08:03 105
/usr/lib64/libgcc_s.so.1
7f2ff7a2d000-7f2ff7a2e000 rw-p 00016000 08:03 105
/usr/lib64/libgcc_s.so.1
7f2ff7a2e000-7f2ff7b95000 r-xp 00000000 08:01 10709
/lib64/libc-2.9.so
7f2ff7b95000-7f2ff7d94000 ---p 00167000 08:01 10709
/lib64/libc-2.9.so
7f2ff7d94000-7f2ff7d98000 r--p 00166000 08:01 10709
/lib64/libc-2.9.so
7f2ff7d98000-7f2ff7d99000 rw-p 0016a000 08:01 10709
/lib64/libc-2.9.so
7f2ff7d99000-7f2ff7d9e000 rw-p 7f2ff7d99000 00:00 0
7f2ff7d9e000-7f2ff7dbe000 r-xp 00000000 08:01 10751
/lib64/ld-2.9.so
7f2ff7f96000-7f2ff7f98000 rw-p 7f2ff7f96000 00:00 0
7f2ff7fba000-7f2ff7fbd000 rw-p 7f2ff7fba000 00:00 0
7f2ff7fbd000-7f2ff7fbe000 r--p 0001f000 08:01 10751
/lib64/ld-2.9.so
7f2ff7fbe000-7f2ff7fbf000 rw-p 00020000 08:01 10751
/lib64/ld-2.9.so
7ffff3ca2000-7ffff3cb7000 rw-p 7ffffffea000 00:00 0
[stack]
7ffff3d2b000-7ffff3d2c000 r-xp 7ffff3d2b000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
I have now started strace (on the T105, AMD, 64 bit Slackware, etc.) from
the command line in a terminal and it seems to be holding up and the output
is going to a rapidly growing file.
I have the latest IBM JVM on the T110. I will switch this to production in
the morning while holding the above T105 in reserve in case this one goes
down.
Still looking for the exorcist but we now know that an older version of the
Sun JVM doesn't do the trick. Be interesting to see if the IBM JVM cures
the problem.
Thanks,
Carl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat dies suddenly
Posted by Carl <ca...@etrak-plus.com>.
Chuck,
Thanks, I will give it a try after I complete the two experiments currently
underway (the IBM JVM and running with strace from the command line.)
Tried bringing the server running the IBM JVM into production this morning
(tested it yesterday with different browsers... can't use IE 6 but all the
others seemed to work) but couldn't as there is some problem with SSL. The
indicators are that some people can access the application just fine but
others (they appear to be mostly IE users, will be getting the IE versions
later this morning) never get past the first page (I can see from the log
what is requested)... the first page starts a series of processes including
copying some signed jars down, if needed, starting an applet that serves as
a conduit for moving information to printers, etc. The odd part is all of
these worked fine yesterday when we were using the firewall to redirect
traffic on a specific port to this server (yesterday, we were sending
traffic on port 8084 to 443 on this server, this morning we were sending all
443 traffic to this server.) More testing this morning but all ideas are
certainly welcome.
Thanks,
Carl
----- Original Message -----
From: "Caldarale, Charles R" <Ch...@unisys.com>
To: "Tomcat Users List" <us...@tomcat.apache.org>
Sent: Tuesday, March 02, 2010 10:10 PM
Subject: RE: Tomcat dies suddenly
>> From: Carl [mailto:carl@etrak-plus.com]
>> Subject: Re: Tomcat dies suddenly
>>
>> Still looking for the exorcist but we now know that an older
>> version of the Sun JVM doesn't do the trick.
>
> Not necessarily. 6u10 was a major internal upgrade, so it might be
> interesting to try 6u7:
>
> http://java.sun.com/products/archive/j2se/6u7/index.html
>
> (There were no 6u8 or 6u9 releases.)
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail
> and its attachments from all computers.
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat dies suddenly
Posted by Carl <ca...@etrak-plus.com>.
The application has been running for 14 days without failing using the Sun
JVM 1.6.0_7 (per Chuck's suggestion... thanks Chuck.) However, we all know
that is just a work around.
To find the casue, my plan is to bring the latest JVM source code down and
build the JVM on the server with debugging information so that the core file
can tell us where it failed and the condition at failure. Is ther as easier
path?
Thanks,
Carl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat dies suddenly
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Carl [mailto:carl@etrak-plus.com]
> Subject: Re: Tomcat dies suddenly
>
> Still looking for the exorcist but we now know that an older
> version of the Sun JVM doesn't do the trick.
Not necessarily. 6u10 was a major internal upgrade, so it might be interesting to try 6u7:
http://java.sun.com/products/archive/j2se/6u7/index.html
(There were no 6u8 or 6u9 releases.)
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.