You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by keeplearning <p_...@yahoo.com> on 2008/11/01 18:02:09 UTC
Thread Dumps/ Emailing Results...
Hi All,
I am a load test engineer and pretty new to tomcat. I would really
appreciate if someone can help me with couple of my basic questions:
1) have seen some of my colleagues using kill -3 to take the thread dumps?
So, do i need to supply process id with this command. Like: kill -3 <process
id>. Then does the output goes to catalina.out which is our log file or do i
need to do anything else? And do i need to be in specific directory to run
this command?
2) once the thread dumps are in catalina.out, how do i mail the catalina.out
as an attachment? Should something like below work? As you know it's a huge
file
mail -s "Catalina.out" ps@ps.com < catalina.out
Thanks in Advance
Parki
--
View this message in context: http://www.nabble.com/Thread-Dumps--Emailing-Results...-tp20282144p20282144.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Thread Dumps/ Emailing Results...
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuck,
Caldarale, Charles R wrote:
>> From: keeplearning [mailto:p_sodhani@yahoo.com]
>> Subject: RE: Thread Dumps/ Emailing Results...
>>
>> 1) I tried using following to mail the file as attachment (we
>> are also using MS Outlook):
>>
>> mail -s "Subject:test" ps@ps.com < test.log
>
> That's using a Linux mailer, not Outlook; can't help you there. From
> Outlook, I just insert the file from the Linux box directly over an
> SMB connection.
If you want to actually attach a file to a message, regular
sendmail-style "mail" won't cut it: you have to use mutt:
mutt -s "${EMAIL_SUBJECT}" -a "${REPORT_FILE}" ${EMAIL_RECIPIENTS}
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkkQwswACgkQ9CaO5/Lv0PDqDQCfXDaRATaf1WJOjvIMLkIY63P3
HDMAoK146Q3QWW+vcqn/hY8q1sGA333h
=hgqH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Thread Dumps/ Emailing Results...
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: keeplearning [mailto:p_sodhani@yahoo.com]
> Subject: RE: Thread Dumps/ Emailing Results...
>
> I can get pid using ps -ef | grep tomcat.
Not likely. Tomcat is a Java program, so unless something on the command line includes the symbol "tomcat" (mine doesn't), that won't find it. Again, using jps is much easier - no grep required.
> What do i need to do restart tomcat?
I use the standard Tomcat scripts (startup.sh and shutdown.sh) to manipulate the Tomcat process, and I always use a real Tomcat from tomcat.apache.org, not some 3rd-party repackaged version. If you're using the latter, you're on your own.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Thread Dumps/ Emailing Results...
Posted by keeplearning <p_...@yahoo.com>.
OK. Actually i also use putty. So, may be i just need kill -3 pid. I can get
pid using ps -ef | grep tomcat.
One more question:
What do i need to do restart tomcat? Someone told me <service tomcat
restart> is enough?
Thanks
Caldarale, Charles R wrote:
>
>> From: keeplearning [mailto:p_sodhani@yahoo.com]
>> Subject: RE: Thread Dumps/ Emailing Results...
>>
>> 1) I tried using following to mail the file as attachment (we
>> are also using MS Outlook):
>>
>> mail -s "Subject:test" ps@ps.com < test.log
>
> That's using a Linux mailer, not Outlook; can't help you there. From
> Outlook, I just insert the file from the Linux box directly over an SMB
> connection.
>
>> 2) We used jstack before but it was not that useful.
>
> I'm curious: why do you find it "not that useful"? It's the same output
> as kill -3, or more if you use the -l option.
>
>> nexec <server name> kill -3 <process id>
>
> Sorry, I access our Linux boxes through putty from my Windows
> workstations; I've never used nexec.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail
> and its attachments from all computers.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Thread-Dumps--Emailing-Results...-tp20282144p20283227.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Thread Dumps/ Emailing Results...
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: keeplearning [mailto:p_sodhani@yahoo.com]
> Subject: RE: Thread Dumps/ Emailing Results...
>
> 1) I tried using following to mail the file as attachment (we
> are also using MS Outlook):
>
> mail -s "Subject:test" ps@ps.com < test.log
That's using a Linux mailer, not Outlook; can't help you there. From Outlook, I just insert the file from the Linux box directly over an SMB connection.
> 2) We used jstack before but it was not that useful.
I'm curious: why do you find it "not that useful"? It's the same output as kill -3, or more if you use the -l option.
> nexec <server name> kill -3 <process id>
Sorry, I access our Linux boxes through putty from my Windows workstations; I've never used nexec.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Thread Dumps/ Emailing Results...
Posted by keeplearning <p_...@yahoo.com>.
Thanks for your reply, Chuck.
couple of quick questions:
1) I tried using following to mail the file as attachment (we are also using
MS Outlook):
mail -s "Subject:test" ps@ps.com < test.log
But it sends the output in the body rather than as an attachment. I even
tried uuencode but still get in the body.
2) We used jstack before but it was not that useful. So, we returned to
using kill -3. Will following work?
nexec <server name> kill -3 <process id>
Do we need this full command even if i am logged in the server?
Thaks for your help
Parki
Caldarale, Charles R wrote:
>
>> From: keeplearning [mailto:p_sodhani@yahoo.com]
>> Subject: Thread Dumps/ Emailing Results...
>>
>> So, do i need to supply process id with this command. Like:
>> kill -3 <process id>.
>
> That's one way to do it. My preference is to use the jstack tool from the
> JDK, assuming you're running a 1.5 or 1.6 Sun JVM. You still need the
> process id, and the JDK includes the jps tool which is easier than wading
> through the output of ps. The JDK tools also work on Windows, whereas
> kill doesn't normally exist there.
>
>> Then does the output goes to catalina.out which is our
>> log file
>
> The output of kill -3 is sent to stdout or stderr (I forget which), which
> *may* be redirected to catalina.out, depending on your Tomcat
> configuration. The jstack output goes to your command window, rather than
> any of Tomcat's files.
>
>> And do i need to be in specific directory to run
>> this command?
>
> No, as long as the command you're using (kill or jstack) is in your PATH.
>
>> 2) once the thread dumps are in catalina.out, how do i mail
>> the catalina.out as an attachment?
>
> Edit it first to remove everything but the stack trace. (Another reason
> to use jstack.) How you mail it depends on your mail client and what
> additional information you need to provide. (I usually just insert the
> file as text, but then I'm using the corporate standard MS Outlook rather
> than any Linux mailer.) If you really need to make the whole log
> available, posting it on a web site somewhere may be preferable to dumping
> it in an e-mail.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail
> and its attachments from all computers.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Thread-Dumps--Emailing-Results...-tp20282144p20283039.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Thread Dumps/ Emailing Results...
Posted by Rainer Jung <ra...@kippdata.de>.
Christopher Schultz schrieb:
>> Also, in JDK 6 and above, there's another intriguing option:
>> -XX:OnOutOfMemoryError="<command>"
>
> That's pretty cool. For the time being, we're on JRE 1.5, so that's not
> an option. I'll look into it as we look forward to upgrading.
>
>> The above will run an external command to do whatever you deem
>> appropriate, such us running the thread dump script you provided.
>
> It would be great if that command could be parameterized and, say, take
> something like the thread dump on standard input. If you know anyone at
> Sun, feel free to give them that idea ;)
You can add white space separated arguments inside the flag value behind
the command name. It looks like oyu could also list several semicolon
separated commands. Any '%p' in the value will be automatically resolved
with the pid (which is part of the default heap file dump name). What
you can't get though, is the full heap dump file name:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6457610
Although usually you should be able to figure out, how to calculate it
based on the pid.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Thread Dumps/ Emailing Results...
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuck,
Caldarale, Charles R wrote:
>> From: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Subject: Re: Thread Dumps/ Emailing Results...
>>
>> I have written a script to check for OOMs by checking catalina.out
>> for "OutOfMemoryError". When such an error is detected, I run the
>> following commands (apologies for the line wrapping):
>
> An easier way to get the heap dump on a modern HotSpot JVM is to add the command-line option:
> -XX:+HeapDumpOnOutOfMemoryError
I noticed that, but we only run our OOM-check every 5 minutes, and we
want to avoid getting too much extra stuff in our email notifications.
If the thread dump happened a few minutes before, then we might not get
an up-to-date thread dump. One could argue that the exact timing of the
thread dump is not particularly relevant; merely having one /is/ useful.
Dumping threads at the time of the OOME would probably be better.
> Also, in JDK 6 and above, there's another intriguing option:
> -XX:OnOutOfMemoryError="<command>"
That's pretty cool. For the time being, we're on JRE 1.5, so that's not
an option. I'll look into it as we look forward to upgrading.
> The above will run an external command to do whatever you deem
> appropriate, such us running the thread dump script you provided.
It would be great if that command could be parameterized and, say, take
something like the thread dump on standard input. If you know anyone at
Sun, feel free to give them that idea ;)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkkQzUcACgkQ9CaO5/Lv0PADZACgqgA/aL47vYm50iF2g50VhQy4
40oAnRCS2tcjNiecOPsJEdlU/jgRlDlD
=75eD
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Thread Dumps/ Emailing Results...
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: Re: Thread Dumps/ Emailing Results...
>
> I have written a script to check for OOMs by checking catalina.out
> for "OutOfMemoryError". When such an error is detected, I run the
> following commands (apologies for the line wrapping):
An easier way to get the heap dump on a modern HotSpot JVM is to add the command-line option:
-XX:+HeapDumpOnOutOfMemoryError
Also, in JDK 6 and above, there's another intriguing option:
-XX:OnOutOfMemoryError="<command>"
The above will run an external command to do whatever you deem appropriate, such us running the thread dump script you provided.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Thread Dumps/ Emailing Results...
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuek,
Caldarale, Charles R wrote:
>> From: keeplearning [mailto:p_sodhani@yahoo.com]
>> Subject: Thread Dumps/ Emailing Results...
>>
>> So, do i need to supply process id with this command. Like:
>> kill -3 <process id>.
>
> That's one way to do it. My preference is to use the jstack tool from the JDK, assuming you're running a 1.5 or 1.6 Sun JVM.
I have written a script to check for OOMs by checking catalina.out for
"OutOfMemoryError". When such an error is detected, I run the following
commands (apologies for the line wrapping):
APP_PID=[fill this in with a way to get your app's pid]
# Make sure the pid looks like a pid
if [ "${APP_PID}" != `expr match "${APP_PID}" '\([0-9]\+\)'` ] ; then
echo "Could not determine the correct pid for ${APP_NAME}" >> ${OUTPUT}
else
# Make sure that this pid belongs to the proper process
ps -e x | grep "^[ \t]*${APP_PID}.*${PROCESS_STRING}" > /dev/null
if [ "$?" == "0" ] ; then
# This is the correct pid
if [ -x ${JSTACK} ] ; then
# jstack is available on the system: this is way easier
echo '=============== THREAD DUMP =================' >> ${OUTPUT}
${JSTACK} ${APP_PID} >> "${OUTPUT}" 2>&1
else
# Generate a thread dump on the java process
kill -QUIT ${APP_PID}
# Now, go find that thread dump in the log file
echo '=============== THREAD DUMP =================' >> ${OUTPUT}
dump_start_line=`grep -n '^Full thread' ${LOGFILE} | tail -n 1
| sed -e
's/\([0-9]\+\):.*/\1/'`
tail -n +${dump_start_line} ${LOGFILE} >> ${OUTPUT}
fi
# What the heck, let's get a heap dump, too.
echo '================ HEAP SUMMARY ===================' >> ${OUTPUT}
${JMAP} -heap ${APP_PID} >> ${OUTPUT} 2>&1
echo '================ OBJECT HISTOGRAM ===============' >> ${OUTPUT}
${JMAP} -histo ${APP_PID} >> ${OUTPUT} 2>&1
else
echo 'Could not determine the correct pid for ${APP_NAME}' >>
${OUTPUT}
fi
fi
mail -s "Found OOME in ${APP_NAME}!" ${ADMIN_EMAIL} < ${OUTPUT}
####
If jstack isn't available, we use a hack to generate a thread dump and
extract it from catalina.out.
> You still need the process id, and the JDK includes the jps tool
> which is easier than wading through the output of ps. The JDK tools
> also work on Windows, whereas kill doesn't normally exist there.
Yes, this is a UNIX-specific script, unless you want to run it on Cygwin
(untested, by the way). But, if you're running on Windows, you don't
really care about uptime, anyway, do you? ;)
Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEUEARECAAYFAkkQwkMACgkQ9CaO5/Lv0PB2vgCYvDQJNbkkVEFcpL/QnEegGg6x
iQCgmpuTbVPR6THyGbXE0U8GTLEK/No=
=Kg1V
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Thread Dumps/ Emailing Results...
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: keeplearning [mailto:p_sodhani@yahoo.com]
> Subject: Thread Dumps/ Emailing Results...
>
> So, do i need to supply process id with this command. Like:
> kill -3 <process id>.
That's one way to do it. My preference is to use the jstack tool from the JDK, assuming you're running a 1.5 or 1.6 Sun JVM. You still need the process id, and the JDK includes the jps tool which is easier than wading through the output of ps. The JDK tools also work on Windows, whereas kill doesn't normally exist there.
> Then does the output goes to catalina.out which is our
> log file
The output of kill -3 is sent to stdout or stderr (I forget which), which *may* be redirected to catalina.out, depending on your Tomcat configuration. The jstack output goes to your command window, rather than any of Tomcat's files.
> And do i need to be in specific directory to run
> this command?
No, as long as the command you're using (kill or jstack) is in your PATH.
> 2) once the thread dumps are in catalina.out, how do i mail
> the catalina.out as an attachment?
Edit it first to remove everything but the stack trace. (Another reason to use jstack.) How you mail it depends on your mail client and what additional information you need to provide. (I usually just insert the file as text, but then I'm using the corporate standard MS Outlook rather than any Linux mailer.) If you really need to make the whole log available, posting it on a web site somewhere may be preferable to dumping it in an e-mail.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org