You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by rp...@apache.org on 2005/10/19 15:45:50 UTC
svn commit: r326560 - in /httpd/site/trunk: docs/dev/debugging.html
xdocs/dev/debugging.xml
Author: rpluem
Date: Wed Oct 19 06:45:43 2005
New Revision: 326560
URL: http://svn.apache.org/viewcvs?rev=326560&view=rev
Log:
* Updated various sections:
* using gdb
* Added hint for further macros
* Added location of .gdbinit in 2.x
* Debugging intermittent crashes
* Stress out the necessity of platform specific tasks on Solaris
* Using 'truss/trace/strace' to trace system calls and signals
* -f works also with strace
* Give a hint to truss option -l for multithreaded servers
* Add further useful truss options
* Solaris and coredumps
* renamed section
* Added further explanations
* Getting and analyzing a TCP packet trace
* Updated / removed outdated / dead links
* Added links and hints to further tools (tcpdump, windump, ethereal,
snoop)
Modified:
httpd/site/trunk/docs/dev/debugging.html
httpd/site/trunk/xdocs/dev/debugging.xml
Modified: httpd/site/trunk/docs/dev/debugging.html
URL: http://svn.apache.org/viewcvs/httpd/site/trunk/docs/dev/debugging.html?rev=326560&r1=326559&r2=326560&view=diff
==============================================================================
--- httpd/site/trunk/docs/dev/debugging.html (original)
+++ httpd/site/trunk/docs/dev/debugging.html Wed Oct 19 06:45:43 2005
@@ -77,7 +77,7 @@
<li><a href="#truss">Using '<code>truss/trace/strace</code>' to
trace system calls and signals</a></li>
<li><a href="#gcore">Getting the server to dump core</a></li>
-<li><a href="#sol27">Solaris 2.7 and coredumps</a></li>
+<li><a href="#sol27">Solaris and coredumps</a></li>
<li><a href="#tcpdump">Getting and analyzing a TCP packet trace</a></li>
</ol>
</blockquote>
@@ -199,9 +199,11 @@
parameter values.</li>
<li>use the "<code>p</code>" command to print the value of a variable.</li>
</ol>
-<p>A file in the <code>src/</code> directory, <code>.gdbinit</code>,
-provides a useful macro for printing out the contents of a table
-structure, called <code>dump_table</code>.</p>
+<p>A file in the <code>src/</code> directory (1.3) or the root directory
+(2.x) called <code>.gdbinit</code> provides useful macros for printing
+out various internal structures of httpd like tables (<code>dump_table</code>),
+brigades (<code>dump_brigade</code>) and filter chains
+(<code>dump_filters</code>).</p>
<p>If you are debugging a repeatable crash, simply run gdb as above
and make the request -- gdb should capture the crash and provide a
prompt where it occurs.</p>
@@ -285,7 +287,7 @@
<blockquote>
<p>For situations where a child process is crashing intermittently,
the server must be configured and started such that it produces core
-dumps which can analyzed further.</p>
+dumps which can be analyzed further.</p>
<p>To ensure that a core dump is written to a directory which is
writable by the user which child processes run as (such as
<code>apache</code>), the <a href="http://httpd.apache.org/docs/2.0/mod/mpm_common.html#coredumpdirectory"><code>CoreDumpDirectory</code></a>
@@ -300,16 +302,17 @@
# apachectl start
</pre>
<p>On some platforms, further steps might be needed to enable core
-dumps - see <a href="#sol27">Solaris 2.7 and coredumps</a> below.</p>
+dumps - see <a href="#sol27">Solaris and coredumps</a> below.</p>
<p>When a child process crashes, a message like the following will be
logged to the error_log:</p>
<pre>
[Mon Sep 05 13:35:39 2005] [notice] child pid 2027 exit signal Segmentation fault (11), possible coredump in /tmp
</pre>
<p>If the text "possible coredump in /tmp" does not appear in the
-error line, check that the ulimit was set correctly and that the
+error line, check that the ulimit was set correctly, that the
permissions on the configured <code>CoreDumpDirectory</code> are
-suitable.</p>
+suitable and that platform specific steps
+(<a href="#sol27">Solaris and coredumps</a>) have been done if needed.</p>
<p>To analyse the core dump, pass the core dump filename on the gdb
command-line, and enter the command <code>bt full</code> at the gdb
prompt:</p>
@@ -343,9 +346,10 @@
systems and either <code>trace</code> or <code>strace</code> on many
other systems.</p>
<p>A useful tip for using the <code>truss</code> command on Solaris is
-the <code>-f</code> option; it tells truss to follow and continue tracing
-any child processes forked by the main process. The easiest way to get
-a full trace of a server is to do something like:</p>
+the <code>-f</code> option (often also works with <code>strace</code>); it tells
+truss to follow and continue tracing any child processes forked by the main
+process. The easiest way to get a full trace of a server is to do something
+like:</p>
<pre>
% <font color="green">truss -f httpd -d /usr/local/apache >& outfile</font>
</pre>
@@ -353,6 +357,26 @@
% <font color="green">egrep '^10698:' outfile</font>
</pre>
<p>to view just the trace of the process id 10698.</p>
+<p>If attempting to truss a threaded server, for example when using
+the <code>worker</code> MPM, the <code>truss</code> option <code>-l</code>
+is very useful as it prints also the LWP id after the process id. You can use
+something like</p>
+<pre>
+ % <font color="green">egrep '^10698/1:' outfile</font>
+</pre>
+<p>to view just the trace of the process id 10698 and LWP id 1.</p>
+<p>Other useful options for truss are</p>
+<ul>
+<li>
+<code>-a</code> to print all command line parameters used for this executable.
+</li>
+<li>
+<code>-e</code> to print all environment variables used for this executable.
+</li>
+<li>
+<code>-d</code> to print timestamps.
+</li>
+</ul>
</blockquote>
</td></tr>
</table>
@@ -395,14 +419,22 @@
<table border="0" cellspacing="0" cellpadding="2" width="100%">
<tr><td bgcolor="#525D76">
<font color="#ffffff" face="arial,helvetica,sanserif">
- <a name="sol27"><strong>Solaris 2.7 and coredumps</strong></a>
+ <a name="sol27"><strong>Solaris and coredumps</strong></a>
</font>
</td></tr>
<tr><td>
<blockquote>
-<p>On Solaris 2.7 use <b><code>coreadm</code></b> to make <code>setuid()</code>
-processes actually dump core. By default an setuid() process does not
-dump core.</p>
+<p>On Solaris (at least 2.7 and up) use <b><code>coreadm</code></b> to make
+<code>setuid()</code> processes actually dump core. By default a setuid()
+process does not dump core. This is the reason why httpd servers started as
+root with child processes running as a different user (such as
+<code>apache</code>) do not coredump even when the
+<a href="http://httpd.apache.org/docs/2.0/mod/mpm_common.html#coredumpdirectory">
+<code>CoreDumpDirectory</code></a>
+directive had been set to an appropriate and writable directory and
+<b><code>ulimit -c</code></b> has a sufficient size. See also
+<a href="#crashes">Debugging intermittent crashes</a> above.
+</p>
<p>Jens-Uwe Mager wrote:</p>
<blockquote>
<p><i>For example I am using:</i></p>
@@ -431,16 +463,30 @@
<p>This is more difficult than I have time to describe at the moment.
Here are some pointers to useful discussions and tools:</p>
<ul>
-<li><a href="http://jarok.cs.ohiou.edu/software/tcptrace/pcap.html">tools for
- producing TCP dumps</a></li>
-<li><a href="http://jarok.cs.ohiou.edu/software/tcptrace/tcptrace.html">
- tcptrace</a> is a TCP dump file analysis tool</li>
-<li><a href="http://HTTP.CS.Berkeley.EDU/~daw/mike/">tcpshow</a> is
- another</li>
+<li>
+snoop is a packet sniffer that is part of Solaris.
+</li>
+<li>
+<a href="http://www.tcpdump.org/">tcpdump</a> is a packet sniffer that is
+available for Unix-based systems and Windows
+(<a href="http://www.winpcap.org/windump/">windump</a>).
+It is part of many free Unix-based distributions.
+</li>
+<li>
+<a href="http://www.ethereal.com/">ethereal</a> is another packet sniffer
+that is available for Unix-based systems and Windows. It has a nice GUI and allows the
+analysis of the sniffed data.
+</li>
+<li><a href="http://jarok.cs.ohiou.edu/software/tcptrace/">
+ tcptrace</a> is a TCP dump file analysis tool.</li>
+<li><a href="http://www.tcpshow.org/">tcpshow</a> is
+ another one.</li>
</ul>
<p>There is also a simple ASCII viewer for TCP dump traces in the Apache
repository in the file
-<code><a href="http://cvs.apache.org/viewcvs.cgi/apache-1.3/src/test/tcpdumpscii.txt">src/test/tcpdumpscii.txt</a></code>.</p>
+<code>
+<a href="http://cvs.apache.org/viewcvs.cgi/apache-1.3/src/test/tcpdumpscii.txt">
+src/test/tcpdumpscii.txt</a></code>.</p>
</blockquote>
</td></tr>
</table>
Modified: httpd/site/trunk/xdocs/dev/debugging.xml
URL: http://svn.apache.org/viewcvs/httpd/site/trunk/xdocs/dev/debugging.xml?rev=326560&r1=326559&r2=326560&view=diff
==============================================================================
--- httpd/site/trunk/xdocs/dev/debugging.xml (original)
+++ httpd/site/trunk/xdocs/dev/debugging.xml Wed Oct 19 06:45:43 2005
@@ -22,7 +22,7 @@
<li><a href="#truss">Using '<code>truss/trace/strace</code>' to
trace system calls and signals</a></li>
<li><a href="#gcore">Getting the server to dump core</a></li>
-<li><a href="#sol27">Solaris 2.7 and coredumps</a></li>
+<li><a href="#sol27">Solaris and coredumps</a></li>
<li><a href="#tcpdump">Getting and analyzing a TCP packet trace</a></li>
</ol>
</section>
@@ -144,9 +144,11 @@
<li>use the "<code>p</code>" command to print the value of a variable.</li>
</ol>
-<p>A file in the <code>src/</code> directory, <code>.gdbinit</code>,
-provides a useful macro for printing out the contents of a table
-structure, called <code>dump_table</code>.</p>
+<p>A file in the <code>src/</code> directory (1.3) or the root directory
+(2.x) called <code>.gdbinit</code> provides useful macros for printing
+out various internal structures of httpd like tables (<code>dump_table</code>),
+brigades (<code>dump_brigade</code>) and filter chains
+(<code>dump_filters</code>).</p>
<p>If you are debugging a repeatable crash, simply run gdb as above
and make the request -- gdb should capture the crash and provide a
@@ -227,7 +229,7 @@
<p>For situations where a child process is crashing intermittently,
the server must be configured and started such that it produces core
-dumps which can analyzed further.</p>
+dumps which can be analyzed further.</p>
<p>To ensure that a core dump is written to a directory which is
writable by the user which child processes run as (such as
@@ -248,7 +250,7 @@
</pre>
<p>On some platforms, further steps might be needed to enable core
-dumps - see <a href="#sol27">Solaris 2.7 and coredumps</a> below.</p>
+dumps - see <a href="#sol27">Solaris and coredumps</a> below.</p>
<p>When a child process crashes, a message like the following will be
logged to the error_log:</p>
@@ -258,9 +260,10 @@
</pre>
<p>If the text "possible coredump in /tmp" does not appear in the
-error line, check that the ulimit was set correctly and that the
+error line, check that the ulimit was set correctly, that the
permissions on the configured <code>CoreDumpDirectory</code> are
-suitable.</p>
+suitable and that platform specific steps
+(<a href="#sol27">Solaris and coredumps</a>) have been done if needed.</p>
<p>To analyse the core dump, pass the core dump filename on the gdb
command-line, and enter the command <code>bt full</code> at the gdb
@@ -294,20 +297,47 @@
other systems.</p>
<p>A useful tip for using the <code>truss</code> command on Solaris is
-the <code>-f</code> option; it tells truss to follow and continue tracing
-any child processes forked by the main process. The easiest way to get
-a full trace of a server is to do something like:</p>
+the <code>-f</code> option (often also works with <code>strace</code>); it tells
+truss to follow and continue tracing any child processes forked by the main
+process. The easiest way to get a full trace of a server is to do something
+like:</p>
<pre>
% <font color="green">truss -f httpd -d /usr/local/apache >& outfile</font>
</pre>
-and let it run through a few requests before killing the parent. You can
+and let it run through a few requests before killing the parent. You can
then view the entire trace in outfile, or use something like
<pre>
% <font color="green">egrep '^10698:' outfile</font>
</pre>
<p>to view just the trace of the process id 10698.</p>
+
+<p>If attempting to truss a threaded server, for example when using
+the <code>worker</code> MPM, the <code>truss</code> option <code>-l</code>
+is very useful as it prints also the LWP id after the process id. You can use
+something like</p>
+
+<pre>
+ % <font color="green">egrep '^10698/1:' outfile</font>
+</pre>
+
+<p>to view just the trace of the process id 10698 and LWP id 1.</p>
+
+<p>Other useful options for truss are</p>
+
+<ul>
+<li>
+<code>-a</code> to print all command line parameters used for this executable.
+</li>
+<li>
+<code>-e</code> to print all environment variables used for this executable.
+</li>
+<li>
+<code>-d</code> to print timestamps.
+</li>
+</ul>
+
</section>
<section id="gcore">
@@ -346,11 +376,19 @@
</section>
<section id="sol27">
-<title>Solaris 2.7 and coredumps</title>
+<title>Solaris and coredumps</title>
-<p>On Solaris 2.7 use <b><code>coreadm</code></b> to make <code>setuid()</code>
-processes actually dump core. By default an setuid() process does not
-dump core.</p>
+<p>On Solaris (at least 2.7 and up) use <b><code>coreadm</code></b> to make
+<code>setuid()</code> processes actually dump core. By default a setuid()
+process does not dump core. This is the reason why httpd servers started as
+root with child processes running as a different user (such as
+<code>apache</code>) do not coredump even when the
+<a href="http://httpd.apache.org/docs/2.0/mod/mpm_common.html#coredumpdirectory">
+<code>CoreDumpDirectory</code></a>
+directive had been set to an appropriate and writable directory and
+<b><code>ulimit -c</code></b> has a sufficient size. See also
+<a href="#crashes">Debugging intermittent crashes</a> above.
+</p>
<p>Jens-Uwe Mager wrote:</p>
<blockquote>
@@ -375,17 +413,31 @@
Here are some pointers to useful discussions and tools:</p>
<ul>
-<li><a href="http://jarok.cs.ohiou.edu/software/tcptrace/pcap.html">tools for
- producing TCP dumps</a></li>
-<li><a href="http://jarok.cs.ohiou.edu/software/tcptrace/tcptrace.html">
- tcptrace</a> is a TCP dump file analysis tool</li>
-<li><a href="http://HTTP.CS.Berkeley.EDU/~daw/mike/">tcpshow</a> is
- another</li>
+<li>
+snoop is a packet sniffer that is part of Solaris.
+</li>
+<li>
+<a href="http://www.tcpdump.org/">tcpdump</a> is a packet sniffer that is
+available for Unix-based systems and Windows
+(<a href="http://www.winpcap.org/windump/">windump</a>).
+It is part of many free Unix-based distributions.
+</li>
+<li>
+<a href="http://www.ethereal.com/">ethereal</a> is another packet sniffer
+that is available for Unix-based systems and Windows. It has a nice GUI and allows the
+analysis of the sniffed data.
+</li>
+<li><a href="http://jarok.cs.ohiou.edu/software/tcptrace/">
+ tcptrace</a> is a TCP dump file analysis tool.</li>
+<li><a href="http://www.tcpshow.org/">tcpshow</a> is
+ another one.</li>
</ul>
<p>There is also a simple ASCII viewer for TCP dump traces in the Apache
repository in the file
-<code><a href="http://cvs.apache.org/viewcvs.cgi/apache-1.3/src/test/tcpdumpscii.txt">src/test/tcpdumpscii.txt</a></code>.</p>
+<code>
+<a href="http://cvs.apache.org/viewcvs.cgi/apache-1.3/src/test/tcpdumpscii.txt">
+src/test/tcpdumpscii.txt</a></code>.</p>
</section>
</body>