You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by sl...@apache.org on 2006/07/22 21:46:17 UTC
svn commit: r424627 - in /httpd/site/trunk: docs/dev/debugging.html
xdocs/dev/debugging.xml
Author: slive
Date: Sat Jul 22 12:46:17 2006
New Revision: 424627
URL: http://svn.apache.org/viewvc?rev=424627&view=rev
Log:
Some advice from wrowe on live debugging under windows.
Plus fix the overlong <pre> sections to allow proper wrapping.
Modified:
httpd/site/trunk/docs/dev/debugging.html
httpd/site/trunk/xdocs/dev/debugging.xml
Modified: httpd/site/trunk/docs/dev/debugging.html
URL: http://svn.apache.org/viewvc/httpd/site/trunk/docs/dev/debugging.html?rev=424627&r1=424626&r2=424627&view=diff
==============================================================================
--- httpd/site/trunk/docs/dev/debugging.html (original)
+++ httpd/site/trunk/docs/dev/debugging.html Sat Jul 22 12:46:17 2006
@@ -69,7 +69,8 @@
<a href="mailto:docs@httpd.apache.org">docs@httpd.apache.org</a>. Thanks!</p>
<ol>
<li><a href="#gdb">Using <code>gdb</code></a></li>
-<li><a href="#backtrace">Getting a live backtrace</a></li>
+<li><a href="#backtrace">Getting a live backtrace on unix</a></li>
+<li><a href="#backtrace-win">Getting a live backtrace on Windows</a></li>
<li><a href="#crashes">Debugging intermittent crashes</a></li>
<li><a href="#truss">Using '<code>truss/trace/strace</code>' to
trace system calls and signals</a></li>
@@ -124,7 +125,7 @@
Breakpoint 1, ap_process_request (r=0x95250) at http_request.c:1164
1164 if (ap_extended_status)
(gdb) <font color="green">s</font>
- 1165 ap_time_process_request(r->connection->child_num, START_PREQUEST);
+ 1165 ap_time_process_request(r->connection->child_num, ...
(gdb) <font color="green">n</font>
1167 process_request_internal(r);
(gdb) <font color="green">s</font>
@@ -135,7 +136,7 @@
(gdb) <font color="green">n</font>
1030 if (access_status) {
(gdb) <font color="green">s</font>
- 1036 ap_getparents(r->uri); /* OK --- shrinking transformations... */
+ 1036 ap_getparents(r->uri); /* OK ...
(gdb) <font color="green">n</font>
1038 if ((access_status = location_walk(r))) {
(gdb) <font color="green">n</font>
@@ -145,12 +146,12 @@
(gdb) <font color="green">n</font>
1053 if (r->method_number == M_TRACE) {
(gdb) <font color="green">n</font>
- 1062 if (r->proto_num > HTTP_VERSION(1,0) && ap_table_get(r->subprocess_env, "downgrade-1.0")) {
+ 1062 if (r->proto_num > HTTP_VERSION(1,0) && ap_ ...
(gdb) <font color="green">n</font>
1071 if ((access_status = directory_walk(r))) {
(gdb) <font color="green">s</font>
directory_walk (r=0x95250) at http_request.c:288
- 288 core_server_config *sconf = ap_get_module_config(r->server->module_config,
+ 288 core_server_config *sconf = ap_get_module_ ...
(gdb) <font color="green">b ap_send_error_response</font>
Breakpoint 2 at 0x47dcc: file http_protocol.c, line 2090.
(gdb) <font color="green">c</font>
@@ -221,7 +222,7 @@
<table border="0" cellspacing="0" cellpadding="2" width="100%">
<tr><td bgcolor="#525D76">
<font color="#ffffff" face="arial,helvetica,sanserif">
- <a name="backtrace"><strong>Getting a live backtrace</strong></a>
+ <a name="backtrace"><strong>Getting a live backtrace on unix</strong></a>
</font>
</td></tr>
<tr><td>
@@ -267,7 +268,7 @@
#0 0xef5b68d8 in ()
#1 0xef5d21e8 in select ()
#2 0x4257c in wait_or_timeout (status=0x0) at http_main.c:2357
- #3 0x44318 in standalone_main (argc=392552, argv=0x75800) at http_main.c:4273
+ #3 0x44318 in standalone_main (argc=392552, argv=0x75800) at ...
#4 0x449fc in main (argc=3, argv=0xefffeee4) at http_main.c:4534
(gdb)
</pre>
@@ -277,6 +278,67 @@
<table border="0" cellspacing="0" cellpadding="2" width="100%">
<tr><td bgcolor="#525D76">
<font color="#ffffff" face="arial,helvetica,sanserif">
+ <a name="backtrace-win"><strong>Getting a live backtrace on Windows</strong></a>
+ </font>
+ </td></tr>
+ <tr><td>
+ <blockquote>
+<ol>
+<li> Unzip the <code>-symbols.zip</code> files (obtained from the
+ Apache download site) in the root Apache2 directory tree (where
+ bin\, htdocs\, modules\ etc. are usually found.) These .pdb files
+ should unpack alongside the .exe, .dll, .so binary files they
+ represent, e.g., mod_usertrack.pdb will unpack alongside
+ mod_usertrack.so.</li>
+
+<li> Invoke <code>drwtsn32</code> and ensure you are creating a crash
+ dump file, you are dumping all thread contexts, your log and crash
+ dump paths make sense, and (depending on the nature of the bug) you
+ pick an appropriate crash dump type. (Full is quite large, but
+ necessary sometimes for a programmer-type to load your crash dump
+ into a debugger and begin unwinding exactly what has happened.
+ Mini is sufficient for your first pass through the process.)</li>
+
+<li> Note that if you previously installed and then uninstalled other debugging
+ software, you may need to invoke <code>drwtsn32 -i</code> in order to make Dr Watson
+ your default crash dump tool. This will replace the 'report problem to MS'
+ dialogs. Don't do this if you have a full debugger such as Visual Studio
+ or windbg installed on the machine.</li>
+
+<li> Invoke the Task Manager, Choose 'show processes from all users',
+ and modify the <code>View -> Select Columns</code> to include
+ <strong>at least</strong> the <code>PID</code> and <code>Thread
+ Count</code>. You can change this just once and Task Manager
+ should keep your preference.</li>
+
+<li> Now, track down the errant Apache that is hanging. The parent
+ process has about three threads, we don't care about that one. The
+ child worker process we want has many more threads (a few more than
+ you configured with the ThreadsPerChild directive.) The process
+ name is Apache (for 1.3 and 2.0) or httpd (for 2.2). Make note of
+ the child worker's PID.</li>
+
+<li> Using the {pid} number you noted above, invoke the command
+ <blockquote><code>drwtsn32 -p {pid}</code></blockquote></li>
+</ol>
+<p>Voila, you will find in your 'log file path' a
+<code>drwtsn32.log</code> file, and if you choose to 'append to
+existing log file', jump through the 'App:' sections until you find
+the one for the process you just killed. Now you can identify
+about where 'Stack Back Trace' points to help identify what the server
+is doing.</p>
+<p>You will note that many threads look identical, almost all of them
+polling for the next connection, and you don't care about those. You will
+want to see the ones that are deep inside of a request at the time you
+kill them, and only the stack back trace entries for those. This can
+give folks a clue of where that request is hanging, which handler
+module picked up the request, and what filter it might be stuck in.</p>
+ </blockquote>
+ </td></tr>
+</table>
+ <table border="0" cellspacing="0" cellpadding="2" width="100%">
+ <tr><td bgcolor="#525D76">
+ <font color="#ffffff" face="arial,helvetica,sanserif">
<a name="crashes"><strong>Debugging intermittent crashes</strong></a>
</font>
</td></tr>
@@ -302,9 +364,9 @@
dumps - see <a href="#sol27">Solaris and coredumps</a> below.</p>
<p>When a child process crashes, a message like the following will be
logged to the error_log:</p>
-<pre>
+<blockquote><code>
[Mon Sep 05 13:35:39 2005] [notice] child pid 2027 exit signal Segmentation fault (11), possible coredump in /tmp
-</pre>
+</code></blockquote>
<p>If the text "possible coredump in /tmp" does not appear in the
error line, check that the ulimit was set correctly, that the
permissions on the configured <code>CoreDumpDirectory</code> are
Modified: httpd/site/trunk/xdocs/dev/debugging.xml
URL: http://svn.apache.org/viewvc/httpd/site/trunk/xdocs/dev/debugging.xml?rev=424627&r1=424626&r2=424627&view=diff
==============================================================================
--- httpd/site/trunk/xdocs/dev/debugging.xml (original)
+++ httpd/site/trunk/xdocs/dev/debugging.xml Sat Jul 22 12:46:17 2006
@@ -17,7 +17,8 @@
<ol>
<li><a href="#gdb">Using <code>gdb</code></a></li>
-<li><a href="#backtrace">Getting a live backtrace</a></li>
+<li><a href="#backtrace">Getting a live backtrace on unix</a></li>
+<li><a href="#backtrace-win">Getting a live backtrace on Windows</a></li>
<li><a href="#crashes">Debugging intermittent crashes</a></li>
<li><a href="#truss">Using '<code>truss/trace/strace</code>' to
trace system calls and signals</a></li>
@@ -69,7 +70,7 @@
Breakpoint 1, ap_process_request (r=0x95250) at http_request.c:1164
1164 if (ap_extended_status)
(gdb) <font color="green">s</font>
- 1165 ap_time_process_request(r->connection->child_num, START_PREQUEST);
+ 1165 ap_time_process_request(r->connection->child_num, ...
(gdb) <font color="green">n</font>
1167 process_request_internal(r);
(gdb) <font color="green">s</font>
@@ -80,7 +81,7 @@
(gdb) <font color="green">n</font>
1030 if (access_status) {
(gdb) <font color="green">s</font>
- 1036 ap_getparents(r->uri); /* OK --- shrinking transformations... */
+ 1036 ap_getparents(r->uri); /* OK ...
(gdb) <font color="green">n</font>
1038 if ((access_status = location_walk(r))) {
(gdb) <font color="green">n</font>
@@ -90,12 +91,12 @@
(gdb) <font color="green">n</font>
1053 if (r->method_number == M_TRACE) {
(gdb) <font color="green">n</font>
- 1062 if (r->proto_num > HTTP_VERSION(1,0) && ap_table_get(r->subprocess_env, "downgrade-1.0")) {
+ 1062 if (r->proto_num > HTTP_VERSION(1,0) && ap_ ...
(gdb) <font color="green">n</font>
1071 if ((access_status = directory_walk(r))) {
(gdb) <font color="green">s</font>
directory_walk (r=0x95250) at http_request.c:288
- 288 core_server_config *sconf = ap_get_module_config(r->server->module_config,
+ 288 core_server_config *sconf = ap_get_module_ ...
(gdb) <font color="green">b ap_send_error_response</font>
Breakpoint 2 at 0x47dcc: file http_protocol.c, line 2090.
(gdb) <font color="green">c</font>
@@ -171,7 +172,7 @@
</section>
<section id="backtrace">
-<title>Getting a live backtrace</title>
+<title>Getting a live backtrace on unix</title>
<p>A backtrace will let you know the hierarchy of procedures that
were called to get to a particular point in the process. On some platforms
@@ -218,12 +219,70 @@
#0 0xef5b68d8 in ()
#1 0xef5d21e8 in select ()
#2 0x4257c in wait_or_timeout (status=0x0) at http_main.c:2357
- #3 0x44318 in standalone_main (argc=392552, argv=0x75800) at http_main.c:4273
+ #3 0x44318 in standalone_main (argc=392552, argv=0x75800) at ...
#4 0x449fc in main (argc=3, argv=0xefffeee4) at http_main.c:4534
(gdb)
</pre>
</section>
+<section id="backtrace-win">
+<title>Getting a live backtrace on Windows</title>
+
+<ol>
+<li> Unzip the <code>-symbols.zip</code> files (obtained from the
+ Apache download site) in the root Apache2 directory tree (where
+ bin\, htdocs\, modules\ etc. are usually found.) These .pdb files
+ should unpack alongside the .exe, .dll, .so binary files they
+ represent, e.g., mod_usertrack.pdb will unpack alongside
+ mod_usertrack.so.</li>
+
+<li> Invoke <code>drwtsn32</code> and ensure you are creating a crash
+ dump file, you are dumping all thread contexts, your log and crash
+ dump paths make sense, and (depending on the nature of the bug) you
+ pick an appropriate crash dump type. (Full is quite large, but
+ necessary sometimes for a programmer-type to load your crash dump
+ into a debugger and begin unwinding exactly what has happened.
+ Mini is sufficient for your first pass through the process.)</li>
+
+<li> Note that if you previously installed and then uninstalled other debugging
+ software, you may need to invoke <code>drwtsn32 -i</code> in order to make Dr Watson
+ your default crash dump tool. This will replace the 'report problem to MS'
+ dialogs. Don't do this if you have a full debugger such as Visual Studio
+ or windbg installed on the machine.</li>
+
+<li> Invoke the Task Manager, Choose 'show processes from all users',
+ and modify the <code>View -> Select Columns</code> to include
+ <strong>at least</strong> the <code>PID</code> and <code>Thread
+ Count</code>. You can change this just once and Task Manager
+ should keep your preference.</li>
+
+<li> Now, track down the errant Apache that is hanging. The parent
+ process has about three threads, we don't care about that one. The
+ child worker process we want has many more threads (a few more than
+ you configured with the ThreadsPerChild directive.) The process
+ name is Apache (for 1.3 and 2.0) or httpd (for 2.2). Make note of
+ the child worker's PID.</li>
+
+<li> Using the {pid} number you noted above, invoke the command
+ <blockquote><code>drwtsn32 -p {pid}</code></blockquote></li>
+</ol>
+
+<p>Voila, you will find in your 'log file path' a
+<code>drwtsn32.log</code> file, and if you choose to 'append to
+existing log file', jump through the 'App:' sections until you find
+the one for the process you just killed. Now you can identify
+about where 'Stack Back Trace' points to help identify what the server
+is doing.</p>
+
+<p>You will note that many threads look identical, almost all of them
+polling for the next connection, and you don't care about those. You will
+want to see the ones that are deep inside of a request at the time you
+kill them, and only the stack back trace entries for those. This can
+give folks a clue of where that request is hanging, which handler
+module picked up the request, and what filter it might be stuck in.</p>
+</section>
+
+
<section id="crashes">
<title>Debugging intermittent crashes</title>
@@ -255,9 +314,9 @@
<p>When a child process crashes, a message like the following will be
logged to the error_log:</p>
-<pre>
+<blockquote><code>
[Mon Sep 05 13:35:39 2005] [notice] child pid 2027 exit signal Segmentation fault (11), possible coredump in /tmp
-</pre>
+</code></blockquote>
<p>If the text "possible coredump in /tmp" does not appear in the
error line, check that the ulimit was set correctly, that the