You are viewing a plain text version of this content. The canonical link for it is here.
Posted to docs@httpd.apache.org by Joe Orton <jo...@redhat.com> on 2005/09/05 14:53:12 UTC
[PATCH] site: more debugging tips
After getting bored with writing these into bugzilla for the umpteenth
time, does this look OK?
Index: xdocs/dev/debugging.xml
===================================================================
--- xdocs/dev/debugging.xml (revision 278729)
+++ xdocs/dev/debugging.xml (working copy)
@@ -18,6 +18,7 @@
<ol>
<li><a href="#gdb">Using <code>gdb</code></a></li>
<li><a href="#backtrace">Getting a live backtrace</a></li>
+<li><a href="#crashes">Debugging intermittent crashes</a></li>
<li><a href="#truss">Using '<code>truss/trace/strace</code>' to
trace system calls and signals</a></li>
<li><a href="#gcore">Getting the server to dump core</a></li>
@@ -221,6 +222,70 @@
</pre>
</section>
+<section id="crashes">
+<title>Debugging intermittent crashes</title>
+
+<p>For situations where a child process is crashing intermittently,
+the server must be configured and started such that it produces core
+dumps which can analyzed later.</p>
+
+<p>To ensure that a core dump can be written to a directory which is
+writable by the user which child processes run as (such as
+<code>apache</code>), the <a
+href="http://httpd.apache.org/docs/2.0/mod/mpm_common.html#coredumpdirectory"><code>CoreDumpDirectory</code></a>
+directive must be added to <code>httpd.conf</code>; for example:</p>
+
+<pre>
+ CoreDumpDirectory /tmp
+</pre>
+
+<p>Before starting up the server, any process limits on core dump file
+size must be lifted; for example:</p>
+
+<pre>
+ # ulimit -c unlimited
+ # apachectl start
+</pre>
+
+<p>On some platforms, further steps might be needed to enable core
+dumps; see <a href="#sol27">Solaris 2.7 and coredumps</a> below.</p>
+
+<p>When a child process crashes, a message will be logged to the
+error_log like the following:</p>
+
+<pre>
+[Mon Sep 05 13:35:39 2005] [notice] child pid 2027 exit signal Segmentation fault (11), possible coredump in /tmp
+</pre>
+
+<p>if the text "possible coredump in /tmp" does not appear in the
+error line, check that the ulimit was set correctly and that the
+permissions on the configured <code>CoreDumpDirectory</code> are
+suitable.</p>
+
+<p>To analyse the core dump, pass the core dump filename on the gdb
+command-line, and enter the command <code>bt full</code> at the gdb
+prompt:</p>
+
+<pre>
+ % <b>gdb /usr/local/apache2/bin/httpd /tmp/core.2027</b>
+ ...
+ Core was generated by `/usr/local/apache2/bin/httpd -k start'
+ ...
+ (gdb) <b>bt full</b>
+</pre>
+
+<p>if attempting to debug a threaded server, such as using the
+<code>worker</code> MPM, use the following gdb command:</p>
+
+<pre>
+ (gdb) <b>thread apply all bt full</b>
+</pre>
+
+<p>this will produce a backtrace for each of the threads within the
+process.</p>
+
+</section>
+
<section id="truss">
<title>Using 'truss/trace/strace' to trace system calls and
signals</title>
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: [PATCH] site: more debugging tips
Posted by Jeff Trawick <tr...@gmail.com>.
On 9/7/05, Joe Orton <jo...@redhat.com> wrote:
> On Tue, Sep 06, 2005 at 07:12:12AM -0400, Jeff Trawick wrote:
> > We discussed different patches previously on dev@httpd to make some of
> > this stuff more obvious by reporting configuration issues to the error
> > log at startup. Is that worth revisiting?
> > As I recall, the critical question was whether or not the user had to
> > know about the CoreDumpDirectory directive in order to have
> > information written to error log.
>
> Happy to revisit but I do disagree about logging anything by default if
> RLIMIT_CORE is 0 since all modern Linux systems are like that and people
> should not really have to care about core dumps. The patch we use in
> Fedora to bump RLIMIT_CORE iff it is zero and CoreDumpDirectory is
> configured has been pretty useful.
+1 for the Fedora patch
(I will continue to mangle somewhat ;) )
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: [PATCH] site: more debugging tips
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Sep 06, 2005 at 07:12:12AM -0400, Jeff Trawick wrote:
> We discussed different patches previously on dev@httpd to make some of
> this stuff more obvious by reporting configuration issues to the error
> log at startup. Is that worth revisiting?
> As I recall, the critical question was whether or not the user had to
> know about the CoreDumpDirectory directive in order to have
> information written to error log.
Happy to revisit but I do disagree about logging anything by default if
RLIMIT_CORE is 0 since all modern Linux systems are like that and people
should not really have to care about core dumps. The patch we use in
Fedora to bump RLIMIT_CORE iff it is zero and CoreDumpDirectory is
configured has been pretty useful.
joe
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: [PATCH] site: more debugging tips
Posted by Jeff Trawick <tr...@gmail.com>.
On 9/5/05, Joe Orton <jo...@redhat.com> wrote:
> After getting bored with writing these into bugzilla for the umpteenth
> time, does this look OK?
+1, with or without the changes suggested by Noirin
We discussed different patches previously on dev@httpd to make some of
this stuff more obvious by reporting configuration issues to the error
log at startup. Is that worth revisiting?
As I recall, the critical question was whether or not the user had to
know about the CoreDumpDirectory directive in order to have
information written to error log.
Extra useful logic would be to find out on Solaris via coreadm if
setuid processes are prevented from writing cores.
struct rlimit rl;
int rc;
int already_complained = 0;
rc = getrlimit(RLIMIT_CORE, &rl);
if (rc == 0) {
if (rl.rlim_cur == 0) {
/* Regardless of whether or not the server was started as root,
* a soft limit of zero for core file size prevents core dumps
* from being written for any web server processes.
* Theoretically we could try to raise the soft limit to the
* hard limit (if it is higher), but perhaps this is the
* administrator's way to say they want no core dumps.
*/
ap_log_error(APLOG_MARK, APLOG_NOTICE,
0,
s,
"Core file limit is 0; core dumps will be not be "
"written for server crashes");
already_complained = 1;
}
}
if (!already_complained &&
geteuid() == 0 &&
!ap_coredumpdir_configured) {
/* The server has started as root, so child processes run as another
* user. They SHOULD NOT have permission to write to the web server
* install directory, so if CoreDumpDirectory hasn't been specified,
* then in general no core dump can be written.
* Exception: On Solaris, the coreadm command can be used to specify
* an alternate directory for core dumps to be written.
* Potentially the same feature is/will be available on other platforms.
*/
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_NOTICE,
0,
s,
"CoreDumpDirectory not set; core dumps may not be
written for "
"child process crashes");
}
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: [PATCH] site: more debugging tips
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Sep 05, 2005 at 02:33:37PM +0100, Noirin Plunkett wrote:
> On Mon, Sep 05, 2005 at 01:53:12PM +0100, Joe Orton wrote:
> > After getting bored with writing these into bugzilla for the umpteenth
> > time, does this look OK?
>
> Looks good, just a few style/readability suggestions below.
Thanks a lot!
joe
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: [PATCH] site: more debugging tips
Posted by Noirin Plunkett <fi...@nerdchic.net>.
On Mon, Sep 05, 2005 at 01:53:12PM +0100, Joe Orton wrote:
> After getting bored with writing these into bugzilla for the umpteenth
> time, does this look OK?
Looks good, just a few style/readability suggestions below.
> +
> +<p>To ensure that a core dump can be written to a directory which is
I'd suggest this read
"To ensure that the core dump is written to a directory which is"
Otherwise, it seems tautological - the core dump can always be written
to a directory which is writeable by the apache user.
> +directive must be added to <code>httpd.conf</code>; for example:</p>
> +size must be lifted; for example:</p>
> +<p>On some platforms, further steps might be needed to enable core
> +dumps; see <a href="#sol27">Solaris 2.7 and coredumps</a> below.</p>
Semicolons are inappropriate here - try a dash, comma, or fullstop.
> +<p>When a child process crashes, a message will be logged to the
> +error_log like the following:</p>
"like the following" should come immediately after "a message".
Otherwise, it's unclear as to what will be like the following text - the
message, or the error log.
> +<p>if the text "possible coredump in /tmp" does not appear in the
> +<p>if attempting to debug a threaded server, such as using the
> +<p>this will produce a backtrace for each of the threads within the
> +process.</p>
I think these should start with capital letters - it just makes the text
easier to read.
Thanks!
Noirin
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: [PATCH] site: more debugging tips
Posted by Joshua Slive <jo...@slive.ca>.
On Mon, 5 Sep 2005, Joe Orton wrote:
> After getting bored with writing these into bugzilla for the umpteenth
> time, does this look OK?
+1.
Joshua.
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org