Crash inside worker mpm's ap_mpm_pod_check()

[Graham Leggett <mi...@sharp.fm>]

Hi all,

I am currently trying to get to the bottom of a crash that is  
occurring under load (during an Avalanche load test, most  
specifically) inside the worker mpm, with the stacktrace as below.

Most specifically, it segfaults inside the read() below:

AP_DECLARE(int) ap_mpm_pod_check(ap_pod_t *pod)
{
     char c;
     apr_os_file_t fd;
     int rc;

     /* we need to surface EINTR so we'll have to grab the
      * native file descriptor and do the OS read() ourselves
      */
     apr_os_file_get(&fd, pod->pod_in);
     rc = read(fd, &c, 1);

In theory, looking at the parameters of read(), &c and 1 are  
calculated by the compiler, which leaves the possibility that the fd  
is invalid, although fd does seem to have a sensible value (4).

What does seem strange is this line in the backtrace:

#1  0x00002aca224a5e77 in read (pod=<value optimized out>)

The parameters of read() are (int fd, void *buf, size_t nbytes), not  
(ap_pod_t *pod), this looks a bit odd.

The next step for me is to attempt to deploy a non optimised httpd to  
see if I can reproduce it, in the mean time, does this ring any bells  
with anybody?

(gdb) bt full
#0  0x00002aca23d3491b in read () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00002aca224a5e77 in read (pod=<value optimized out>)
     at /usr/include/bits/unistd.h:35
No locals.
#2  ap_mpm_pod_check (pod=<value optimized out>)
     at /usr/src/debug/httpd-2.2.17/server/mpm/worker/pod.c:54
         c = <value optimized out>
         fd = 4
#3  0x00002aca224a3e42 in child_main (child_num_arg=<value optimized  
out>)
     at /usr/src/debug/httpd-2.2.17/server/mpm/worker/worker.c:1258
         threads = <value optimized out>
         rv = 577498496
         ts = 0x2aca2c4883f8
         thread_attr = 0x2aca2c488418
         start_thread_id = <value optimized out>
#4  0x00002aca224a3ff5 in make_child (s=0x2aca287d5848, slot=2)
     at /usr/src/debug/httpd-2.2.17/server/mpm/worker/worker.c:1341
         pid = 0
#5  0x00002aca224a40af in startup_children (number_to_start=8)
     at /usr/src/debug/httpd-2.2.17/server/mpm/worker/worker.c:1375
         i = 2
#6  0x00002aca224a4b7f in ap_mpm_run (_pconf=0x2aca287ce138,
---Type <return> to continue, or q <return> to quit---
     plog=<value optimized out>, s=0x2aca287d5848)
     at /usr/src/debug/httpd-2.2.17/server/mpm/worker/worker.c:1742
         remaining_children_to_start = 10
         rv = <value optimized out>
#7  0x00002aca2247dc73 in main (argc=4, argv=0x7fff25e2ca78)
     at /usr/src/debug/httpd-2.2.17/server/main.c:739
         c = 68 'D'
         configtestonly = 0
         confname = 0x2aca224a7478 "conf/httpd.conf"
         def_server_root = 0x2aca224a7488 "/etc/httpd"
         temp_error_log = 0x0
         error = <value optimized out>
         process = 0x2aca287cc218
         server_conf = <value optimized out>
         pglobal = 0x2aca287cc128
         pconf = 0x2aca287ce138
         plog = 0x2aca288002c8
         ptemp = 0x2aca287d2158
         pcommands = 0x2aca287d0148
         opt = 0x2aca287d0238
         rv = <value optimized out>
         optarg = 0x7fff25e2cf32 "SERVER_CLASS_STATICP"
(gdb)

Regards,
Graham
--

Re: Crash inside worker mpm's ap_mpm_pod_check()

[Jeff Trawick <tr...@gmail.com>]

On Mon, Dec 13, 2010 at 8:06 PM, Graham Leggett <mi...@sharp.fm> wrote:
> Hi all,
>
> I am currently trying to get to the bottom of a crash that is occurring
> under load (during an Avalanche load test, most specifically) inside the
> worker mpm, with the stacktrace as below.
>
> Most specifically, it segfaults inside the read() below:

Check the other threads.  It didn't segfault on this one.

If gdb doesn't properly identify the crasher, find the threads that
that aren't in a syscall; one of those is the culprit.