You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Igor Cicimov <ic...@gmail.com> on 2009/10/07 02:20:05 UTC
Re: [users@httpd] apache2 core dump
Thought to post some more debugging regarding this problem I have.
This time I had help from one of our Unix engineers with the debugging
and you can see he used mdb in this case:
bash-2.05$ mdb core.httpd.28499.wenpweb1.500.500.1253659663
mdb: warning: failed to infer pathname to executable; symbol table
will not be available
Loading modules: [ libc.so.1 libthread.so.1 ld.so.1 ]
> $C (display call stack)
fbef95d8 libapr-1.so.0`apr_palloc+4(44617465, 7, 70740020, 7efefeff,
81010100, ff00)
fbef9648 libapr-1.so.0`apr_pstrdup+0x20(44617465, 27a758, 0, 13f75a4, 0, 27a75a)
fbef96b8 libapr-1.so.0`apr_table_add+0xa8(13f75a0, 27a758, 27a760, 5,
fffc0000, d)
fbef9728 mod_mem_cache.so`deep_table_copy+0x60(13f75a0, 0, 5, 2ba944, 38, 1)
fbef9798 mod_mem_cache.so`store_headers+0x1c(ecba0, 2797f8, ecba8,
26b900, ecba8, fec522d4)
(snip)
> apr_palloc::dis (disassemble apr_palloc)
libapr-1.so.0`apr_palloc: save %sp, -0x70, %sp
libapr-1.so.0`apr_palloc+4: ld [%i0 + 0x2c], %l0
libapr-1.so.0`apr_palloc+8: mov %i0, %l1
(snip)
> $r (display registers)
(snip)
%o0 = 0x00008ca0 %i0 = 0x44617465
(snip)
%i0 (the first parameter being passed to a function) is an invalid
pointer. Looking at it, it actually appears to be part of a text
string:
0x44 = D
0x61 = a
0x74 = t
0x65 = e
Putting all this together, apr_table_add is dereferencing a string and
passing the contents of it to apr_pstrdup, instead of passing a
pointer to the memory pool (which is what apr_pstrdup expects)
Most likely something in apr_table_add is getting over-written.
Other cores show similar behaviour:
hmurn@wenpweb1# mdb core.httpd.28752.wenpweb1.500.500.1254735017
> $C
fcff95d8 libapr-1.so.0`apr_palloc+4(436f6e74 ('Cont'), 7, 70740020,
7efefeff, 81010100, ff00)
bash-2.05# mdb core.httpd.12737.wenpweb1.500.500.1254699635
> $C
fccf96b8 libapr-1.so.0`apr_table_add+0xb4(56f5c10, 527d0f8, 5281740, 5, 1, 6)
> apr_table_add+0xb4::dis
libapr-1.so.0`apr_table_add+0xb4: st %o0, [%l0]
> $r
%g0 = 0x00000000 %l0 = 0x68747470 ('http')
So it really looks like a bug in the apache apr so the patch that Nick
suggested makes more sense for me now.
Nik, do you know if this is the same issue that the patch is suppose
to fix? Can you please point me to the patch release documentation so
I can read some more about it?
Thanks for your help.
Igor
On 9/24/09, Igor Cicimov <ic...@gmail.com> wrote:
> Ah forgot about the tool ... used pstack to debug the core dumps since it
> is
> a production server and don't have gdb available.
>
> Cheers,
>
> Igor
>
> On Thu, Sep 24, 2009 at 12:37 PM, Igor Cicimov <ic...@gmail.com> wrote:
>
>> Hi Nick,
>>
>> First thanks for your reply much appreciate it.
>>
>> Yes, the server is reverse proxy only and from what I could see from
>> couple
>> of other core dumps they all look the same. First I thought it might be
>> the
>> proxy_html module causing this but now I'm not that sure and might be
>> something more serious.
>>
>> I'll have a look in the patch and see if it is applicable to me.
>>
>> Thanks,
>>
>> Igor
>>
>>
>> On Wed, Sep 23, 2009 at 6:55 PM, Nick Kew <ni...@webthing.com> wrote:
>>
>>>
>>> On 23 Sep 2009, at 07:32, Igor Cicimov wrote:
>>>
>>> Hi all,
>>>>
>>>> I get the following core dumps from the apache:
>>>> [chop]
>>>>
>>>> It's apache 2.2.12 running as reverse proxy on Solaris 9 box with
>>>> mpm_worker module. Looks like the threads get in some conditional wait
>>>> state
>>>> and the process becomes zombie. I can see individual child processes
>>>> dieing
>>>> in the log file with segmentation fault.
>>>>
>>>
>>> What tool produced that dump? Do you have gdb available?
>>> (a gdb backtrace means more to me than what you show).
>>>
>>> You mention it's a reverse proxy, and the first thread you show is
>>> a proxy request. Are the crashes always associated with that,
>>> so we could look there for a cause? Or is that a meaningless
>>> question because the server does nothing else?
>>>
>>> You might also apply Jeff's APR patch at
>>> https://issues.apache.org/bugzilla/attachment.cgi?id=24161
>>> and see if that helps.
>>>
>>> --
>>> Nick Kew
>>>
>>> ---------------------------------------------------------------------
>>> The official User-To-User support forum of the Apache HTTP Server
>>> Project.
>>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>> For additional commands, e-mail: users-help@httpd.apache.org
>>>
>>>
>>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: apache2 core dump
Posted by Igor Cicimov <ic...@gmail.com>.
Hi,
I tried to recompile and install the patched mod_mem_cache.c file from the
bug report above but I got some errors. The command I used to create and
install the loadable module:
# /usr/local/apache2/bin/apxs -c -i mod_mem_cache.c cache_storage.c
cache_util.c
This created new mod_mem_cache.so file in the modules folder but when I
tried to start the server I got the following error:
httpd: Syntax error on line 70 of /usr/local/apache2/conf/httpd.conf: Cannot
load /usr/local/apache2/modules/mod_mem_cache.so into server: ld.so.1:
./bin/httpd: fatal: relocation error: file
/usr/local/apache2/modules/mod_mem_cache.so: symbol cache_find: referenced
symbol not found
I don't know if the way I did it is the correct one and if the error I get
is maybe related with some problem installing the module on Solaris 9 (I
have the same problem with the latest mod_security.2.5.10)?
Thanks,
Igor
On Thu, Oct 8, 2009 at 11:57 PM, Igor Cicimov <ic...@gmail.com> wrote:
> Hi Dan, thanks for your reply. I'm not sure about 2.2.14 since this is
> production server. I'll have to build it first in model and test it.
> Strangely I didn't face the same problem on the model servers with the
> same Solaris 9 OS but that might have been because of lack of traffic.
>
> Is there any other way to fix this, a patch that I can apply maybe?
>
> Thanks,
> Igor
>
> On 10/8/09, Dan Poirier <po...@pobox.com> wrote:
> > That looks like Apache PR 47672, which has been fixed. Can you try
> > Apache 2.2.14?
> >
> > Dan Poirier
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
> Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > " from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
> >
> >
>
Re: [users@httpd] Re: apache2 core dump
Posted by Igor Cicimov <ic...@gmail.com>.
Hi Dan, thanks for your reply. I'm not sure about 2.2.14 since this is
production server. I'll have to build it first in model and test it.
Strangely I didn't face the same problem on the model servers with the
same Solaris 9 OS but that might have been because of lack of traffic.
Is there any other way to fix this, a patch that I can apply maybe?
Thanks,
Igor
On 10/8/09, Dan Poirier <po...@pobox.com> wrote:
> That looks like Apache PR 47672, which has been fixed. Can you try
> Apache 2.2.14?
>
> Dan Poirier
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: apache2 core dump
Posted by Dan Poirier <po...@pobox.com>.
That looks like Apache PR 47672, which has been fixed. Can you try
Apache 2.2.14?
Dan Poirier
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org