You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Ewald Dieterich <ew...@t-online.de> on 2013/12/12 10:16:51 UTC

Reverse proxy, mod_security, segmentation fault

I already asked on the mod_security developer mailing list for help, but 
didn't get a response. So I'm trying my luck here.

On a Debian unstable installation (Apache 2.4.6, apr 1.4.8, apr-util 
1.5.3, mod_security 2.7.5) I enabled mpm_worker and configured a simple 
reverse proxy. When I enable mod_security and then send large amounts of 
  POST requests to a misconfigured backend server that just drops the 
requests, I get segmentation faults.

For mod_security I only set "SecRequestBodyAccess On", I didn't enable 
any rules. Here is the configuration for the location:

<Location />
     SecRuleEngine On
     SecRequestBodyAccess On

     ProxyPass http://backend:8080/
     ProxyPassReverse http://backend:8080/
</Location>

On the backend I run faucet to simulate the request-dropping backend 
server:

faucet 8080 --out echo ""

If a client sends a request, the reverse proxy sends a "502 Bad Gateway" 
response and logs the following errors:

[...] (104)Connection reset by peer: [client 10.128.128.81:49143] 
AH01102: error reading status line from remote server backend:8080
[...] AH00898: Error reading from remote server returned by /

So everything works as expected.

Now I send POST requests in parallel by starting this loop on multiple 
shells, the more the better (data_file is 25k):

while true ; do curl -d @data_file http://frontend/ ; done

Every once in a while I get a segmentation fault. If I enable 
mpm_prefork (or disable mod_security) I don't get any segmentation faults.

This seems to be a multithreading race condition, so I'm not sure if 
backtraces are of any help, but I attached two files with the output of 
"thread apply all bt" from gdb for two segmentation faults.

Is this a bug or am I doing something wrong? Thanks for your help!

Re: Reverse proxy, mod_security, segmentation fault

Posted by Micha Lenk <mi...@lenk.info>.
Hi Ewald,

Am 12.12.2013 10:16, schrieb Ewald Dieterich:
> [...] Is this a bug or am I doing something wrong?

I would consider the segmentation faults to be bugs. The question is
whether they are bugs in httpd or in mod_security...

Looking at the backtraces I noticed that most threads are busy in a
syscall (i.e. poll(), read() and the like) or waiting on a thread mutex
(i.e. pthread_cond_wait() etc.). Those threads are blocked and most
likely do not actively contribute to a segfault. In both segfaults you
can find a thread that is currently running a signal handler (i.e.
kill()). If I remember correctly, it is quite error prone to handle
signals in multi-threaded applications. I would assume your segfaults
are related to these typical signal handling issues with multithreading.

Hope that helps...

Regards,
Micha

Re: Reverse proxy, mod_security, segmentation fault

Posted by Rainer Jung <ra...@kippdata.de>.
On 12.12.2013 16:16, Ewald Dieterich wrote:
> On 12/12/2013 11:53 AM, Rainer Jung wrote:
>> On 12.12.2013 10:16, Ewald Dieterich wrote:
>>> On a Debian unstable installation (Apache 2.4.6, apr 1.4.8, apr-util
>>> 1.5.3, mod_security 2.7.5) I enabled mpm_worker and configured a simple
>>> reverse proxy. When I enable mod_security and then send large amounts of
>>>   POST requests to a misconfigured backend server that just drops the
>>> requests, I get segmentation faults.
>>
>> Could it be
>>
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=50335
>>
>> See the patch discussion starting at comment #28.
>>
>> The currently committed trunk patches are
>>
>> http://svn.apache.org/viewvc?view=revision&revision=1534321
>>
>> and
>>
>> http://svn.apache.org/viewvc?view=revision&revision=1550061
>>
>> Those fixes might not yet be a complete solution to the problem, but
>> might be easy to backport to 2.4 to check whether they fix your problem.
> 
> The patches fix my problem, no more segmentation faults.


Thanks for letting us know and updating the ticket.

Rainer

Re: Reverse proxy, mod_security, segmentation fault

Posted by Jim Jagielski <ji...@jaguNET.com>.
I've proposed for backport...

On Dec 12, 2013, at 10:16 AM, Ewald Dieterich <ew...@t-online.de> wrote:

> On 12/12/2013 11:53 AM, Rainer Jung wrote:
>> On 12.12.2013 10:16, Ewald Dieterich wrote:
>>> On a Debian unstable installation (Apache 2.4.6, apr 1.4.8, apr-util
>>> 1.5.3, mod_security 2.7.5) I enabled mpm_worker and configured a simple
>>> reverse proxy. When I enable mod_security and then send large amounts of
>>>  POST requests to a misconfigured backend server that just drops the
>>> requests, I get segmentation faults.
>> 
>> Could it be
>> 
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=50335
>> 
>> See the patch discussion starting at comment #28.
>> 
>> The currently committed trunk patches are
>> 
>> http://svn.apache.org/viewvc?view=revision&revision=1534321
>> 
>> and
>> 
>> http://svn.apache.org/viewvc?view=revision&revision=1550061
>> 
>> Those fixes might not yet be a complete solution to the problem, but
>> might be easy to backport to 2.4 to check whether they fix your problem.
> 
> The patches fix my problem, no more segmentation faults.
>

Re: Reverse proxy, mod_security, segmentation fault

Posted by Ewald Dieterich <ew...@t-online.de>.
On 12/12/2013 11:53 AM, Rainer Jung wrote:
> On 12.12.2013 10:16, Ewald Dieterich wrote:
>> On a Debian unstable installation (Apache 2.4.6, apr 1.4.8, apr-util
>> 1.5.3, mod_security 2.7.5) I enabled mpm_worker and configured a simple
>> reverse proxy. When I enable mod_security and then send large amounts of
>>   POST requests to a misconfigured backend server that just drops the
>> requests, I get segmentation faults.
>
> Could it be
>
> https://issues.apache.org/bugzilla/show_bug.cgi?id=50335
>
> See the patch discussion starting at comment #28.
>
> The currently committed trunk patches are
>
> http://svn.apache.org/viewvc?view=revision&revision=1534321
>
> and
>
> http://svn.apache.org/viewvc?view=revision&revision=1550061
>
> Those fixes might not yet be a complete solution to the problem, but
> might be easy to backport to 2.4 to check whether they fix your problem.

The patches fix my problem, no more segmentation faults.

Re: Reverse proxy, mod_security, segmentation fault

Posted by Rainer Jung <ra...@kippdata.de>.
On 12.12.2013 10:16, Ewald Dieterich wrote:
> I already asked on the mod_security developer mailing list for help, but
> didn't get a response. So I'm trying my luck here.
> 
> On a Debian unstable installation (Apache 2.4.6, apr 1.4.8, apr-util
> 1.5.3, mod_security 2.7.5) I enabled mpm_worker and configured a simple
> reverse proxy. When I enable mod_security and then send large amounts of
>  POST requests to a misconfigured backend server that just drops the
> requests, I get segmentation faults.
> 
> For mod_security I only set "SecRequestBodyAccess On", I didn't enable
> any rules. Here is the configuration for the location:
> 
> <Location />
>     SecRuleEngine On
>     SecRequestBodyAccess On
> 
>     ProxyPass http://backend:8080/
>     ProxyPassReverse http://backend:8080/
> </Location>
> 
> On the backend I run faucet to simulate the request-dropping backend
> server:
> 
> faucet 8080 --out echo ""
> 
> If a client sends a request, the reverse proxy sends a "502 Bad Gateway"
> response and logs the following errors:
> 
> [...] (104)Connection reset by peer: [client 10.128.128.81:49143]
> AH01102: error reading status line from remote server backend:8080
> [...] AH00898: Error reading from remote server returned by /
> 
> So everything works as expected.
> 
> Now I send POST requests in parallel by starting this loop on multiple
> shells, the more the better (data_file is 25k):
> 
> while true ; do curl -d @data_file http://frontend/ ; done
> 
> Every once in a while I get a segmentation fault. If I enable
> mpm_prefork (or disable mod_security) I don't get any segmentation faults.
> 
> This seems to be a multithreading race condition, so I'm not sure if
> backtraces are of any help, but I attached two files with the output of
> "thread apply all bt" from gdb for two segmentation faults.
> 
> Is this a bug or am I doing something wrong? Thanks for your help!

Could it be

https://issues.apache.org/bugzilla/show_bug.cgi?id=50335

See the patch discussion starting at comment #28.

The currently committed trunk patches are

http://svn.apache.org/viewvc?view=revision&revision=1534321

and

http://svn.apache.org/viewvc?view=revision&revision=1550061

Those fixes might not yet be a complete solution to the problem, but
might be easy to backport to 2.4 to check whether they fix your problem.
Please add your finding to the above bugzilla.

Regards,

Rainer