You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2017/12/05 21:24:22 UTC
[Bug 61861] New: "double free or corruption" under high load when
also used with mod_authn_dbd
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
Bug ID: 61861
Summary: "double free or corruption" under high load when also
used with mod_authn_dbd
Product: Apache httpd-2
Version: 2.4.29
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_authnz_ldap
Assignee: bugs@httpd.apache.org
Reporter: airikh@gmail.com
Target Milestone: ---
Created attachment 35588
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35588&action=edit
memory errors under high load
Server version: Apache/2.4.29 (Unix)
Server built: Nov 2 2017 12:15:50
Server's Module Magic Number: 20120211:68
Server loaded: APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
OS: RHEL 6.9
We've recently started experimenting with multiple AuthBasicProvider's, namely
ldap and dbd. For the authentication database, we are using a local SQLite3 DB.
The ldap provider is specified first in the list (AuthBasicProvider ldap dbd).
We ran the Apache Benchmark tool (ab) on a different server to check the speed
at high load, using -n1000 and -c1000. The -A option was used to specify either
valid LDAP credentials or valid credentials in the SQLite database.
When valid LDAP credentials are used, there are no bad requests or errors in
the log. When valid credentials in the SQLite database are used, however, there
are sometime failed requests with messages in the error log similar to what is
attached.
We should also note that when we just specify the dbd provider
(AuthBasicProvider dbd), we haven't experienced any failed requests or errors
in the log. We only see failed requests and errors when using both providers
(AuthBasicProvider ldap dbd) with valid credentials for the dbd provider.
Please let me know if you require more information than what I've provided.
Thank you for your time and effort!
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61861] Segmentation fault under load when password is incorrect
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
Eric Hontz <ai...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #5 from Eric Hontz <ai...@gmail.com> ---
I was able to replicate the issue with Apache 2.4.32. I haven't tried 2.4.33
yet, but the changelog doesn't indicate an LDAP-related fix.
I was able to obtain a core dump from a crash, which I'm attaching to this
report.
Also, I ran httpd with Valgrind, but I got such a ridiculous about of output.
I'm pretty sure the command I ran was
valgrind --tool=memcheck --undef-value-errors=no --memcheck:leak-check=yes
--memcheck:show-reachable=yes /local/ark/pkg/apache/2.4.32/bin/httpd -k start
-f /local/ark/pkg/apache/current/conf/httpd.conf
I'm not familiar with Valgrind, so I'm probably running with the wrong options.
If the Valgrind output is small enough in size to add as an attachment, I'll
attach that as well.
I plan to do further investigation, like running with the correct Valgrind
options to get a more decipherable output and setting the log level to trace8.
Unfortunately, I will need to stand up my own LDAP server and point Apache to
it, because I can't continue hammering the company's LDAP server with invalid
password attempts. Hopefully I will have time for this soon.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61861] Segmentation fault under load when password is incorrect
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
--- Comment #6 from Eric Hontz <ai...@gmail.com> ---
Created attachment 36023
--> https://bz.apache.org/bugzilla/attachment.cgi?id=36023&action=edit
core dump
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61861] Segmentation fault under load when password is incorrect
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
Eric Hontz <ai...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #35588|memory errors under high |backtrace
description|load |
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61861] Segmentation fault under load when password is incorrect
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
--- Comment #7 from Eric Hontz <ai...@gmail.com> ---
Created attachment 36024
--> https://bz.apache.org/bugzilla/attachment.cgi?id=36024&action=edit
Valgrind output
As I explained, this output seems indecipherable to me, so I probably need to
use different Valgrind options.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61861] Segmentation fault under load when password is incorrect
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
Eric Hontz <ai...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|"double free or corruption" |Segmentation fault under
|under high load when also |load when password is
|used with mod_authn_dbd |incorrect
--- Comment #1 from Eric Hontz <ai...@gmail.com> ---
I recently confirmed that the crashes are present when only the ldap provider
is configured (AuthBasicProvider ldap) but an incorrect password is provided.
I've even seen the seg faults when only using a concurency level of 50 with the
Apache Benchmark tool (ab -n2000 -c50 ...).
Occasionally, after the seg fault happens, all of the threads of a worker will
be in the waiting state, and CPU usage will be maxed on the server.
Please let me know if there is further information I can provide to help
determine the root cause.
Thank you.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61861] Segmentation fault under load when password is incorrect
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
--- Comment #4 from Luca Toscano <to...@gmail.com> ---
Hi Eric, any news?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61861] Segmentation fault under load when password is incorrect
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
--- Comment #3 from Eric Hontz <ai...@gmail.com> ---
Thank you for the reply, Eric!
I will await 2.4.32 to see if the concurrency fix solves the problem and report
back. If I still experience the issue, I'll provide Valgrind and trace8 info
when I report back.
Thanks again!
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 61861] Segmentation fault under load when password is incorrect
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61861
Eric Covener <co...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #2 from Eric Covener <co...@gmail.com> ---
2.4.32 is around the corner and does have an LDAP concurrency fix, but usually
it leads to high CPU and not a crash. But, it is probably worth revisiting
this if you can.
If it doesn't pan out -- could you try to run your scenario under
valgrind to potentially make it fail earlier? and/or grab "trace8" output
leading to the crash?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org