You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2008/09/18 18:03:06 UTC
DO NOT REPLY [Bug 45834] New: Stale LDAP connections take 15+
minutes to finish queries
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
Summary: Stale LDAP connections take 15+ minutes to finish
queries
Product: Apache httpd-2
Version: 2.2.9
Platform: PC
OS/Version: Linux
Status: NEW
Severity: major
Priority: P2
Component: mod_ldap
AssignedTo: bugs@httpd.apache.org
ReportedBy: lucasr@tampabay.rr.com
I am running a RedHat 5 server to serve SVN and I am using mod_authnz_ldap for
user authentication. The server is configured correctly, I know this because
shortly after a restart the authentications occur within a reasonable amount of
time. However, I have noticed that after long periods of inactivity (say over
night) the authentication process will take approximately 15 minutes to
complete. During the time it is attempting to run the query I see through
netstat that a connection was established and there is something in the
transmit queue but for some reason it doesn't complete for 15+ minutes. The
interesting thing is that it does seem to complete. However, I'm not sure if
it is because it gives up and tries again or if that connection completes. The
log files in debug mode so very little information, basically only the request
and then some time later the user was authenticated. Through my debugging I
have noticed that the ldap module does not close the connection to the ldap
server even after long periods of no requests. Our internal network is complex
and there is a firewall sitting between our server and the LDAP server. However
the latency is relatively low (less then 85ms) and the load on the LDAP server
is minimal. I'm assuming it is intentional the connections don't close but it
may be contributing to the issue.
I have tried this on apache 2.2.3 that comes with Redhat and I have compiled
apache 2.2.9 and both show the same behavior. I have included the netstat info
and my configuration below. I am using the defaults configurations for just
about everything else.
Questions:
Is there a way to disable the mod_ldap connection pooling? I see that each
idld/spare httpd process opens and maintains a connection.
Is there a way to have mod_ldap disconnect after some period of time? I have
no insight into the firewall, it is possible it does not like the persistent
connection and is playing a factor. Ideally I would think if there was no
activity for 15 minutes or so you could disconnect.
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 138 xxx.xxx.xxx.xxx:33871 ldap.x.x:ldaps ESTABLISHED
SVN and LDAP configurations files:
LDAPCacheEntries 0
LDAPCacheTTL 600
LDAPConnectionTimeout 10
LDAPOpCacheEntries 1024
LDAPOpCacheTTL 600
LDAPSharedCacheSize 10240
LDAPTrustedMode SSL
LDAPVerifyServerCert Off
<Location /svn>
DAV svn
SSLRequireSSL
# Subversion Paths
SVNParentPath <PATH TO REPOS>
SVNListParentPath on
AuthzSVNAccessFile <PATH TO SVN AUTH FILE>
# Access control policy
AuthBasicProvider ldap file
AuthzLDAPAuthoritative Off
AuthType Basic
AuthName "Repositories"
AuthLDAPBindDN XXXXXX
AuthLDAPBindPassword XXXXXX
AuthLDAPURL
"ldaps://XXXXXX.com:636/ou=XXXXXX,o=XXXXXX?uid?sub?(objectClass=*)"
AuthUserFile <PATH TO WWW AUTH FILE>
Require valid-user
</Location>
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
--- Comment #10 from Michael Marineau <mi...@marineau.org> 2009-05-18 14:51:46 PST ---
Just a quick note, recent versions of openldap properly enable tcp keep alive
which fixes the original issue of the firewall killing idle connections. Just
tune the net.ipv4.tcp_keepalive_* sysctl parameters according to your
firewall's timeout and all will be happy. However it would still be nice if
apache handled dead connections better by setting a timeout for any request it
makes, not just when opening a connection.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
--- Comment #4 from Eric Covener <co...@gmail.com> 2008-09-19 18:20:58 PST ---
(In reply to comment #3)
> After further research this issue seems to be caused by the firewall between
> the requesting server and the LDAP server. The firewall will break idle
> connections after a period of time thus explaining the timeout after a
> connection has been made.
This sequence doesn't really explain the timeout. On a normal network, when the
connection is severed the host OS finds out, then when the application goes to
use the socket it finds out too -- without a delay. This is the part that is
failing.
you should be able to add something like:
http://people.apache.org/~covener/trunk-ldap-nokeep.diff
to
http://svn.apache.org/viewvc?view=rev&revision=599164
and use SetEnvIf to disable the re-use.
(but as you can see, actually cleaning up the connections is unchartered
territory)
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
--- Comment #3 from Lucas <lu...@gmail.com> 2008-09-19 15:37:50 PST ---
After further research this issue seems to be caused by the firewall between
the requesting server and the LDAP server. The firewall will break idle
connections after a period of time thus explaining the timeout after a
connection has been made. However, this does seem to show a deficiency in
Apache in that you cannot configure it to close connections after some
specifiable time after a query and you cannot set a reasonable timeout for
existing connections that don't get a response.
The development httpd code (2.3.0) now has a "keep" option in the
uldap_connection_close function in file util_ldap.c. I believe this to imply
that some might want an option to disconnect. However, the development code
always hard sets it to keep the connection. Unfortunately, this does not solve
my problem unless it is back ported to the 2.2.3 or 2.2.9 branches or I figure
out how to safely modify the source.
uldap_connection_close function with "keep":
http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/ldap/util_ldap.c?revision=646582&view=markup
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
Lucas <lu...@tampabay.rr.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |lucasr@tampabay.rr.com
Status|NEEDINFO |ASSIGNED
--- Comment #2 from Lucas <lu...@tampabay.rr.com> 2008-09-18 12:16:47 PST ---
Q: I assume you have 1 authenticated request that works, then a hang during a
subsequent request -- can you get a backtrace of the relevant child process
when it's hanging?
A: Correct when httpd is freshly started authentication works correctly for
some time (during my debugging I have turned the caching practically off) it
isn't until it becomes idle for a period of time that things start stalling.
Though from the strace it does look like the connection is timing out but why
it takes so long (~28 minutes), I don't understand.
An strace on the process results in (I removed the encoded data and replaced it
with XXXXXX:
$strace -r -f -v -p 3863
Process 3863 attached - interrupt to quit
0.000000 time(NULL) = 1221762688
0.001216 write(48, "XXXXXX"..., 138) = 138
0.000617 poll([{fd=48, events=POLLIN|POLLPRI|POLLERR|POLLHUP,
revents=POLLIN|POLLERR|POLLHUP}], 1, -1) = 1
1707.565544 read(48, 0x92cb6a0, 5) = -1 ETIMEDOUT (Connection timed out)
0.001745 time(NULL) = 1221764396
0.000463 write(48, "XXXXXX"..., 138) = -1 EPIPE (Broken pipe)
0.000709 --- SIGPIPE (Broken pipe) @ 0 (0) ---
0.000931 time(NULL) = 1221764396
0.000076 time(NULL) = 1221764396
...
Note: It sits at "poll(" while waiting for it to authenticate.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
--- Comment #5 from Rainer Jung <ra...@kippdata.de> 2008-09-20 01:38:45 PST ---
(In reply to comment #4)
> (In reply to comment #3)
> > After further research this issue seems to be caused by the firewall between
> > the requesting server and the LDAP server. The firewall will break idle
> > connections after a period of time thus explaining the timeout after a
> > connection has been made.
>
> This sequence doesn't really explain the timeout. On a normal network, when the
> connection is severed the host OS finds out, then when the application goes to
> use the socket it finds out too -- without a delay. This is the part that is
> failing.
I think it does. If the connection is already established and the firewall does
an idle connection drop, none of the two communication partners are made aware
of that. Communication packets apart from maybe RST and FIN are simply dropped
by the firewall. As a consequence the sending side starts to do TCP retransmits
which depending on OS and TCP tuning can take several minutes before the TCP
stack finally gives up (I vaguely remember 7 minutes for some version of
Solaris).
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
--- Comment #7 from Lucas <lu...@luc-ace.com> 2008-09-22 18:20:14 PST ---
Created an attachment (id=22623)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=22623)
Patch that forces httpd's module mod_ldap to unbind after every requests
The patch is for the 2.2.3 branch. This version was chosen to maintain version
numbers with RedHat Linux 5.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
Eric Covener <co...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #1 from Eric Covener <co...@gmail.com> 2008-09-18 09:57:03 PST ---
Nothing in mod_ldap monitors or cares about a pooled connection until some
normal request-processing thread goes to use it.
I do not see any way to turn off the connection pooling, and there is no search
timeout either.
I assume you have 1 authenticated request that works, then a hang during a
subsequent request -- can you get a backtrace of the relevant child process
when it's hanging?
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
Stefan Fritsch <sf...@sfritsch.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution| |FIXED
--- Comment #11 from Stefan Fritsch <sf...@sfritsch.de> 2010-08-18 19:08:02 EDT ---
This is a ldap-toolkit / firewall issue.
The new LDAPTimeout directive in trunk allows to work around the problem.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
--- Comment #9 from Lucas <lu...@luc-ace.com> 2008-10-06 14:03:13 PST ---
Follow-up, I built from source Apache 2.2.9 with APR 1.3.3 and APR-Util 1.3.4
and the problem persists. For this test I was using the default ttl value.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
Lucas <lu...@luc-ace.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |ASSIGNED
--- Comment #8 from Lucas <lu...@luc-ace.com> 2008-09-22 18:33:11 PST ---
I attached a patch for the 2.2.3 that forces an unbind after every query. This
is obviously not most efficient solution but does resolve my issues. I chose
this version to patch so that I could maintain the RedHat RPM build process and
patches. Though it does mean my APR version is 1.2.7.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 45834] Stale LDAP connections take 15+ minutes to
finish queries
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=45834
Nick Kew <ni...@webthing.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |NEEDINFO
--- Comment #6 from Nick Kew <ni...@webthing.com> 2008-09-20 03:05:21 PST ---
What APR version do you have?
The changes in late May (r6660219 in 1.3.x) may be relevant here, if you set
ttl below any timeouts imposed by external agents like your firewall.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org