You are viewing a plain text version of this content. The canonical link for it is here.
Posted to fortress@directory.apache.org by Jan Sindberg <ja...@gmail.com> on 2016/02/06 20:53:03 UTC
LDAP connection timeout
When an application such as Fortress Commander is left inactive for hours -
all connections in the connection pool is closed (timed out ?). But for
each connection there is a 30 sec. timeout before the conclusion that the
connection is dead. With 10 connections that is a five minutes wait for the
first "unknowing victim" and an application-log with ten equal logs
including stack trace.
How are you dealing with this in your systems?
Can I configure this differently from Fortress or in ApacheDS?
Could we make Fortress be a little faster realising that the connection
pool needs to be "refreshed"?
I guess the issue could be simulated by restarting ApacheDS (I haven't
tested this yet) - that should break whatever open connections there we.
// Jan
Re: LDAP connection timeout
Posted by Jan Sindberg <ja...@gmail.com>.
2016-02-06 23:41 GMT+01:00 Shawn McKinney <sm...@apache.org>:
>
> > On Feb 6, 2016, at 1:53 PM, Jan Sindberg <ja...@gmail.com> wrote:
> >
> > When an application such as Fortress Commander is left inactive for
> hours -
> > all connections in the connection pool is closed (timed out ?). But for
> > each connection there is a 30 sec. timeout before the conclusion that the
> > connection is dead. With 10 connections that is a five minutes wait for
> the
> > first "unknowing victim" and an application-log with ten equal logs
> > including stack trace.
>
> A connection sometimes times out inside the client connection pool, on the
> server-side, or within the intermediaries (firewalls). First thing we’ll
> need is the stack trace and your fortress.properties, with necessary
> redactions of course, in order to (try and) pinpoint the root cause.
>
> >
> > On Feb 6, 2016, at 1:53 PM, Jan Sindberg <ja...@gmail.com> wrote:
> >
> > How are you dealing with this in your systems?
> > Can I configure this differently from Fortress or in ApacheDS?
> > Could we make Fortress be a little faster realising that the connection
> > pool needs to be "refreshed"?
> > I guess the issue could be simulated by restarting ApacheDS (I haven't
> > tested this yet) - that should break whatever open connections there we.
>
> There are settings in the client-side pooling mechanism that affect this
> behavior. There are code changes that can be made. There are settings on
> the server. But first we must understand why it’s happening in your tests.
>
> Just to be clear - what you're describing here is very bad behavior for an
> application to have, and deserves serious attention from us. To put it
> simply, it’s a show-stopper.
>
> Just now, I ran a couple of tests inside my local env. Using openldap as
> the server, and a max idle timeout of 30 seconds. After 30 seconds of
> inactivity, slapd will forcefully terminate any connections being held open
> by a client. I also had fortress web deployed to tomcat.
>
> First I restarted slapd daemon with open browser session logged into the
> app. Next I left the web session open for around 10 minutes.
>
> Neither test case resulted in a failure. For my tests, the ldap
> connection pool managed to recover.
>
> I need to re-run the same tests using ApacheDS server, and will will once
> I learn how to set its max idle timeout.
>
> But, just because I can’t recreate your issue inside my local env doesn’t
> tell us much. The good news, once we’ve got repeatable steps to recreate,
> this problem ought to get solved quickly.
>
> Thanks for reporting it,
>
> Shawn
>
To be clear, eventually, after 10 failed connections (five minutes), the
pool will recover.
The application might be idle over night. Then the first person trying to
use it will experience the long wait. Interestingly, I could not duplicate
this for Fortress Commander today, which points in the direction that it is
something on our own servers.
The log goes like this:
2016-02-06 13:07:26,485 ERROR
[org.apache.directory.ldap.client.api.LdapNetworkConnection]
Thread='http-0.0.0.0-14880-7' Bind failed : timeout occurred
2016-02-06 13:07:26,485 ERROR
[org.apache.directory.ldap.client.api.LdapNetworkConnection]
Thread='http-0.0.0.0-14880-7' The response queue has been emptied, no
response was found.
org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
occurred
at
org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1198)
at
org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:127)
at
org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:112)
at
org.apache.directory.ldap.client.api.MonitoringLdapConnection.bind(MonitoringLdapConnection.java:112)
at
org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory.bindConnection(DefaultLdapConnectionFactory.java:64)
at
org.apache.directory.ldap.client.api.AbstractPoolableLdapConnectionFactory.activateObject(AbstractPoolableLdapConnectionFactory.java:60)
at
org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionFactory.activateObject(ValidatingPoolableLdapConnectionFactory.java:111)
at
org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionFactory.activateObject(ValidatingPoolableLdapConnectionFactory.java:55)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1204)
at
org.apache.directory.ldap.client.api.LdapConnectionPool.getConnection(LdapConnectionPool.java:123)
at
org.apache.directory.fortress.core.ldap.ApacheDsDataProvider.getAdminConnection(ApacheDsDataProvider.java:1432)
at
org.apache.directory.fortress.core.impl.UserDAO.getUser(UserDAO.java:829)
at
org.apache.directory.fortress.core.impl.UserP.read(UserP.java:217)
at
org.apache.directory.fortress.core.impl.UserP.createSessionTrusted(UserP.java:523)
at
org.apache.directory.fortress.core.impl.UserP.createSession(UserP.java:455)
at
org.apache.directory.fortress.core.impl.AccessMgrImpl.createSession(AccessMgrImpl.java:171)
at com.autorola.rbac.RBACUtil.createSession(RBACUtil.java:64)
at
com.autorola.rbac.RBACUtil.createSessionForTrustedUser(RBACUtil.java:51)
at
com.autorola.fmsecurity.FMRBACUtil.createSubject(FMRBACUtil.java:40)
at
com.autorola.fmsecurity.FMRBACUtil.updateSubject(FMRBACUtil.java:20)
at
dk.autocom.auction.common.PresSessionInformation.setUser(PresSessionInformation.java:494)
at
dk.autocom.auction.actions.AuctionAction.doLogin(AuctionAction.java:1065)
at
dk.autocom.auction.actions.bilsalg.LoginAction.checkLogin(LoginAction.java:329)
at
dk.autocom.auction.actions.bilsalg.LoginAction.execute(LoginAction.java:88)
at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at
dk.autocom.auction.struts.AutocomRequestProcessor.processActionPerform(AutocomRequestProcessor.java:206)
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at
dk.autocom.auction.servlets.StrutsServlet.processRequest(StrutsServlet.java:753)
at
dk.autocom.auction.servlets.StrutsServlet.service(StrutsServlet.java:169)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
dk.autocom.lib.servletutils.CharsetFilter.doFilter(CharsetFilter.java:38)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:562)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
dk.autocom.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:210)
at
org.apache.catalina.connector.RemoteIpValve.invoke(RemoteIpValve.java:612)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:745)
2016-02-06 13:07:26,485 ERROR
[org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory]
Thread='http-0.0.0.0-14880-7' unable to bind connection: The response queue
has been emptied, no response was found.
2016-02-06 13:07:26,487 ERROR
[org.apache.directory.ldap.client.api.AbstractPoolableLdapConnectionFactory]
Thread='http-0.0.0.0-14880-7' unable to unbind connection: Cannot connect
on the server, the connection is invalid
.... waiting for 30 sec. ...
2016-02-06 13:07:56,488 ERROR
[org.apache.directory.ldap.client.api.LdapNetworkConnection]
Thread='http-0.0.0.0-14880-7' Bind failed : timeout occurred
2016-02-06 13:07:56,488 ERROR
[org.apache.directory.ldap.client.api.LdapNetworkConnection]
Thread='http-0.0.0.0-14880-7' The response queue has been emptied, no
response was found.
org.apache.directory.api.ldap.model.exception.LdapException: TimeOut
occurred
.... [Repeating ...] ...
Today it only repeated twice. I guess I was the only one logging into the
system yesterday.
For fortress config we use a max of 10 connections, and we use SSL.
// Jan
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Feb 6, 2016, at 1:53 PM, Jan Sindberg <ja...@gmail.com> wrote:
>
> When an application such as Fortress Commander is left inactive for hours -
> all connections in the connection pool is closed (timed out ?). But for
> each connection there is a 30 sec. timeout before the conclusion that the
> connection is dead. With 10 connections that is a five minutes wait for the
> first "unknowing victim" and an application-log with ten equal logs
> including stack trace.
A connection sometimes times out inside the client connection pool, on the server-side, or within the intermediaries (firewalls). First thing we’ll need is the stack trace and your fortress.properties, with necessary redactions of course, in order to (try and) pinpoint the root cause.
>
> On Feb 6, 2016, at 1:53 PM, Jan Sindberg <ja...@gmail.com> wrote:
>
> How are you dealing with this in your systems?
> Can I configure this differently from Fortress or in ApacheDS?
> Could we make Fortress be a little faster realising that the connection
> pool needs to be "refreshed"?
> I guess the issue could be simulated by restarting ApacheDS (I haven't
> tested this yet) - that should break whatever open connections there we.
There are settings in the client-side pooling mechanism that affect this behavior. There are code changes that can be made. There are settings on the server. But first we must understand why it’s happening in your tests.
Just to be clear - what you're describing here is very bad behavior for an application to have, and deserves serious attention from us. To put it simply, it’s a show-stopper.
Just now, I ran a couple of tests inside my local env. Using openldap as the server, and a max idle timeout of 30 seconds. After 30 seconds of inactivity, slapd will forcefully terminate any connections being held open by a client. I also had fortress web deployed to tomcat.
First I restarted slapd daemon with open browser session logged into the app. Next I left the web session open for around 10 minutes.
Neither test case resulted in a failure. For my tests, the ldap connection pool managed to recover.
I need to re-run the same tests using ApacheDS server, and will will once I learn how to set its max idle timeout.
But, just because I can’t recreate your issue inside my local env doesn’t tell us much. The good news, once we’ve got repeatable steps to recreate, this problem ought to get solved quickly.
Thanks for reporting it,
Shawn
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 11, 2016, at 6:40 PM, Chris Pike <cl...@psu.edu> wrote:
>
> We are using OpenLDAP, but adding the lines into our code seems to have fixed a few infrequent timeout errors. I don't understand the intracacies of the connections and the pooling, but it is probably worth waiting until the ldap api updates to the newer version of commons pool before spending a lot of time diagnosing the problems.
Agreed. Reproducibility is everything. Very hard to test these kinds of problems.
Shawn
Re: LDAP connection timeout
Posted by Chris Pike <cl...@psu.edu>.
We are using OpenLDAP, but adding the lines into our code seems to have fixed a few infrequent timeout errors. I don't understand the intracacies of the connections and the pooling, but it is probably worth waiting until the ldap api updates to the newer version of commons pool before spending a lot of time diagnosing the problems.
----- Original Message -----
From: "Shawn McKinney" <sm...@apache.org>
To: fortress@directory.apache.org
Sent: Friday, April 8, 2016 9:26:17 AM
Subject: Re: LDAP connection timeoutI
> On Apr 7, 2016, at 12:15 PM, Jan Sindberg <js...@autorola.com> wrote:
>
>> Fra: Shawn McKinney [mailto:smckinney@apache.org]
> ...
>> Jan, one of the advantages to being a volunteer is not having to apologize.
>
> Thanks, but I do prefer to answer when I get addressed :-) And I really, really wanted to do more that has been possible lately.
OK
>
> On Apr 7, 2016, at 12:15 PM, Jan Sindberg <js...@autorola.com> wrote:
>
> I am still very short om spare time, so I hope I can outsource completion of this task to some of you. What I can deliver here is a patch of what we have applied in our production which have run successfully since 9. Marts. Note that this is based on an early RC41.
> I can't deliver any automated test, it is either believe that it is a good thing that we don't get any complaints from our customers, or try to formalize what I have wrote earlier in this thread.
Same here. Since Chris brought it up, mebbie he can find the time.
>
> On Apr 7, 2016, at 12:15 PM, Jan Sindberg <js...@autorola.com> wrote:
>
> I think that I also wrote earlier that the patch is a qualified guess meaning that the solution works satisfactorily for us, but that it might be possible to tweak it a little more with more knowledge of for instance the real timeout which ApacheDS uses for connections (iff that is not controlled by some loadbalancer or elasticIp that our OP team has put in front, I don't know). Maybe timeout should be configurable? But the what should be our recommendation and advice on how to find optimum? Maybe it doesn't matter :-)
> Two months since I started this thread, and it feels like very long time B-)
>
> The patch is attached.
Did not see it. Prolly best to create a JIRA and attach it there.
Thanks,
Shawn
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 13, 2016, at 3:11 AM, Emmanuel Lécharny <el...@gmail.com> wrote:
>
> There is a problem in the way we manage roles for the Directory Project
> in JIRA. Apache infra has moved to a role-based administration model,
> and moved many projects to apply this scheme, but the directory projects.
>
> I have asked INFRA to have a look at it.
I also have a problem with JIRA. I too cannot edit tasks that I did not create, nor can I do project tasks like versions and releases. It is ironic that a role-based model is causing these problems. I wonder if we can help there here? ;-)
Shawn
Re: LDAP connection timeout
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 13/04/16 08:07, Stefan Seelmann a écrit :
> On 04/13/2016 01:33 AM, Shawn McKinney wrote:
>>> On Apr 12, 2016, at 1:49 PM, Jan Sindberg <js...@autorola.com> wrote:
>>>
>>>
>>> Can you do something so that I can edit a jira-issue? I have created a duplicate because the first one was not complete.
>>> Will it be better if I create a jira-user on my apache-email: sindberg@apache.org
>>> (I think that is it, haven't tried to use it so far.)
> @Shawn: You need to add Jan to the "Committers" and "Dewelopers" role in
> [1]. Even better would be to add him to the "directory-developers"
> group, but I don't who can edit it, probably Kiran and Emmanuel.
>
> [1] https://issues.apache.org/jira/plugins/servlet/project-config/FC/roles
>
>> Jan, you should use the credentials that were supplied to you when you first were given access to the project. Should be the same as what you use for GIT access.
> AFAIK, Jira is not connected to LDAP, so you have to use a separate
> password.
>
> Kind Regards,
> Stefan
>
There is a problem in the way we manage roles for the Directory Project
in JIRA. Apache infra has moved to a role-based administration model,
and moved many projects to apply this scheme, but the directory projects.
I have asked INFRA to have a look at it.
Re: LDAP connection timeout
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 04/13/2016 01:33 AM, Shawn McKinney wrote:
>
>> On Apr 12, 2016, at 1:49 PM, Jan Sindberg <js...@autorola.com> wrote:
>>
>>
>> Can you do something so that I can edit a jira-issue? I have created a duplicate because the first one was not complete.
>> Will it be better if I create a jira-user on my apache-email: sindberg@apache.org
>> (I think that is it, haven't tried to use it so far.)
@Shawn: You need to add Jan to the "Committers" and "Dewelopers" role in
[1]. Even better would be to add him to the "directory-developers"
group, but I don't who can edit it, probably Kiran and Emmanuel.
[1] https://issues.apache.org/jira/plugins/servlet/project-config/FC/roles
> Jan, you should use the credentials that were supplied to you when you first were given access to the project. Should be the same as what you use for GIT access.
AFAIK, Jira is not connected to LDAP, so you have to use a separate
password.
Kind Regards,
Stefan
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 12, 2016, at 1:49 PM, Jan Sindberg <js...@autorola.com> wrote:
>
>
> Can you do something so that I can edit a jira-issue? I have created a duplicate because the first one was not complete.
> Will it be better if I create a jira-user on my apache-email: sindberg@apache.org
> (I think that is it, haven't tried to use it so far.)
Jan, you should use the credentials that were supplied to you when you first were given access to the project. Should be the same as what you use for GIT access.
Re: Sv: LDAP connection timeout
Posted by Jan Sindberg <ja...@gmail.com>.
I get "You do not have permission to edit issues in this project."
2016-04-13 16:01 GMT+02:00 Emmanuel Lécharny <el...@gmail.com>:
> Le 12/04/16 20:49, Jan Sindberg a écrit :
> > Fra: Shawn McKinney <sm...@apache.org>
> > Sendt: 8. april 2016 15:26
> >
> > ...
> >> Did not see it. Prolly best to create a JIRA and attach it there.
> > ...
> > https://issues.apache.org/jira/browse/FC-167
> >
> > Can you do something so that I can edit a jira-issue? I have created a
> duplicate because the first one was not complete.
> > Will it be better if I create a jira-user on my apache-email:
> sindberg@apache.org
> > (I think that is it, haven't tried to use it so far.)
> I have added you to the list of committers. You should be able to update
> the JIRA ticket. Can you give it a try and tell me if it's OK ?
>
Re: Sv: LDAP connection timeout
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 12/04/16 20:49, Jan Sindberg a écrit :
> Fra: Shawn McKinney <sm...@apache.org>
> Sendt: 8. april 2016 15:26
>
> ...
>> Did not see it. Prolly best to create a JIRA and attach it there.
> ...
> https://issues.apache.org/jira/browse/FC-167
>
> Can you do something so that I can edit a jira-issue? I have created a duplicate because the first one was not complete.
> Will it be better if I create a jira-user on my apache-email: sindberg@apache.org
> (I think that is it, haven't tried to use it so far.)
I have added you to the list of committers. You should be able to update
the JIRA ticket. Can you give it a try and tell me if it's OK ?
Sv: LDAP connection timeout
Posted by Jan Sindberg <js...@autorola.com>.
Fra: Shawn McKinney <sm...@apache.org>
Sendt: 8. april 2016 15:26
...
> Did not see it. Prolly best to create a JIRA and attach it there.
...
https://issues.apache.org/jira/browse/FC-167
Can you do something so that I can edit a jira-issue? I have created a duplicate because the first one was not complete.
Will it be better if I create a jira-user on my apache-email: sindberg@apache.org
(I think that is it, haven't tried to use it so far.)
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 7, 2016, at 12:15 PM, Jan Sindberg <js...@autorola.com> wrote:
>
>> Fra: Shawn McKinney [mailto:smckinney@apache.org]
> ...
>> Jan, one of the advantages to being a volunteer is not having to apologize.
>
> Thanks, but I do prefer to answer when I get addressed :-) And I really, really wanted to do more that has been possible lately.
OK
>
> On Apr 7, 2016, at 12:15 PM, Jan Sindberg <js...@autorola.com> wrote:
>
> I am still very short om spare time, so I hope I can outsource completion of this task to some of you. What I can deliver here is a patch of what we have applied in our production which have run successfully since 9. Marts. Note that this is based on an early RC41.
> I can't deliver any automated test, it is either believe that it is a good thing that we don't get any complaints from our customers, or try to formalize what I have wrote earlier in this thread.
Same here. Since Chris brought it up, mebbie he can find the time.
>
> On Apr 7, 2016, at 12:15 PM, Jan Sindberg <js...@autorola.com> wrote:
>
> I think that I also wrote earlier that the patch is a qualified guess meaning that the solution works satisfactorily for us, but that it might be possible to tweak it a little more with more knowledge of for instance the real timeout which ApacheDS uses for connections (iff that is not controlled by some loadbalancer or elasticIp that our OP team has put in front, I don't know). Maybe timeout should be configurable? But the what should be our recommendation and advice on how to find optimum? Maybe it doesn't matter :-)
> Two months since I started this thread, and it feels like very long time B-)
>
> The patch is attached.
Did not see it. Prolly best to create a JIRA and attach it there.
Thanks,
Shawn
SV: LDAP connection timeout
Posted by Jan Sindberg <js...@autorola.com>.
> Fra: Shawn McKinney [mailto:smckinney@apache.org]
...
> Jan, one of the advantages to being a volunteer is not having to apologize.
Thanks, but I do prefer to answer when I get addressed :-) And I really, really wanted to do more that has been possible lately.
> Volunteering necessarily means it will get done when it gets done. Now wrt
> work quality, that’s another matter entirely. It has to be good on delivery.
> So take your time, make it right. :-)
I am still very short om spare time, so I hope I can outsource completion of this task to some of you. What I can deliver here is a patch of what we have applied in our production which have run successfully since 9. Marts. Note that this is based on an early RC41.
I can't deliver any automated test, it is either believe that it is a good thing that we don't get any complaints from our customers, or try to formalize what I have wrote earlier in this thread.
I think that I also wrote earlier that the patch is a qualified guess meaning that the solution works satisfactorily for us, but that it might be possible to tweak it a little more with more knowledge of for instance the real timeout which ApacheDS uses for connections (iff that is not controlled by some loadbalancer or elasticIp that our OP team has put in front, I don't know). Maybe timeout should be configurable? But the what should be our recommendation and advice on how to find optimum? Maybe it doesn't matter :-)
Two months since I started this thread, and it feels like very long time B-)
The patch is attached.
Best regards - Jan
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 7, 2016, at 2:35 AM, Jan Sindberg <js...@autorola.com> wrote:
>
> Sorry. I have been away on courses and generally been too busy. Will answer in roughly 12 hours.
Jan, one of the advantages to being a volunteer is not having to apologize. Volunteering necessarily means it will get done when it gets done. Now wrt work quality, that’s another matter entirely. It has to be good on delivery. So take your time, make it right. :-)
SV: LDAP connection timeout
Posted by Jan Sindberg <js...@autorola.com>.
Sorry. I have been away on courses and generally been too busy. Will answer in roughly 12 hours.
> -----Oprindelig meddelelse-----
> Fra: Shawn McKinney [mailto:smckinney@apache.org]
> Sendt: 6. april 2016 16:56
> Til: fortress@directory.apache.org
> Emne: Re: LDAP connection timeout
>
>
> > On Apr 5, 2016, at 8:52 AM, Chris Pike <cl...@psu.edu> wrote:
> >
> > Are we going to add this fix (setTestWhileIdle and
> setTimeBetweenEvictionRunsMillis) into the source?
> >
>
> Possibly. I need to have a better understanding of what it’s doing. Could
> start with the code. Jan, where is it, is it ready for a review?
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 5, 2016, at 8:52 AM, Chris Pike <cl...@psu.edu> wrote:
>
> Are we going to add this fix (setTestWhileIdle and setTimeBetweenEvictionRunsMillis) into the source?
>
Possibly. I need to have a better understanding of what it’s doing. Could start with the code. Jan, where is it, is it ready for a review?
Re: LDAP connection timeout
Posted by Chris Pike <cl...@psu.edu>.
Are we going to add this fix (setTestWhileIdle and setTimeBetweenEvictionRunsMillis) into the source?
----- Original Message -----
From: "Jan Sindberg" <ja...@gmail.com>
To: fortress@directory.apache.org
Sent: Friday, February 12, 2016 3:52:17 PM
Subject: Re: LDAP connection timeout
>
>
> Every 30 minutes this will test three (3 is default) idle threads and
> evict them if they fail an attempt of a simple lookup on the rootDSE.
> I have not yet established how long time it takes before a connection is
> stale
> I hope 30 minutes is enough - I suspect that the test will keep the
> connections alive if the time is too short.
>
>
This is enough to mitigate the issue with long waiting time for the first
user in the morning.
// Jan
Re: LDAP connection timeout
Posted by Jan Sindberg <ja...@gmail.com>.
>
>
> Every 30 minutes this will test three (3 is default) idle threads and
> evict them if they fail an attempt of a simple lookup on the rootDSE.
> I have not yet established how long time it takes before a connection is
> stale
> I hope 30 minutes is enough - I suspect that the test will keep the
> connections alive if the time is too short.
>
>
This is enough to mitigate the issue with long waiting time for the first
user in the morning.
// Jan
Re: LDAP connection timeout
Posted by Jan Sindberg <ja...@gmail.com>.
2016-02-11 1:38 GMT+01:00 Shawn McKinney <sm...@apache.org>:
>
> > On Feb 10, 2016, at 8:29 AM, Jan Sindberg <js...@autorola.com> wrote:
> >
> > Summary:
> > If we configure "testWhileIdle" and " timeBetweenEvictionRunsMillis" it
> should activate the LookupLdapConnectionValidator and reduce the issue. We
> should as well consider the other pool-settings for appropriate defaults.
>
> Jan, we need test case(s) to illustrate these fault conditions. Doesn’t
> have to be a junit test necessarily, but a set of steps that can be
> verified - somehow. After that we’ll create a JIRA ticket and go from
> there.
>
> Thanks,
>
> Shawn
The test case, roughly;
Set up a commander connected to ApacheDS. Log in and refresh a lot to
activate a couple of connections. Log out and let it run over night. When
you log in the next day, you will see the delay.
We have applied the following in ApacheDsDataProvider:
adminPool.setTestWhileIdle(true);
adminPool.setTimeBetweenEvictionRunsMillis(1000 * 60 * 30);
...
userPool.setTestWhileIdle(true);
userPool.setTimeBetweenEvictionRunsMillis(1000*60*30);
...
logPool.setTestWhileIdle(true);
logPool.setTimeBetweenEvictionRunsMillis(1000 * 60 * 30);
Every 30 minutes this will test three (3 is default) idle threads and evict
them if they fail an attempt of a simple lookup on the rootDSE.
I have not yet established how long time it takes before a connection is
stale
I hope 30 minutes is enough - I suspect that the test will keep the
connections alive if the time is too short.
// Jan
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Feb 10, 2016, at 8:29 AM, Jan Sindberg <js...@autorola.com> wrote:
>
> Summary:
> If we configure "testWhileIdle" and " timeBetweenEvictionRunsMillis" it should activate the LookupLdapConnectionValidator and reduce the issue. We should as well consider the other pool-settings for appropriate defaults.
Jan, we need test case(s) to illustrate these fault conditions. Doesn’t have to be a junit test necessarily, but a set of steps that can be verified - somehow. After that we’ll create a JIRA ticket and go from there.
Thanks,
Shawn
SV: LDAP connection timeout
Posted by Jan Sindberg <js...@autorola.com>.
> The real pb is that it seems the delay depends on the number of existing
> connections, which is not good (we have to check that in the pool : do we
> switch to a new connection when one has timed out ? What if we set 5 s but
> have 100 connectiuons in the pool ? I guess it would be good to stop trying
> after a few attemps).
I guess that it should be fixed in
org.apache.directory.fortress.core.ldap.ApacheDsDataProvider ?
It uses
org.apache.directory.ldap.client.api.LdapConnectionConfig where a timeout can be set.
We also use
https://commons.apache.org/proper/commons-pool/api-1.6/org/apache/commons/pool/impl/GenericObjectPool.html
We currently sets maxIdle to infinite
userPool.setMaxIdle( -1 );
Setting a default as a percent of max connections could help?
We can take a look at testOnBorrow :
"When testOnBorrow is set, the pool will attempt to validate each object before it is returned from the borrowObject() method. (Using the provided factory's PoolableObjectFactory.validateObject(T) method.) Objects that fail to validate will be dropped from the pool, and a different object will be borrowed. The default setting for this parameter is false."
Optionally, one may configure the pool to examine and possibly evict objects as they sit idle in the pool and to ensure that a minimum number of idle objects are available. This is performed by an "idle object eviction" thread, which runs asynchronously. Caution should be used when configuring this optional feature. Eviction runs contend with client threads for access to objects in the pool, so if they run too frequently performance issues may result. The idle object eviction thread may be configured using the following attributes:
- timeBetweenEvictionRunsMillis indicates how long the eviction thread should sleep before "runs" of examining idle objects. When non-positive, no eviction thread will be launched. The default setting for this parameter is -1 (i.e., idle object eviction is disabled by default).
- minEvictableIdleTimeMillis specifies the minimum amount of time that an object may sit idle in the pool before it is eligible for eviction due to idle time. When non-positive, no object will be dropped from the pool due to idle time alone. This setting has no effect unless timeBetweenEvictionRunsMillis > 0. The default setting for this parameter is 30 minutes.
- testWhileIdle indicates whether or not idle objects should be validated using the factory's PoolableObjectFactory.validateObject(T) method. Objects that fail to validate will be dropped from the pool. This setting has no effect unless timeBetweenEvictionRunsMillis > 0. The default setting for this parameter is false.
- softMinEvictableIdleTimeMillis specifies the minimum amount of time an object may sit idle in the pool before it is eligible for eviction by the idle object evictor (if any), with the extra condition that at least "minIdle" object instances remain in the pool. When non-positive, no objects will be evicted from the pool due to idle time alone. This setting has no effect unless timeBetweenEvictionRunsMillis > 0. and it is superceded by minEvictableIdleTimeMillis (that is, if minEvictableIdleTimeMillis is positive, then softMinEvictableIdleTimeMillis is ignored). The default setting for this parameter is -1 (disabled).
- numTestsPerEvictionRun determines the number of objects examined in each run of the idle object evictor. This setting has no effect unless timeBetweenEvictionRunsMillis > 0. The default setting for this parameter is 3.
The LdapConnection has a method "isConnected()" - maybe that could be used in testOnBorrow, if it does as hoped for.
https://directory.apache.org/api/gen-docs/latest/apidocs/org/apache/directory/ldap/client/api/LdapConnection
The LdapConnection also has a method setTimeOut - timeout for responses - which might or might not have an effect.
Some test using isConnected() are already implemented in
Org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionFactory.
We can add a validator with setValidator - it seem that there already is a default LookupLdapConnectionValidator set if just validation has been configured:
public void setValidator(LdapConnectionValidator validator)
Sets the validator to use when validation occurs. Note that validation will only occur if the connection pool was configured to validate. This means one of:
- setTestOnBorrow
- setTestWhileIdle
- setTestOnReturn
must have been set to true on the pool. The default validator is LookupLdapConnectionValidator.
LdapConnectionValidator
- An LdapConnection validator intended to be used by a GenericObjectPool to determine whether or not a conneciton is still usable.
There are already two validators available:
DefaultLdapConnectionValidator
- An implementation of LdapConnectionValidator that checks to see that the connection isConnected() and isAuthenticated().
LookupLdapConnectionValidator (The default)
- An implementation of LdapConnectionValidator that attempts a simple lookup on the rootDSE.
Summary:
If we configure "testWhileIdle" and " timeBetweenEvictionRunsMillis" it should activate the LookupLdapConnectionValidator and reduce the issue. We should as well consider the other pool-settings for appropriate defaults.
// Jan
Re: LDAP connection timeout
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 08/02/16 18:49, Shawn McKinney a écrit :
>> On Feb 7, 2016, at 1:42 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
>>
>> Le 06/02/16 20:53, Jan Sindberg a écrit :
>>> When an application such as Fortress Commander is left inactive for hours -
>>> all connections in the connection pool is closed (timed out ?). But for
>>> each connection there is a 30 sec.
>> This is the default LDAP API timeout. This is configurable.
> 30 seconds too long. What’s a good setting for this - 1 second?
Whatever fits your need. 1s might be a bit short, though.
The real pb is that it seems the delay depends on the number of existing
connections, which is not good (we have to check that in the pool : do
we switch to a new connection when one has timed out ? What if we set 5
s but have 100 connectiuons in the pool ? I guess it would be good to
stop trying after a few attemps).
Re: LDAP connection timeout
Posted by Shawn McKinney <sm...@apache.org>.
> On Feb 7, 2016, at 1:42 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
>
> Le 06/02/16 20:53, Jan Sindberg a écrit :
>> When an application such as Fortress Commander is left inactive for hours -
>> all connections in the connection pool is closed (timed out ?). But for
>> each connection there is a 30 sec.
> This is the default LDAP API timeout. This is configurable.
30 seconds too long. What’s a good setting for this - 1 second?
> On Feb 7, 2016, at 1:42 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
>
>> timeout before the conclusion that the
>> connection is dead. With 10 connections that is a five minutes wait for the
>> first "unknowing victim" and an application-log with ten equal logs
>> including stack trace.
>> How are you dealing with this in your systems?
>> Can I configure this differently from Fortress or in ApacheDS?
>> Could we make Fortress be a little faster realising that the connection
>> pool needs to be "refreshed"?
>> I guess the issue could be simulated by restarting ApacheDS (I haven't
>> tested this yet) - that should break whatever open connections there we.
>
> We can improve the way the pool detect dead connections.
How?
Shawn
Re: LDAP connection timeout
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 06/02/16 20:53, Jan Sindberg a écrit :
> When an application such as Fortress Commander is left inactive for hours -
> all connections in the connection pool is closed (timed out ?). But for
> each connection there is a 30 sec.
This is the default LDAP API timeout. This is configurable.
> timeout before the conclusion that the
> connection is dead. With 10 connections that is a five minutes wait for the
> first "unknowing victim" and an application-log with ten equal logs
> including stack trace.
> How are you dealing with this in your systems?
> Can I configure this differently from Fortress or in ApacheDS?
> Could we make Fortress be a little faster realising that the connection
> pool needs to be "refreshed"?
> I guess the issue could be simulated by restarting ApacheDS (I haven't
> tested this yet) - that should break whatever open connections there we.
We can improve the way the pool detect dead connections.