You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rene Moser <ma...@renemoser.net> on 2016/03/03 13:53:11 UTC
LDAP auth failures
We are experiencing authentication issues with LDAP since upgrade to 4.5.1.
After some time (...), users can not authenticate anymore, however,
authentication in other services using ldap works during this time. The
issue is only related to cloudstack login it seems.
We haven't found the root cause yet, a network setup issue or openldap
config issue can not be excluded.
Stacktrace:
2016-02-29 10:05:36,375 DEBUG [cloudstack.ldap.LdapContextFactory]
(catalina-exec-4:ctx-9ffa7c60) initializing ldap with provider url:
ldap://ldap.example.com:389
2016-02-29 10:05:42,382 DEBUG [cloudstack.ldap.LdapManagerImpl]
(catalina-exec-4:ctx-9ffa7c60) ldap Exception:
javax.naming.NamingException: LDAP response read timed out, timeout
used:6000ms.; remaining name 'dc=foo,dc=bar'
at com.sun.jndi.ldap.Connection.readReply(Connection.java:485)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
at
org.apache.cloudstack.ldap.LdapUserManager.searchUsers(LdapUserManager.java:206)
at
org.apache.cloudstack.ldap.LdapUserManager.getUser(LdapUserManager.java:122)
at
org.apache.cloudstack.ldap.LdapManagerImpl.getUser(LdapManagerImpl.java:173)
at
org.apache.cloudstack.ldap.LdapManagerImpl.canAuthenticate(LdapManagerImpl.java:97)
at
org.apache.cloudstack.ldap.LdapAuthenticator.authenticate(LdapAuthenticator.java:61)
2016-02-29 10:05:42,383 DEBUG [cloudstack.ldap.LdapManagerImpl]
(catalina-exec-4:ctx-9ffa7c60) Exception while doing an LDAP bind for
user johndoe
org.apache.cloudstack.ldap.NoLdapUserMatchingQueryException: No users
matching: No Ldap User found for username: johndoe
As I understand there is a username lookup (bind with top reader
credentials) to see if a user exists in the ldap. if found a new
connection will be etablished for auth. In the above stacktrace it seem
that the username lookup fails.
Further we see on the ACS management server however, is that LDAP
connection are not going to be closed at any time.
For _every_ successful auth, the tcp connection remains established forever.
In my understanding of
http://docs.oracle.com/javase/jndi/tutorial/ldap/connect/config.html
these connections will become idle after successful authentication and
reused for new authentication.
However, the reuse for the auth doesn't seem to work. _Every_ new
successful auth of a user _creates_ a new ldap connection. We don't know
if this is related to our problem, but at least it doesn't look like a
wanted behavior.
In the docs we read: "By default, idle connections remain in the pool
indefinitely until they are garbage-collected"
But as said, they seem never be gc-ed. After we added
-Dcom.sun.jndi.ldap.connect.pool.timeout=60000 to the
/etc/cloudstack/management/tomcat6.conf which resulted in the
connections beeing gc-ed and we didn't have any report about missing
login since then.
Has anyone also see such an issue? Any thoughts?
René
Re: LDAP auth failures
Posted by Pierre-Luc Dion <pd...@cloudops.com>.
We have a system currently running 4.7.0 which is connected to an
Active-Directory thru LDAP and we had no issues so far.
it got upgraded from: 4.2.1 to 4.6.0 to 4.7.0
Ilya, does your issues could be related to LDAP changes required from an
upgrade:
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.4.4/upgrade/upgrade_notes.html#active-directory-authentication-ldap
PL
On Wed, Mar 9, 2016 at 4:45 AM, Rene Moser <ma...@renemoser.net> wrote:
> Hi
>
> On 03/09/2016 06:33 AM, Abhinandan Prateek wrote:
> > In 4.5 there is a timeout param that was added ‘ldap.read.timeout’ that
> > defaults to 1000,
> > It should be set to about 6000 and that should resolve the read timeout
> > that you guys see.
>
> we already set it to 6000 (and more) as you can see here:
>
> 2016-02-29 10:05:42,382 DEBUG [cloudstack.ldap.LdapManagerImpl]
> (catalina-exec-4:ctx-9ffa7c60) ldap Exception:
> javax.naming.NamingException: LDAP response read timed out, timeout
> used:6000ms.; remaining name 'dc=foo,dc=bar'
>
>
> The only thing that "solved" the problem was
>
> -Dcom.sun.jndi.ldap.connect.pool.timeout=60000
>
> We have a suspicion that the issue is related somehow in our ldap setup
> (tcp, or openldap config) in relation to the connection pooling in
> cloudstack.
>
> René
>
>
>
Re: LDAP auth failures
Posted by Rene Moser <ma...@renemoser.net>.
Hi
On 03/09/2016 06:33 AM, Abhinandan Prateek wrote:
> In 4.5 there is a timeout param that was added ‘ldap.read.timeout’ that
> defaults to 1000,
> It should be set to about 6000 and that should resolve the read timeout
> that you guys see.
we already set it to 6000 (and more) as you can see here:
2016-02-29 10:05:42,382 DEBUG [cloudstack.ldap.LdapManagerImpl]
(catalina-exec-4:ctx-9ffa7c60) ldap Exception:
javax.naming.NamingException: LDAP response read timed out, timeout
used:6000ms.; remaining name 'dc=foo,dc=bar'
The only thing that "solved" the problem was
-Dcom.sun.jndi.ldap.connect.pool.timeout=60000
We have a suspicion that the issue is related somehow in our ldap setup
(tcp, or openldap config) in relation to the connection pooling in
cloudstack.
René
Re: LDAP auth failures
Posted by Abhinandan Prateek <ab...@shapeblue.com>.
In 4.5 there is a timeout param that was added ‘ldap.read.timeout’ that defaults to 1000,
It should be set to about 6000 and that should resolve the read timeout that you guys see.
[ShapeBlue]<http://www.shapeblue.com>
Abhinandan Prateek
Software Architect , ShapeBlue
d: | s: +44 203 603 0540<tel:|%20s:%20+44%20203%20603%200540> | m: +91 970 11 99011<tel:+91%20970%2011%2099011>
e: abhinandan.prateek@shapeblue.com | t: <mailto:abhinandan.prateek@shapeblue.com%20|%20t:> | w: www.shapeblue.com<http://www.shapeblue.com>
a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
[cid:image631b15.png@e7854a64.46aae3f8]
Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.
On 09/03/16, 3:19 AM, "ilya" <il...@gmail.com> wrote:
>I could not get LDAP to work as well in 4.5.x, i could get it to work in 4.3
>
>I also get no stacktrace as to what could be wrong.
>
>
>
>On 3/3/16 4:53 AM, Rene Moser wrote:
>> We are experiencing authentication issues with LDAP since upgrade to 4.5.1.
>>
>> After some time (...), users can not authenticate anymore, however,
>> authentication in other services using ldap works during this time. The
>> issue is only related to cloudstack login it seems.
>>
>> We haven't found the root cause yet, a network setup issue or openldap
>> config issue can not be excluded.
>>
>> Stacktrace:
>>
>> 2016-02-29 10:05:36,375 DEBUG [cloudstack.ldap.LdapContextFactory]
>> (catalina-exec-4:ctx-9ffa7c60) initializing ldap with provider url:
>> ldap://ldap.example.com:389
>> 2016-02-29 10:05:42,382 DEBUG [cloudstack.ldap.LdapManagerImpl]
>> (catalina-exec-4:ctx-9ffa7c60) ldap Exception:
>> javax.naming.NamingException: LDAP response read timed out, timeout
>> used:6000ms.; remaining name 'dc=foo,dc=bar'
>> at com.sun.jndi.ldap.Connection.readReply(Connection.java:485)
>> at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)
>> at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)
>> at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
>> at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
>> at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
>> at
>> org.apache.cloudstack.ldap.LdapUserManager.searchUsers(LdapUserManager.java:206)
>> at
>> org.apache.cloudstack.ldap.LdapUserManager.getUser(LdapUserManager.java:122)
>> at
>> org.apache.cloudstack.ldap.LdapManagerImpl.getUser(LdapManagerImpl.java:173)
>> at
>> org.apache.cloudstack.ldap.LdapManagerImpl.canAuthenticate(LdapManagerImpl.java:97)
>> at
>> org.apache.cloudstack.ldap.LdapAuthenticator.authenticate(LdapAuthenticator.java:61)
>> 2016-02-29 10:05:42,383 DEBUG [cloudstack.ldap.LdapManagerImpl]
>> (catalina-exec-4:ctx-9ffa7c60) Exception while doing an LDAP bind for
>> user johndoe
>> org.apache.cloudstack.ldap.NoLdapUserMatchingQueryException: No users
>> matching: No Ldap User found for username: johndoe
>>
>> As I understand there is a username lookup (bind with top reader
>> credentials) to see if a user exists in the ldap. if found a new
>> connection will be etablished for auth. In the above stacktrace it seem
>> that the username lookup fails.
>>
>> Further we see on the ACS management server however, is that LDAP
>> connection are not going to be closed at any time.
>>
>> For _every_ successful auth, the tcp connection remains established forever.
>>
>> In my understanding of
>> http://docs.oracle.com/javase/jndi/tutorial/ldap/connect/config.html
>> these connections will become idle after successful authentication and
>> reused for new authentication.
>>
>> However, the reuse for the auth doesn't seem to work. _Every_ new
>> successful auth of a user _creates_ a new ldap connection. We don't know
>> if this is related to our problem, but at least it doesn't look like a
>> wanted behavior.
>>
>> In the docs we read: "By default, idle connections remain in the pool
>> indefinitely until they are garbage-collected"
>>
>> But as said, they seem never be gc-ed. After we added
>> -Dcom.sun.jndi.ldap.connect.pool.timeout=60000 to the
>> /etc/cloudstack/management/tomcat6.conf which resulted in the
>> connections beeing gc-ed and we didn't have any report about missing
>> login since then.
>>
>> Has anyone also see such an issue? Any thoughts?
>>
>> René
>>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
Re: LDAP auth failures
Posted by ilya <il...@gmail.com>.
I could not get LDAP to work as well in 4.5.x, i could get it to work in 4.3
I also get no stacktrace as to what could be wrong.
On 3/3/16 4:53 AM, Rene Moser wrote:
> We are experiencing authentication issues with LDAP since upgrade to 4.5.1.
>
> After some time (...), users can not authenticate anymore, however,
> authentication in other services using ldap works during this time. The
> issue is only related to cloudstack login it seems.
>
> We haven't found the root cause yet, a network setup issue or openldap
> config issue can not be excluded.
>
> Stacktrace:
>
> 2016-02-29 10:05:36,375 DEBUG [cloudstack.ldap.LdapContextFactory]
> (catalina-exec-4:ctx-9ffa7c60) initializing ldap with provider url:
> ldap://ldap.example.com:389
> 2016-02-29 10:05:42,382 DEBUG [cloudstack.ldap.LdapManagerImpl]
> (catalina-exec-4:ctx-9ffa7c60) ldap Exception:
> javax.naming.NamingException: LDAP response read timed out, timeout
> used:6000ms.; remaining name 'dc=foo,dc=bar'
> at com.sun.jndi.ldap.Connection.readReply(Connection.java:485)
> at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)
> at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)
> at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
> at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
> at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
> at
> org.apache.cloudstack.ldap.LdapUserManager.searchUsers(LdapUserManager.java:206)
> at
> org.apache.cloudstack.ldap.LdapUserManager.getUser(LdapUserManager.java:122)
> at
> org.apache.cloudstack.ldap.LdapManagerImpl.getUser(LdapManagerImpl.java:173)
> at
> org.apache.cloudstack.ldap.LdapManagerImpl.canAuthenticate(LdapManagerImpl.java:97)
> at
> org.apache.cloudstack.ldap.LdapAuthenticator.authenticate(LdapAuthenticator.java:61)
> 2016-02-29 10:05:42,383 DEBUG [cloudstack.ldap.LdapManagerImpl]
> (catalina-exec-4:ctx-9ffa7c60) Exception while doing an LDAP bind for
> user johndoe
> org.apache.cloudstack.ldap.NoLdapUserMatchingQueryException: No users
> matching: No Ldap User found for username: johndoe
>
> As I understand there is a username lookup (bind with top reader
> credentials) to see if a user exists in the ldap. if found a new
> connection will be etablished for auth. In the above stacktrace it seem
> that the username lookup fails.
>
> Further we see on the ACS management server however, is that LDAP
> connection are not going to be closed at any time.
>
> For _every_ successful auth, the tcp connection remains established forever.
>
> In my understanding of
> http://docs.oracle.com/javase/jndi/tutorial/ldap/connect/config.html
> these connections will become idle after successful authentication and
> reused for new authentication.
>
> However, the reuse for the auth doesn't seem to work. _Every_ new
> successful auth of a user _creates_ a new ldap connection. We don't know
> if this is related to our problem, but at least it doesn't look like a
> wanted behavior.
>
> In the docs we read: "By default, idle connections remain in the pool
> indefinitely until they are garbage-collected"
>
> But as said, they seem never be gc-ed. After we added
> -Dcom.sun.jndi.ldap.connect.pool.timeout=60000 to the
> /etc/cloudstack/management/tomcat6.conf which resulted in the
> connections beeing gc-ed and we didn't have any report about missing
> login since then.
>
> Has anyone also see such an issue? Any thoughts?
>
> René
>