You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@archiva.apache.org by Deng Ching <oc...@apache.org> on 2011/08/24 08:45:16 UTC
Archiva issue with LDAP (MRM-1488)
Hi,
Jev and I are currently working on MRM-1488 (Using Archiva as proxy for
downloading artifacts is very slow when used with LDAP authentication). The
problem we saw here was that for each artifact request, authentication
happens which in turn results to a call to the LDAP server.
To fix this, we're planning to use an in-memory cache for the LDAP
credentials. For every authentication request made, Redback will:
1. look for the user in the cache
2. if user is found in the cache, compare credentials
- if it matches, return successful
- if it doesn't match, reject it
3. if user is not found in the cache
- retrieve user from LDAP server
- check provided credentials against the retrieved user
- if it matches, return successful and add retrieved user to the cache
- if it doesn't match, reject it
We're planning to use EhCache for this so we can also set a TTL
(time-to-live) for the cached objects. A password change done from the
webapp would flush the user in the cache.
Also, in the logs, we've noticed a number of calls to the LDAP server
(search for users then authenticate) which may not be necessary so we'll
also check if we can improve those.
Any thoughts or comments?
Thanks,
Deng
Re: Archiva issue with LDAP (MRM-1488)
Posted by Deng Ching <oc...@apache.org>.
Made a few more changes to the fix. I removed the caching of the ldap
connection as that might also pose a security issue. I configured connection
pooling when Sun's ldap context factory is used (thanks Brent!) then just
cached the ldap users' userDn to minimize the lookups to the LDAP server
during authn bind.
With these changes in, 'clean install' on archiva-parent now took 7:03.851s
in my local.
-Deng
On Fri, Aug 26, 2011 at 12:04 PM, Deng Ching <oc...@apache.org> wrote:
> It was the former Brett. Right, I didn't think about closing the
> resources.. thanks for pointing that out :)
>
> I'll do some testing on this. I'm actually thinking about reverting back
> the authentication part to not use the cache and only have the ldap users
> cached, I have to check though how big the difference would be in the builds
> if that would be the case since I think it's the find/search for user that's
> taking a while.
>
> Thanks,
> Deng
>
> On Fri, Aug 26, 2011 at 11:47 AM, Brett Porter <br...@apache.org> wrote:
>
>> Did you do this with ehcache or the technique Brent outlined? If it's the
>> former, I'm worried about it not closing the resources - we should test it
>> with a lot of concurrent different users.
>>
>> On 26/08/2011, at 12:57 PM, Deng Ching wrote:
>>
>> > I made some changes to the impl, btw. Instead of just caching the ldap
>> > users, I've also cached the ldap connections. Not all ldap servers
>> return a
>> > hashed password (some return just a masked string, eg. ******) for the
>> > userPassword attribute of an ldap user so we can't do a comparison on
>> it.
>> > You need to bind to the ldap server to authenticate, so I just cached
>> the
>> > ldap connection of a user. For the ldap connections, I've set the TTL to
>> > 15secs., then 2 mins. TTL for the ldap users.
>> >
>> > I ran a 'clean install' on archiva-parent against an Archiva repo using
>> JDO
>> > and LDAP for authentication, and these are the results:
>> > - JDO: 7:04.998s
>> > - LDAP: 7:17.382s
>> >
>> > Thanks,
>> > Deng
>> >
>> > On Thu, Aug 25, 2011 at 10:07 AM, Deng Ching <oc...@apache.org> wrote:
>> >
>> >> On Thu, Aug 25, 2011 at 1:44 AM, Brent Atkinson <
>> brent.atkinson@gmail.com>wrote:
>> >>
>> >>> Hi everyone,
>> >>>
>> >>> I actually ran into this when fixing the connection leaks. I realized
>> it
>> >>> was
>> >>> probably building in too many assumptions, but I created and held onto
>> the
>> >>> LdapCtxFactory in redback's LdapConnection for a very specific reason:
>> >>> connection pooling. The sun JNDI ldap implementation can pool
>> connections
>> >>> sharing the same credentials *and config options* as long as they are
>> >>> created from the same LdapCtxFactory.
>> >>>
>> >>>
>> http://download.oracle.com/javase/jndi/tutorial/ldap/connect/pool.html
>> >>>
>> >>>
>> >> Thanks Brent! We'll look into that.
>> >>
>> >>
>> >>> On Wed, Aug 24, 2011 at 8:57 AM, Wendy Smoak <ws...@gmail.com>
>> wrote:
>> >>>
>> >>>> On Wed, Aug 24, 2011 at 2:45 AM, Deng Ching <oc...@apache.org>
>> wrote:
>> >>>>
>> >>>>> We're planning to use EhCache for this so we can also set a TTL
>> >>>>> (time-to-live) for the cached objects. A password change done from
>> the
>> >>>>> webapp would flush the user in the cache.
>> >>>>
>> >>>> If you're using LDAP, would users be doing password changes from the
>> >>>> webapp?
>> >>>>
>> >>>> Making that TTL configurable by the admin would be good, then they
>> can
>> >>>> trade off between extra calls to LDAP and 'how come my new password
>> >>>> doesn't work?'.
>> >>>
>> >>
>> >> Agreed. We'll add this functionality as well :)
>> >>
>> >> Thanks,
>> >> Deng
>> >>
>>
>> --
>> Brett Porter
>> brett@apache.org
>> http://brettporter.wordpress.com/
>> http://au.linkedin.com/in/brettporter
>>
>>
>>
>>
>>
>
Re: Archiva issue with LDAP (MRM-1488)
Posted by Deng Ching <oc...@apache.org>.
It was the former Brett. Right, I didn't think about closing the resources..
thanks for pointing that out :)
I'll do some testing on this. I'm actually thinking about reverting back the
authentication part to not use the cache and only have the ldap users
cached, I have to check though how big the difference would be in the builds
if that would be the case since I think it's the find/search for user that's
taking a while.
Thanks,
Deng
On Fri, Aug 26, 2011 at 11:47 AM, Brett Porter <br...@apache.org> wrote:
> Did you do this with ehcache or the technique Brent outlined? If it's the
> former, I'm worried about it not closing the resources - we should test it
> with a lot of concurrent different users.
>
> On 26/08/2011, at 12:57 PM, Deng Ching wrote:
>
> > I made some changes to the impl, btw. Instead of just caching the ldap
> > users, I've also cached the ldap connections. Not all ldap servers return
> a
> > hashed password (some return just a masked string, eg. ******) for the
> > userPassword attribute of an ldap user so we can't do a comparison on it.
> > You need to bind to the ldap server to authenticate, so I just cached the
> > ldap connection of a user. For the ldap connections, I've set the TTL to
> > 15secs., then 2 mins. TTL for the ldap users.
> >
> > I ran a 'clean install' on archiva-parent against an Archiva repo using
> JDO
> > and LDAP for authentication, and these are the results:
> > - JDO: 7:04.998s
> > - LDAP: 7:17.382s
> >
> > Thanks,
> > Deng
> >
> > On Thu, Aug 25, 2011 at 10:07 AM, Deng Ching <oc...@apache.org> wrote:
> >
> >> On Thu, Aug 25, 2011 at 1:44 AM, Brent Atkinson <
> brent.atkinson@gmail.com>wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> I actually ran into this when fixing the connection leaks. I realized
> it
> >>> was
> >>> probably building in too many assumptions, but I created and held onto
> the
> >>> LdapCtxFactory in redback's LdapConnection for a very specific reason:
> >>> connection pooling. The sun JNDI ldap implementation can pool
> connections
> >>> sharing the same credentials *and config options* as long as they are
> >>> created from the same LdapCtxFactory.
> >>>
> >>> http://download.oracle.com/javase/jndi/tutorial/ldap/connect/pool.html
> >>>
> >>>
> >> Thanks Brent! We'll look into that.
> >>
> >>
> >>> On Wed, Aug 24, 2011 at 8:57 AM, Wendy Smoak <ws...@gmail.com> wrote:
> >>>
> >>>> On Wed, Aug 24, 2011 at 2:45 AM, Deng Ching <oc...@apache.org>
> wrote:
> >>>>
> >>>>> We're planning to use EhCache for this so we can also set a TTL
> >>>>> (time-to-live) for the cached objects. A password change done from
> the
> >>>>> webapp would flush the user in the cache.
> >>>>
> >>>> If you're using LDAP, would users be doing password changes from the
> >>>> webapp?
> >>>>
> >>>> Making that TTL configurable by the admin would be good, then they can
> >>>> trade off between extra calls to LDAP and 'how come my new password
> >>>> doesn't work?'.
> >>>
> >>
> >> Agreed. We'll add this functionality as well :)
> >>
> >> Thanks,
> >> Deng
> >>
>
> --
> Brett Porter
> brett@apache.org
> http://brettporter.wordpress.com/
> http://au.linkedin.com/in/brettporter
>
>
>
>
>
Re: Archiva issue with LDAP (MRM-1488)
Posted by Brett Porter <br...@apache.org>.
Did you do this with ehcache or the technique Brent outlined? If it's the former, I'm worried about it not closing the resources - we should test it with a lot of concurrent different users.
On 26/08/2011, at 12:57 PM, Deng Ching wrote:
> I made some changes to the impl, btw. Instead of just caching the ldap
> users, I've also cached the ldap connections. Not all ldap servers return a
> hashed password (some return just a masked string, eg. ******) for the
> userPassword attribute of an ldap user so we can't do a comparison on it.
> You need to bind to the ldap server to authenticate, so I just cached the
> ldap connection of a user. For the ldap connections, I've set the TTL to
> 15secs., then 2 mins. TTL for the ldap users.
>
> I ran a 'clean install' on archiva-parent against an Archiva repo using JDO
> and LDAP for authentication, and these are the results:
> - JDO: 7:04.998s
> - LDAP: 7:17.382s
>
> Thanks,
> Deng
>
> On Thu, Aug 25, 2011 at 10:07 AM, Deng Ching <oc...@apache.org> wrote:
>
>> On Thu, Aug 25, 2011 at 1:44 AM, Brent Atkinson <br...@gmail.com>wrote:
>>
>>> Hi everyone,
>>>
>>> I actually ran into this when fixing the connection leaks. I realized it
>>> was
>>> probably building in too many assumptions, but I created and held onto the
>>> LdapCtxFactory in redback's LdapConnection for a very specific reason:
>>> connection pooling. The sun JNDI ldap implementation can pool connections
>>> sharing the same credentials *and config options* as long as they are
>>> created from the same LdapCtxFactory.
>>>
>>> http://download.oracle.com/javase/jndi/tutorial/ldap/connect/pool.html
>>>
>>>
>> Thanks Brent! We'll look into that.
>>
>>
>>> On Wed, Aug 24, 2011 at 8:57 AM, Wendy Smoak <ws...@gmail.com> wrote:
>>>
>>>> On Wed, Aug 24, 2011 at 2:45 AM, Deng Ching <oc...@apache.org> wrote:
>>>>
>>>>> We're planning to use EhCache for this so we can also set a TTL
>>>>> (time-to-live) for the cached objects. A password change done from the
>>>>> webapp would flush the user in the cache.
>>>>
>>>> If you're using LDAP, would users be doing password changes from the
>>>> webapp?
>>>>
>>>> Making that TTL configurable by the admin would be good, then they can
>>>> trade off between extra calls to LDAP and 'how come my new password
>>>> doesn't work?'.
>>>
>>
>> Agreed. We'll add this functionality as well :)
>>
>> Thanks,
>> Deng
>>
--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter
Re: Archiva issue with LDAP (MRM-1488)
Posted by Deng Ching <oc...@apache.org>.
I made some changes to the impl, btw. Instead of just caching the ldap
users, I've also cached the ldap connections. Not all ldap servers return a
hashed password (some return just a masked string, eg. ******) for the
userPassword attribute of an ldap user so we can't do a comparison on it.
You need to bind to the ldap server to authenticate, so I just cached the
ldap connection of a user. For the ldap connections, I've set the TTL to
15secs., then 2 mins. TTL for the ldap users.
I ran a 'clean install' on archiva-parent against an Archiva repo using JDO
and LDAP for authentication, and these are the results:
- JDO: 7:04.998s
- LDAP: 7:17.382s
Thanks,
Deng
On Thu, Aug 25, 2011 at 10:07 AM, Deng Ching <oc...@apache.org> wrote:
> On Thu, Aug 25, 2011 at 1:44 AM, Brent Atkinson <br...@gmail.com>wrote:
>
>> Hi everyone,
>>
>> I actually ran into this when fixing the connection leaks. I realized it
>> was
>> probably building in too many assumptions, but I created and held onto the
>> LdapCtxFactory in redback's LdapConnection for a very specific reason:
>> connection pooling. The sun JNDI ldap implementation can pool connections
>> sharing the same credentials *and config options* as long as they are
>> created from the same LdapCtxFactory.
>>
>> http://download.oracle.com/javase/jndi/tutorial/ldap/connect/pool.html
>>
>>
> Thanks Brent! We'll look into that.
>
>
>> On Wed, Aug 24, 2011 at 8:57 AM, Wendy Smoak <ws...@gmail.com> wrote:
>>
>> > On Wed, Aug 24, 2011 at 2:45 AM, Deng Ching <oc...@apache.org> wrote:
>> >
>> > > We're planning to use EhCache for this so we can also set a TTL
>> > > (time-to-live) for the cached objects. A password change done from the
>> > > webapp would flush the user in the cache.
>> >
>> > If you're using LDAP, would users be doing password changes from the
>> > webapp?
>> >
>> > Making that TTL configurable by the admin would be good, then they can
>> > trade off between extra calls to LDAP and 'how come my new password
>> > doesn't work?'.
>>
>
> Agreed. We'll add this functionality as well :)
>
> Thanks,
> Deng
>
Re: Archiva issue with LDAP (MRM-1488)
Posted by Deng Ching <oc...@apache.org>.
On Thu, Aug 25, 2011 at 1:44 AM, Brent Atkinson <br...@gmail.com>wrote:
> Hi everyone,
>
> I actually ran into this when fixing the connection leaks. I realized it
> was
> probably building in too many assumptions, but I created and held onto the
> LdapCtxFactory in redback's LdapConnection for a very specific reason:
> connection pooling. The sun JNDI ldap implementation can pool connections
> sharing the same credentials *and config options* as long as they are
> created from the same LdapCtxFactory.
>
> http://download.oracle.com/javase/jndi/tutorial/ldap/connect/pool.html
>
>
Thanks Brent! We'll look into that.
> On Wed, Aug 24, 2011 at 8:57 AM, Wendy Smoak <ws...@gmail.com> wrote:
>
> > On Wed, Aug 24, 2011 at 2:45 AM, Deng Ching <oc...@apache.org> wrote:
> >
> > > We're planning to use EhCache for this so we can also set a TTL
> > > (time-to-live) for the cached objects. A password change done from the
> > > webapp would flush the user in the cache.
> >
> > If you're using LDAP, would users be doing password changes from the
> > webapp?
> >
> > Making that TTL configurable by the admin would be good, then they can
> > trade off between extra calls to LDAP and 'how come my new password
> > doesn't work?'.
>
Agreed. We'll add this functionality as well :)
Thanks,
Deng
Re: Archiva issue with LDAP (MRM-1488)
Posted by Brent Atkinson <br...@gmail.com>.
Hi everyone,
I actually ran into this when fixing the connection leaks. I realized it was
probably building in too many assumptions, but I created and held onto the
LdapCtxFactory in redback's LdapConnection for a very specific reason:
connection pooling. The sun JNDI ldap implementation can pool connections
sharing the same credentials *and config options* as long as they are
created from the same LdapCtxFactory.
http://download.oracle.com/javase/jndi/tutorial/ldap/connect/pool.html
Brent
On Wed, Aug 24, 2011 at 8:57 AM, Wendy Smoak <ws...@gmail.com> wrote:
> On Wed, Aug 24, 2011 at 2:45 AM, Deng Ching <oc...@apache.org> wrote:
>
> > We're planning to use EhCache for this so we can also set a TTL
> > (time-to-live) for the cached objects. A password change done from the
> > webapp would flush the user in the cache.
>
> If you're using LDAP, would users be doing password changes from the
> webapp?
>
> Making that TTL configurable by the admin would be good, then they can
> trade off between extra calls to LDAP and 'how come my new password
> doesn't work?'.
>
> --
> Wendy
>
Re: Archiva issue with LDAP (MRM-1488)
Posted by Wendy Smoak <ws...@gmail.com>.
On Wed, Aug 24, 2011 at 2:45 AM, Deng Ching <oc...@apache.org> wrote:
> We're planning to use EhCache for this so we can also set a TTL
> (time-to-live) for the cached objects. A password change done from the
> webapp would flush the user in the cache.
If you're using LDAP, would users be doing password changes from the webapp?
Making that TTL configurable by the admin would be good, then they can
trade off between extra calls to LDAP and 'how come my new password
doesn't work?'.
--
Wendy
Re: Archiva issue with LDAP (MRM-1488)
Posted by Deng Ching <oc...@apache.org>.
On Wed, Aug 24, 2011 at 3:07 PM, Brett Porter <br...@apache.org> wrote:
>
> On 24/08/2011, at 4:45 PM, Deng Ching wrote:
>
> > Hi,
> >
> > Jev and I are currently working on MRM-1488 (Using Archiva as proxy for
> > downloading artifacts is very slow when used with LDAP authentication).
> The
> > problem we saw here was that for each artifact request, authentication
> > happens which in turn results to a call to the LDAP server.
> >
> > To fix this, we're planning to use an in-memory cache for the LDAP
> > credentials. For every authentication request made, Redback will:
> >
> > 1. look for the user in the cache
> > 2. if user is found in the cache, compare credentials
> > - if it matches, return successful
> > - if it doesn't match, reject it
>
> The credentials will only be there if you've stored them from a previous
> request as you can't get them back out of LDAP. You'll certainly want to
> hash this and compare the hash rather that storing the cleartext/base64
> password in memory.
>
Yep, we'll be storing and comparing the hash :)
>
> > 3. if user is not found in the cache
> > - retrieve user from LDAP server
> > - check provided credentials against the retrieved user
> > - if it matches, return successful and add retrieved user to the
> cache
> > - if it doesn't match, reject it
>
> The current code looks for the user, doesn't get any details, then tries
> authentication. This is presumably to check it matches the filter. It'd be
> good to check if we can do that ourselves, or as part of the authenticate
> call, rather than retrieving the user for the authn step.
>
> Are you doing this at the redback level, similar to the keys/rbac/users
> caching?
>
Yes
>
> >
> > We're planning to use EhCache for this so we can also set a TTL
> > (time-to-live) for the cached objects.
>
> I think Olivier recently changed the caching strategy in trunk, so we
> should stick with that.
>
Ok, will check that out.
Thanks!
-Deng
Re: Archiva issue with LDAP (MRM-1488)
Posted by Brett Porter <br...@apache.org>.
On 24/08/2011, at 4:45 PM, Deng Ching wrote:
> Hi,
>
> Jev and I are currently working on MRM-1488 (Using Archiva as proxy for
> downloading artifacts is very slow when used with LDAP authentication). The
> problem we saw here was that for each artifact request, authentication
> happens which in turn results to a call to the LDAP server.
>
> To fix this, we're planning to use an in-memory cache for the LDAP
> credentials. For every authentication request made, Redback will:
>
> 1. look for the user in the cache
> 2. if user is found in the cache, compare credentials
> - if it matches, return successful
> - if it doesn't match, reject it
The credentials will only be there if you've stored them from a previous request as you can't get them back out of LDAP. You'll certainly want to hash this and compare the hash rather that storing the cleartext/base64 password in memory.
> 3. if user is not found in the cache
> - retrieve user from LDAP server
> - check provided credentials against the retrieved user
> - if it matches, return successful and add retrieved user to the cache
> - if it doesn't match, reject it
The current code looks for the user, doesn't get any details, then tries authentication. This is presumably to check it matches the filter. It'd be good to check if we can do that ourselves, or as part of the authenticate call, rather than retrieving the user for the authn step.
Are you doing this at the redback level, similar to the keys/rbac/users caching?
>
> We're planning to use EhCache for this so we can also set a TTL
> (time-to-live) for the cached objects.
I think Olivier recently changed the caching strategy in trunk, so we should stick with that.
> A password change done from the
> webapp would flush the user in the cache.
> Also, in the logs, we've noticed a number of calls to the LDAP server
> (search for users then authenticate) which may not be necessary so we'll
> also check if we can improve those.
>
> Any thoughts or comments?
>
> Thanks,
> Deng
--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter
Re: Archiva issue with LDAP (MRM-1488)
Posted by Olivier Lamy <ol...@apache.org>.
+1.
Nice improvement !
2011/8/24 Deng Ching <oc...@apache.org>:
> Hi,
>
> Jev and I are currently working on MRM-1488 (Using Archiva as proxy for
> downloading artifacts is very slow when used with LDAP authentication). The
> problem we saw here was that for each artifact request, authentication
> happens which in turn results to a call to the LDAP server.
>
> To fix this, we're planning to use an in-memory cache for the LDAP
> credentials. For every authentication request made, Redback will:
>
> 1. look for the user in the cache
> 2. if user is found in the cache, compare credentials
> - if it matches, return successful
> - if it doesn't match, reject it
> 3. if user is not found in the cache
> - retrieve user from LDAP server
> - check provided credentials against the retrieved user
> - if it matches, return successful and add retrieved user to the cache
> - if it doesn't match, reject it
>
> We're planning to use EhCache for this so we can also set a TTL
> (time-to-live) for the cached objects. A password change done from the
> webapp would flush the user in the cache.
> Also, in the logs, we've noticed a number of calls to the LDAP server
> (search for users then authenticate) which may not be necessary so we'll
> also check if we can improve those.
>
> Any thoughts or comments?
>
> Thanks,
> Deng
>
--
Olivier Lamy
Talend : http://talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy