about lock mechanism in apacheDS

["Wu, James C." <Ja...@disney.com>]

Hi Guys,

I am wondering if we can make the lock mechanism in ApacheDS more flexible. Right now, both the read lock and the write lock seem to be global, which does not allow concurrent write to different entities in the LDAP store. If we can make the lock more granular, it would improve the throughput of the ApacheDS service. For example, we can have one thread serving one ldap session trying to read, and another thread serving another ldap session applying some modifications.

Another issue with the current lock mechanism is that, a lock is not associate with the owner, leading to a strange behavior.

Here is an example. I would like to setup ApacheDS in this way by overriding the simple authentication class so that if a user fails to authenticate himself, an external authentication service is used. When external authentication succeeds, I would like to insert this user as an entry in the ldap store or update his password. Users usually do a ldap binding with password to authenticate to ApacheDS. During the handling of the bind request, ApacheDS will first request a readlock through the DefaultOperationManager, then call the simple authenticator to authenticate the user. In my case, if the authentication fails, then external authentication service is used instead. The problem comes at when trying to insert the user as a new entity or update its password, the write lock can't be obtained because it is still in the bind process and the read lock is only released after the bind finishes.  If we keep track of the owner of the lock, we would be able to allow the promotion of read-lock to write-lock as how lock is done in many database systems.

Regards,

James

Re: about lock mechanism in apacheDS

[Emmanuel Lécharny <el...@gmail.com>]

Le 4/19/13 2:13 AM, Wu, James C. a écrit :
> Hi,
>
> I changed the lockRead() and the unlockRead() methods in the DefaultOperationManager from private to public so that I can change the lock settings in my customized authenticator.
>
> May I suggest changing them to public methods in trunk? It will give customer more flexibility when they provide their customized implementation for interceptors, authenticators etc.

I have created a specific issue for this :

https://issues.apache.org/jira/browse/DIRSERVER-1832

This is not a simple problem, and we have to be extremely careful in
this area, as it has a lot of potential impact all over the server.

But defintively something we have to work out..

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

Re: about lock mechanism in apacheDS

[Emmanuel Lécharny <el...@gmail.com>]

Le 4/19/13 2:13 AM, Wu, James C. a écrit :
> Hi,
>
> I changed the lockRead() and the unlockRead() methods in the DefaultOperationManager from private to public so that I can change the lock settings in my customized authenticator.
>
> May I suggest changing them to public methods in trunk? It will give customer more flexibility when they provide their customized implementation for interceptors, authenticators etc.

We can, but this is an extremly dangerous change. Not that it will break
anything to make those methods to public, but you can't simply manage
the locks from your cod without potentially breakig the underlying database.

I have to check if there is not another solution...


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

RE: about lock mechanism in apacheDS

["Wu, James C." <Ja...@disney.com>]

Hi,

I changed the lockRead() and the unlockRead() methods in the DefaultOperationManager from private to public so that I can change the lock settings in my customized authenticator.

May I suggest changing them to public methods in trunk? It will give customer more flexibility when they provide their customized implementation for interceptors, authenticators etc.

Regards,

James


-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com] 
Sent: Thursday, April 18, 2013 5:03 PM
To: Apache Directory Developers List
Subject: Re: about lock mechanism in apacheDS

Le 4/19/13 1:38 AM, Wu, James C. a écrit :
> Hi Guys,
>
> I am wondering if we can make the lock mechanism in ApacheDS more flexible. Right now, both the read lock and the write lock seem to be global, which does not allow concurrent write to different entities in the LDAP store. If we can make the lock more granular, it would improve the throughput of the ApacheDS service. For example, we can have one thread serving one ldap session trying to read, and another thread serving another ldap session applying some modifications.

The real problem is the backend. JDBM does not allow us to work nicely when we have concurrent reads and writes. We have written a new backend (http://svn.apache.org/repos/asf/labs/mavibot/ and
https://cwiki.apache.org/confluence/display/labs/Mavibot) which will be way more efficient, as it will allow concurrent reads and more important, concurrent reads and write (although the write ar still
serialized)
>
> Another issue with the current lock mechanism is that, a lock is not associate with the owner, leading to a strange behavior.
>
> Here is an example. I would like to setup ApacheDS in this way by overriding the simple authentication class so that if a user fails to authenticate himself, an external authentication service is used. When external authentication succeeds, I would like to insert this user as an entry in the ldap store or update his password. Users usually do a ldap binding with password to authenticate to ApacheDS. During the handling of the bind request, ApacheDS will first request a readlock through the DefaultOperationManager, then call the simple authenticator to authenticate the user. In my case, if the authentication fails, then external authentication service is used instead. The problem comes at when trying to insert the user as a new entity or update its password, the write lock can't be obtained because it is still in the bind process and the read lock is only released after the bind finishes.  If we keep track of the owner of the lock, we would be able to allow the promotion of read-lock to write-lock as how lock is done in many database systems.

Currently, this is not possible. The only possible option would be to modify the BindHandler so that if the bind fails, you trp it and add the user (as the BindHnalder is calling the DirectoryService, it's not affected by the lock). Sdly, this part has no extension point.

As soon as we drop JDBM, we will not have the pb you are exposing.

The ongoig work sounds promizing, as Kiran was able to get the core-integ tests passing with the Mavibot partition, but there is a lot to do still (I would evaluate the remainingw work to a couple of months).

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

Re: about lock mechanism in apacheDS

[Emmanuel Lécharny <el...@gmail.com>]

Le 4/19/13 1:38 AM, Wu, James C. a écrit :
> Hi Guys,
>
> I am wondering if we can make the lock mechanism in ApacheDS more flexible. Right now, both the read lock and the write lock seem to be global, which does not allow concurrent write to different entities in the LDAP store. If we can make the lock more granular, it would improve the throughput of the ApacheDS service. For example, we can have one thread serving one ldap session trying to read, and another thread serving another ldap session applying some modifications.

The real problem is the backend. JDBM does not allow us to work nicely
when we have concurrent reads and writes. We have written a new backend
(http://svn.apache.org/repos/asf/labs/mavibot/ and
https://cwiki.apache.org/confluence/display/labs/Mavibot) which will be
way more efficient, as it will allow concurrent reads and more
important, concurrent reads and write (although the write ar still
serialized)
>
> Another issue with the current lock mechanism is that, a lock is not associate with the owner, leading to a strange behavior.
>
> Here is an example. I would like to setup ApacheDS in this way by overriding the simple authentication class so that if a user fails to authenticate himself, an external authentication service is used. When external authentication succeeds, I would like to insert this user as an entry in the ldap store or update his password. Users usually do a ldap binding with password to authenticate to ApacheDS. During the handling of the bind request, ApacheDS will first request a readlock through the DefaultOperationManager, then call the simple authenticator to authenticate the user. In my case, if the authentication fails, then external authentication service is used instead. The problem comes at when trying to insert the user as a new entity or update its password, the write lock can't be obtained because it is still in the bind process and the read lock is only released after the bind finishes.  If we keep track of the owner of the lock, we would be able to allow the promotion of read-lock to write-lock as how lock is done in many database systems.

Currently, this is not possible. The only possible option would be to
modify the BindHandler so that if the bind fails, you trp it and add the
user (as the BindHnalder is calling the DirectoryService, it's not
affected by the lock). Sdly, this part has no extension point.

As soon as we drop JDBM, we will not have the pb you are exposing.

The ongoig work sounds promizing, as Kiran was able to get the
core-integ tests passing with the Mavibot partition, but there is a lot
to do still (I would evaluate the remainingw work to a couple of months).

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com