You are viewing a plain text version of this content. The canonical link for it is here.
Posted to fortress@directory.apache.org by Shawn McKinney <sm...@apache.org> on 2019/03/19 12:42:55 UTC
Replacing Caching with LDAP Persistent Searches
This idea has been kicked around before, we discussed on the dev list a several months ago:
http://mail-archives.apache.org/mod_mbox/directory-dev/201811.mbox/%3cB5AFAF88-F17C-4D41-9992-E3C53C9FD008@apache.org%3e
The biggest problem with caching is creates consistency problems between highly-available and/or load-balanced nodes. In today’s computing environment (everything’s running in a container/cluster) it’s an untenable situation.
This is a proposal to replace fortress usage of ehchache with the LDAP persistent search control.
Specifically these cached datasets would be targeted:
a. cache name="fortress.policies”
b. cache name="fortress.ous”
c. cache name="fortress.roles”
d. cache name="fortress.admin.roles”
e. cache name="fortress.pso”
f. cache name="fortress.uso”
g. cache name="fortress.dsd”
h. cache name=“fortress.ssd”
My plan, start playing in a sandbox, get an estimate of amount of work / complexity of the change. It may require changing how Fortress handles state, to be more inline with what can be done using persistent search. Of course the public APIs should not have to change nor should the behavior with the client (hint requirement). Let me know if you have any interest in participation (providing requirements, design, test) in this effort.
Thanks,
—Shawn
Re: Replacing Caching with LDAP Persistent Searches
Posted by Shawn McKinney <sm...@apache.org>.
> On Mar 19, 2019, at 11:46 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
>
> On 20/03/2019 02:57, Yudhi Karunia Surtan wrote:
>> Hi Emanuel,
>>
>> Sorry i don't get it. Do you mean LDAP have their own internal cache?
>
> No. What I mean is that the persistent search operation will send to the client (ie Fortress) all the updates on the fly, so Fortress can maintain a local cache without having to deal with replicating it with other instances. This cache will always be up to date, assuming the LDAP server is up to date.
Yudhi, I tend to agree with Emmanuel, that if we do this right, you won’t see a degradation in performance on the client, regardless of load.
In fact, the performance might become better, due to no longer having to flush the cache and reload (after TTL expires).
That being said, I’m not opposed to making this behavior configurable, assuming we can do it cleanly.
Certainly we’ll make benchmarks before/after to determine performance characteristics of both approaches.
—Shawn
Re: Replacing Caching with LDAP Persistent Searches
Posted by Emmanuel Lécharny <el...@gmail.com>.
On 20/03/2019 02:57, Yudhi Karunia Surtan wrote:
> Hi Emanuel,
>
> Sorry i don't get it. Do you mean LDAP have their own internal cache?
No. What I mean is that the persistent search operation will send to the
client (ie Fortress) all the updates on the fly, so Fortress can
maintain a local cache without having to deal with replicating it with
other instances. This cache will always be up to date, assuming the LDAP
server is up to date.
Re: Replacing Caching with LDAP Persistent Searches
Posted by Yudhi Karunia Surtan <br...@gmail.com>.
Hi Emanuel,
Sorry i don't get it. Do you mean LDAP have their own internal cache?
Yes, I agree. In most of use case once the permissions are set, it is very
rare to change.
On Tue, Mar 19, 2019, 23:35 Emmanuel Lécharny <el...@gmail.com> wrote:
>
> On 19/03/2019 14:45, Yudhi Karunia Surtan wrote:
> > Hi Shawn,
> >
> > Is that possible that by design we put the optional cache interface at
> > fortress so later people can choose their own cache implementations? Well
> > ya, by default it will use LDAP if they not implement their own class
> > implementation.
> >
> > I think caching is very important when you have a lot user using it.
>
>
> Using persistent search is a better solution : you keep a local cache
> which get informed live when some changes are made on the LDAP server.
>
> Most of the time, you just have to read the cache.
>
Re: Replacing Caching with LDAP Persistent Searches
Posted by Emmanuel Lécharny <el...@gmail.com>.
On 19/03/2019 14:45, Yudhi Karunia Surtan wrote:
> Hi Shawn,
>
> Is that possible that by design we put the optional cache interface at
> fortress so later people can choose their own cache implementations? Well
> ya, by default it will use LDAP if they not implement their own class
> implementation.
>
> I think caching is very important when you have a lot user using it.
Using persistent search is a better solution : you keep a local cache
which get informed live when some changes are made on the LDAP server.
Most of the time, you just have to read the cache.
Re: Replacing Caching with LDAP Persistent Searches
Posted by Yudhi Karunia Surtan <br...@gmail.com>.
Hi Shawn,
Is that possible that by design we put the optional cache interface at
fortress so later people can choose their own cache implementations? Well
ya, by default it will use LDAP if they not implement their own class
implementation.
I think caching is very important when you have a lot user using it.
On Tue, Mar 19, 2019, 20:26 Emmanuel Lécharny <el...@gmail.com> wrote:
>
> On 19/03/2019 13:42, Shawn McKinney wrote:
> > This idea has been kicked around before, we discussed on the dev list a
> several months ago:
> >
> http://mail-archives.apache.org/mod_mbox/directory-dev/201811.mbox/%3cB5AFAF88-F17C-4D41-9992-E3C53C9FD008@apache.org%3e
> >
> > The biggest problem with caching is creates consistency problems between
> highly-available and/or load-balanced nodes. In today’s computing
> environment (everything’s running in a container/cluster) it’s an untenable
> situation.
> >
> > This is a proposal to replace fortress usage of ehchache with the LDAP
> persistent search control.
> >
> > Specifically these cached datasets would be targeted:
> >
> > a. cache name="fortress.policies”
> > b. cache name="fortress.ous”
> > c. cache name="fortress.roles”
> > d. cache name="fortress.admin.roles”
> > e. cache name="fortress.pso”
> > f. cache name="fortress.uso”
> > g. cache name="fortress.dsd”
> > h. cache name=“fortress.ssd”
> >
> > My plan, start playing in a sandbox, get an estimate of amount of work /
> complexity of the change. It may require changing how Fortress handles
> state, to be more inline with what can be done using persistent search. Of
> course the public APIs should not have to change nor should the behavior
> with the client (hint requirement). Let me know if you have any interest
> in participation (providing requirements, design, test) in this effort.
>
>
> I can give you and hand with that. The only aspect that needs to be
> checked is the fact that persistent search is not necessarily
> implemented the same way on all the LDAP servers, but AFAICT, for
> OpenLDAP and ApacheDS, it should be just fine.
>
>
> And, yes, that is definitively a better solution than managing a local
> cache with all the complexity of having it consistent across various
> machines.
>
>
> It should also be simple to implement, and fast enough for your needs.
>
>
>
Re: Replacing Caching with LDAP Persistent Searches
Posted by Emmanuel Lécharny <el...@gmail.com>.
On 19/03/2019 13:42, Shawn McKinney wrote:
> This idea has been kicked around before, we discussed on the dev list a several months ago:
> http://mail-archives.apache.org/mod_mbox/directory-dev/201811.mbox/%3cB5AFAF88-F17C-4D41-9992-E3C53C9FD008@apache.org%3e
>
> The biggest problem with caching is creates consistency problems between highly-available and/or load-balanced nodes. In today’s computing environment (everything’s running in a container/cluster) it’s an untenable situation.
>
> This is a proposal to replace fortress usage of ehchache with the LDAP persistent search control.
>
> Specifically these cached datasets would be targeted:
>
> a. cache name="fortress.policies”
> b. cache name="fortress.ous”
> c. cache name="fortress.roles”
> d. cache name="fortress.admin.roles”
> e. cache name="fortress.pso”
> f. cache name="fortress.uso”
> g. cache name="fortress.dsd”
> h. cache name=“fortress.ssd”
>
> My plan, start playing in a sandbox, get an estimate of amount of work / complexity of the change. It may require changing how Fortress handles state, to be more inline with what can be done using persistent search. Of course the public APIs should not have to change nor should the behavior with the client (hint requirement). Let me know if you have any interest in participation (providing requirements, design, test) in this effort.
I can give you and hand with that. The only aspect that needs to be
checked is the fact that persistent search is not necessarily
implemented the same way on all the LDAP servers, but AFAICT, for
OpenLDAP and ApacheDS, it should be just fine.
And, yes, that is definitively a better solution than managing a local
cache with all the complexity of having it consistent across various
machines.
It should also be simple to implement, and fast enough for your needs.