ContextCSN handling

[Emmanuel Lécharny <el...@gmail.com>]

Hi guys,

we have had a long chat with Kiran about the best way to handle the
contextCSN attribute.

This attribute is used for replication, as we need to know if one server
is up to date. The way to do it is to compare the local contextCSN with
the remote one. The pb is where to store this contextCSN, and how to get
it when doing a search.

o the contextCSN is associated with the root of a partition, it's
associated with teh contextEntry (see further for some new ideas about
where to store teh contextCSN).
o it contains the newest entryCSN in the partition.
o it's updated every time an update is done on the partition.
o it's persisted (ie, when the server is restarted, it's present)

Now, when one does a search, we have to have this contextCSN attrbute
present in the contextEntry (assumng the user has requested the
attribute, of course, as it's operationnal).

If we update the contextEntry after *each* update, we wil double to time
for updating an entry. No way...

The solution is to store this value in memory, and to recompute it when
the server is started (finding the newest entryCSN of each partition).

That being said, we still have to inject this attribute in the
contextEntry when a search is done, which means we have to detect that
an entry is a context entry. Not easy, and costly.

There are two ways to do that :
o assume that the entry's cache always contain the contextEntry, which
always contains the contextCsn. In this case, we just have to update the
contextEntry after any modification (all in all, we just update the cache).
o modify the OperationalAttributeInterceptor to injetc the contextCSN
when we detect that a fetched entry is the contextEntry. In order to do
that, we need a quick way to detect the contextEntry. We need to
propagate the partition's DN into the SearchOperationContext, so that we
can compare the length of this DN with the length of the fetched entry's
DN : if they are equal, it's the contextEntry.

IMO the second approach is a bit more complicated (but I don't know why,
I thik it's the best approach...)

thoughts ?

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

Re: ContextCSN handling

[Kiran Ayyagari <ka...@apache.org>]

On Thu, Feb 13, 2014 at 5:20 PM, Emmanuel Lécharny <el...@gmail.com>wrote:

> Hi guys,
>
> we have had a long chat with Kiran about the best way to handle the
> contextCSN attribute.
>
> This attribute is used for replication, as we need to know if one server
> is up to date. The way to do it is to compare the local contextCSN with
> the remote one. The pb is where to store this contextCSN, and how to get
>
this is also needed to determine what content needs to be sent to a
syncrepl client

> it when doing a search.
>
> o the contextCSN is associated with the root of a partition, it's
> associated with teh contextEntry (see further for some new ideas about
> where to store teh contextCSN).
> o it contains the newest entryCSN in the partition.
> o it's updated every time an update is done on the partition.
> o it's persisted (ie, when the server is restarted, it's present)
>
> Now, when one does a search, we have to have this contextCSN attrbute
> present in the contextEntry (assumng the user has requested the
> attribute, of course, as it's operationnal).
>
> If we update the contextEntry after *each* update, we wil double to time
> for updating an entry. No way...
>
> The solution is to store this value in memory, and to recompute it when
> the server is started (finding the newest entryCSN of each partition).
>
> That being said, we still have to inject this attribute in the
> contextEntry when a search is done, which means we have to detect that
> an entry is a context entry. Not easy, and costly.
>
> There are two ways to do that :
> o assume that the entry's cache always contain the contextEntry, which
> always contains the contextCsn. In this case, we just have to update the
> contextEntry after any modification (all in all, we just update the cache).
> o modify the OperationalAttributeInterceptor to injetc the contextCSN
> when we detect that a fetched entry is the contextEntry. In order to do
> that, we need a quick way to detect the contextEntry. We need to
> propagate the partition's DN into the SearchOperationContext, so that we
> can compare the length of this DN with the length of the fetched entry's
> DN : if they are equal, it's the contextEntry.
>
> IMO the second approach is a bit more complicated (but I don't know why,
> I thik it's the best approach...)
>
> yes, but a slight change, instead of searching for the context entry in
the interceptor, the context entry gets updated in the backed if it the
contextCsn is dirty and the search base is the partition's root and then the
search continues.

besides this, the contextCsn will always be maintained in memory and will
be written(if it is changed) during
shutdown


> thoughts ?
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Kiran Ayyagari
http://keydap.com

Re: ContextCSN handling

[Emmanuel Lécharny <el...@gmail.com>]

Le 2/13/14 4:13 PM, Howard Chu a écrit :
> Emmanuel Lécharny wrote:
>> Hi guys,
>>
>> we have had a long chat with Kiran about the best way to handle the
>> contextCSN attribute.
>>
>> This attribute is used for replication, as we need to know if one server
>> is up to date. The way to do it is to compare the local contextCSN with
>> the remote one. The pb is where to store this contextCSN, and how to get
>> it when doing a search.
>>
>> o the contextCSN is associated with the root of a partition, it's
>> associated with teh contextEntry (see further for some new ideas about
>> where to store teh contextCSN).
>> o it contains the newest entryCSN in the partition.
>> o it's updated every time an update is done on the partition.
>> o it's persisted (ie, when the server is restarted, it's present)
>>
>> Now, when one does a search, we have to have this contextCSN attrbute
>> present in the contextEntry (assumng the user has requested the
>> attribute, of course, as it's operationnal).
>>
>> If we update the contextEntry after *each* update, we wil double to time
>> for updating an entry. No way...
>>
>> The solution is to store this value in memory, and to recompute it when
>> the server is started (finding the newest entryCSN of each partition).
>>
>> That being said, we still have to inject this attribute in the
>> contextEntry when a search is done, which means we have to detect that
>> an entry is a context entry. Not easy, and costly.
>>
>> There are two ways to do that :
>> o assume that the entry's cache always contain the contextEntry, which
>> always contains the contextCsn. In this case, we just have to update the
>> contextEntry after any modification (all in all, we just update the
>> cache).
>> o modify the OperationalAttributeInterceptor to injetc the contextCSN
>> when we detect that a fetched entry is the contextEntry. In order to do
>> that, we need a quick way to detect the contextEntry. We need to
>> propagate the partition's DN into the SearchOperationContext, so that we
>> can compare the length of this DN with the length of the fetched entry's
>> DN : if they are equal, it's the contextEntry.
>>
>> IMO the second approach is a bit more complicated (but I don't know why,
>> I thik it's the best approach...)
>
> In OpenLDAP we cache the value in memory by default. 

Yes, this is what we are going to do too : using a VolatileRef to store
the reference to this contextCSN, and update it on the fly.
> There is a  configurable syncprov-checkpoint so after T time or N
> updates we do an actual write of the attribute into the underlying entry.
I'm not sure we need to do taht at all : when the server starts, it's
good enough to look for the newest entryCSN anyway, and it as to be done.


> We also have a flag on the backends indicating whether or not it
> previously had a clean shutdown; if this flag indicates an unclean
> shutdown then we can do a search for the highest entryCSN in the DB.
> (This flag is somewhat obsolete now with back-mdb.)
>
> Aside from that, we of course write the attribute on clean shutdowns.
I wonder if it's necessary. The gain is minimal, you pay the price of
writing the element every T times or N updates, for a server whch is not
supposed to stop, compared to a simpe search on an indexed EntryCSN
attribute at startup...


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

Re: ContextCSN handling

[Howard Chu <hy...@symas.com>]

Emmanuel Lécharny wrote:
> Hi guys,
>
> we have had a long chat with Kiran about the best way to handle the
> contextCSN attribute.
>
> This attribute is used for replication, as we need to know if one server
> is up to date. The way to do it is to compare the local contextCSN with
> the remote one. The pb is where to store this contextCSN, and how to get
> it when doing a search.
>
> o the contextCSN is associated with the root of a partition, it's
> associated with teh contextEntry (see further for some new ideas about
> where to store teh contextCSN).
> o it contains the newest entryCSN in the partition.
> o it's updated every time an update is done on the partition.
> o it's persisted (ie, when the server is restarted, it's present)
>
> Now, when one does a search, we have to have this contextCSN attrbute
> present in the contextEntry (assumng the user has requested the
> attribute, of course, as it's operationnal).
>
> If we update the contextEntry after *each* update, we wil double to time
> for updating an entry. No way...
>
> The solution is to store this value in memory, and to recompute it when
> the server is started (finding the newest entryCSN of each partition).
>
> That being said, we still have to inject this attribute in the
> contextEntry when a search is done, which means we have to detect that
> an entry is a context entry. Not easy, and costly.
>
> There are two ways to do that :
> o assume that the entry's cache always contain the contextEntry, which
> always contains the contextCsn. In this case, we just have to update the
> contextEntry after any modification (all in all, we just update the cache).
> o modify the OperationalAttributeInterceptor to injetc the contextCSN
> when we detect that a fetched entry is the contextEntry. In order to do
> that, we need a quick way to detect the contextEntry. We need to
> propagate the partition's DN into the SearchOperationContext, so that we
> can compare the length of this DN with the length of the fetched entry's
> DN : if they are equal, it's the contextEntry.
>
> IMO the second approach is a bit more complicated (but I don't know why,
> I thik it's the best approach...)

In OpenLDAP we cache the value in memory by default. There is a  configurable 
syncprov-checkpoint so after T time or N updates we do an actual write of the 
attribute into the underlying entry. We also have a flag on the backends 
indicating whether or not it previously had a clean shutdown; if this flag 
indicates an unclean shutdown then we can do a search for the highest entryCSN 
in the DB. (This flag is somewhat obsolete now with back-mdb.)

Aside from that, we of course write the attribute on clean shutdowns.
>
> thoughts ?
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/