You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Pierre-Arnaud Marcelot <pa...@marcelot.net> on 2013/01/08 15:59:38 UTC

Configuration of replication consumers

Hi guys,

While I was writing the UI for the replication consumers configuration, I came into an interesting issue.

I opened a JIRA for it : DIRSERVER-1789 - Changes to an existing replication consumer may not be taken into account
https://issues.apache.org/jira/browse/DIRSERVER-1789

Here's what's happening:

1/ We configure on Server A a replication consumer over Server B, and we restart Server A for the modifications to take effect.

2/ Replication takes place and an entry is created on Server B under the branch "ou=consumers, ou=system". This entry corresponds to the replication consumer of Server A and stores some information about it (like the search base, referral information, etc.).
This information is required because Server B needs to maintain a queue of modifications to send to the consumer of Server A (especially in the Refresh Only mode or when a disconnection occurs).

3/ We reconfigure on Server A the replication consumer and change one of its values (search base, attributes, etc.) and we restart Server A for the modifications to take effect.

4/ The replication is resumed, but, here's the problem... The configuration items are not modified on the corresponding entry on Server A under "ou=consumers, ou=system"...


We thought about a solution to this problem with Kiran yesterday and we came up with two solutions.

Solution 1:
We need a mechanism to detect any change of replication consumer configuration at the boot of ApacheDS (probably by holding a copy of each replication consumer configuration entry somewhere and comparing it to the current configuration).
If a difference is detected, then the server has to reset the replication consumer and start the replication from scratch.

Solution 2:
We don't implement a change detection mechanism, but we include a specific attribute in the entry of the replication consumer to indicate to the server that it has been updated and should be reset (with a restart of the replication from scratch).


My personal preference is Solution 1, which is a little bit harder to implement on the server side, but easier to use on the client side.


The impact on Server B by these two solutions is exactly the same. A new entry will be created under "ou=consumers, ou=system" and the previous entry for the replication consumer will be deleted once we reach the usual timeout.

What do you think of these solutions?
Which one would you pick?

Thanks!
Pierre-Arnaud

Re: Configuration of replication consumers

Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
On 9 janv. 2013, at 12:15, Emmanuel Lécharny <el...@gmail.com> wrote:

> Le 1/8/13 3:59 PM, Pierre-Arnaud Marcelot a écrit :
>> Hi guys,
>> 
>> While I was writing the UI for the replication consumers configuration, I came into an interesting issue.
>> 
>> I opened a JIRA for it : DIRSERVER-1789 - Changes to an existing replication consumer may not be taken into account
>> https://issues.apache.org/jira/browse/DIRSERVER-1789
>> 
>> Here's what's happening:
>> 
>> 1/ We configure on Server A a replication consumer over Server B, and we restart Server A for the modifications to take effect.
>> 
>> 2/ Replication takes place and an entry is created on Server B under the branch "ou=consumers, ou=system". This entry corresponds to the replication consumer of Server A and stores some information about it (like the search base, referral information, etc.).
>> This information is required because Server B needs to maintain a queue of modifications to send to the consumer of Server A (especially in the Refresh Only mode or when a disconnection occurs).
>> 
>> 3/ We reconfigure on Server A the replication consumer and change one of its values (search base, attributes, etc.) and we restart Server A for the modifications to take effect.
>> 
>> 4/ The replication is resumed, but, here's the problem... The configuration items are not modified on the corresponding entry on Server A under "ou=consumers, ou=system"...
>> 
>> 
>> We thought about a solution to this problem with Kiran yesterday and we came up with two solutions.
>> 
>> Solution 1:
>> We need a mechanism to detect any change of replication consumer configuration at the boot of ApacheDS (probably by holding a copy of each replication consumer configuration entry somewhere and comparing it to the current configuration).
>> If a difference is detected, then the server has to reset the replication consumer and start the replication from scratch.
>> 
>> Solution 2:
>> We don't implement a change detection mechanism, but we include a specific attribute in the entry of the replication consumer to indicate to the server that it has been updated and should be reset (with a restart of the replication from scratch).
>> 
>> 
>> My personal preference is Solution 1, which is a little bit harder to implement on the server side, but easier to use on the client side.
>> 
>> 
>> The impact on Server B by these two solutions is exactly the same. A new entry will be created under "ou=consumers, ou=system" and the previous entry for the replication consumer will be deleted once we reach the usual timeout.
>> 
>> What do you think of these solutions?
>> Which one would you pick?
> 
> This is not simple... At first, I thought that solution 1 was better :
> no need to transmit anything more than what the RFC says, we just detect
> the change when the consumer reconnects. the problem is that we have no
> idea if the consumer is the one which was registred before, unless it
> says so. The transmitted data are the connection itself (IP, port), plus
> the search request. If the config has changed, this is a new replication
> request, and we must delet the old one. Now, how do we determinate if
> the consumer was previously connected and should be deleted ? I don't
> see any way to do that efficiently.
> 
> I'm thinking on a third solution there : we regularly check the
> 'sleeping' registred consumers on the producer, and if they haven't be
> active for more than a determinated period of time, we simply delete
> them (say, after 7 days).
> 
> This way, we don't store many entries for nothing, and we don't need to
> send extra informations.
> 
> Does it sounds reasonnable ?

Actually, that's not really a third solution but a required path to make solution 1 or 2 work.

That's exactly what I meant by "and the previous entry for the replication consumer will be deleted once we reach the usual timeout."

Kiran and I thought that process was already in place in the server, but it turns out it isn't.

Regards,
Pierre-Arnaud


> -- 
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com 
>

Re: Configuration of replication consumers

Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, Jan 9, 2013 at 4:45 PM, Emmanuel Lécharny <el...@gmail.com>wrote:

> Le 1/8/13 3:59 PM, Pierre-Arnaud Marcelot a écrit :
> > Hi guys,
> >
> > While I was writing the UI for the replication consumers configuration,
> I came into an interesting issue.
> >
> > I opened a JIRA for it : DIRSERVER-1789 - Changes to an existing
> replication consumer may not be taken into account
> > https://issues.apache.org/jira/browse/DIRSERVER-1789
> >
> > Here's what's happening:
> >
> > 1/ We configure on Server A a replication consumer over Server B, and we
> restart Server A for the modifications to take effect.
> >
> > 2/ Replication takes place and an entry is created on Server B under the
> branch "ou=consumers, ou=system". This entry corresponds to the replication
> consumer of Server A and stores some information about it (like the search
> base, referral information, etc.).
> > This information is required because Server B needs to maintain a queue
> of modifications to send to the consumer of Server A (especially in the
> Refresh Only mode or when a disconnection occurs).
> >
> > 3/ We reconfigure on Server A the replication consumer and change one of
> its values (search base, attributes, etc.) and we restart Server A for the
> modifications to take effect.
> >
> > 4/ The replication is resumed, but, here's the problem... The
> configuration items are not modified on the corresponding entry on Server A
> under "ou=consumers, ou=system"...
> >
> >
> > We thought about a solution to this problem with Kiran yesterday and we
> came up with two solutions.
> >
> > Solution 1:
> > We need a mechanism to detect any change of replication consumer
> configuration at the boot of ApacheDS (probably by holding a copy of each
> replication consumer configuration entry somewhere and comparing it to the
> current configuration).
> > If a difference is detected, then the server has to reset the
> replication consumer and start the replication from scratch.
> >
> > Solution 2:
> > We don't implement a change detection mechanism, but we include a
> specific attribute in the entry of the replication consumer to indicate to
> the server that it has been updated and should be reset (with a restart of
> the replication from scratch).
> >
> >
> > My personal preference is Solution 1, which is a little bit harder to
> implement on the server side, but easier to use on the client side.
> >
> >
> > The impact on Server B by these two solutions is exactly the same. A new
> entry will be created under "ou=consumers, ou=system" and the previous
> entry for the replication consumer will be deleted once we reach the usual
> timeout.
> >
> > What do you think of these solutions?
> > Which one would you pick?
>
> This is not simple... At first, I thought that solution 1 was better :
> no need to transmit anything more than what the RFC says, we just detect
> the change when the consumer reconnects. the problem is that we have no
> idea if the consumer is the one which was registred before, unless it
> says so. The transmitted data are the connection itself (IP, port), plus
> the search request. If the config has changed, this is a new replication
> request, and we must delet the old one. Now, how do we determinate if
> the consumer was previously connected and should be deleted ? I don't
> see any way to do that efficiently.
>
> I'm thinking on a third solution there : we regularly check the
> 'sleeping' registred consumers on the producer, and if they haven't be
> active for more than a determinated period of time, we simply delete
> them (say, after 7 days).
>
> we have all the configuration options needed to support this, all we need
now is some
code to hook them together

> This way, we don't store many entries for nothing, and we don't need to
> send extra informations.
>
> Does it sounds reasonnable ?
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Kiran Ayyagari
http://keydap.com

Re: Configuration of replication consumers

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 1/8/13 3:59 PM, Pierre-Arnaud Marcelot a écrit :
> Hi guys,
>
> While I was writing the UI for the replication consumers configuration, I came into an interesting issue.
>
> I opened a JIRA for it : DIRSERVER-1789 - Changes to an existing replication consumer may not be taken into account
> https://issues.apache.org/jira/browse/DIRSERVER-1789
>
> Here's what's happening:
>
> 1/ We configure on Server A a replication consumer over Server B, and we restart Server A for the modifications to take effect.
>
> 2/ Replication takes place and an entry is created on Server B under the branch "ou=consumers, ou=system". This entry corresponds to the replication consumer of Server A and stores some information about it (like the search base, referral information, etc.).
> This information is required because Server B needs to maintain a queue of modifications to send to the consumer of Server A (especially in the Refresh Only mode or when a disconnection occurs).
>
> 3/ We reconfigure on Server A the replication consumer and change one of its values (search base, attributes, etc.) and we restart Server A for the modifications to take effect.
>
> 4/ The replication is resumed, but, here's the problem... The configuration items are not modified on the corresponding entry on Server A under "ou=consumers, ou=system"...
>
>
> We thought about a solution to this problem with Kiran yesterday and we came up with two solutions.
>
> Solution 1:
> We need a mechanism to detect any change of replication consumer configuration at the boot of ApacheDS (probably by holding a copy of each replication consumer configuration entry somewhere and comparing it to the current configuration).
> If a difference is detected, then the server has to reset the replication consumer and start the replication from scratch.
>
> Solution 2:
> We don't implement a change detection mechanism, but we include a specific attribute in the entry of the replication consumer to indicate to the server that it has been updated and should be reset (with a restart of the replication from scratch).
>
>
> My personal preference is Solution 1, which is a little bit harder to implement on the server side, but easier to use on the client side.
>
>
> The impact on Server B by these two solutions is exactly the same. A new entry will be created under "ou=consumers, ou=system" and the previous entry for the replication consumer will be deleted once we reach the usual timeout.
>
> What do you think of these solutions?
> Which one would you pick?

This is not simple... At first, I thought that solution 1 was better :
no need to transmit anything more than what the RFC says, we just detect
the change when the consumer reconnects. the problem is that we have no
idea if the consumer is the one which was registred before, unless it
says so. The transmitted data are the connection itself (IP, port), plus
the search request. If the config has changed, this is a new replication
request, and we must delet the old one. Now, how do we determinate if
the consumer was previously connected and should be deleted ? I don't
see any way to do that efficiently.

I'm thinking on a third solution there : we regularly check the
'sleeping' registred consumers on the producer, and if they haven't be
active for more than a determinated period of time, we simply delete
them (say, after 7 days).

This way, we don't store many entries for nothing, and we don't need to
send extra informations.

Does it sounds reasonnable ?


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com