You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Martin Alderson (JIRA)" <ji...@apache.org> on 2007/11/14 17:14:43 UTC

[jira] Created: (DIRSERVER-1098) New replicas may never be synchronised

New replicas may never be synchronised
--------------------------------------

                 Key: DIRSERVER-1098
                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1098
             Project: Directory ApacheDS
          Issue Type: Bug
          Components: mitosis
    Affects Versions: 1.5.1
            Reporter: Martin Alderson
            Assignee: Martin Alderson
            Priority: Minor


When the currently running replicas no longer have the original modification logs and a new replica is added and then modified before it has been brought up to date then it will never be brought up to date.

I am currently testing a fix for this which will disallow all modifications to the new replica until it has been brought up to date.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [jira] Created: (DIRSERVER-1098) New replicas may never be synchronised

Posted by Howard Chu <hy...@symas.com>.
Emmanuel Lecharny wrote:
> Howard Chu wrote:
>> Is there a doc somewhere that describes the replication protocol?
> Here you are :
> http://cwiki.apache.org/confluence/display/DIRxSRVx11/Mitosis
> 
> These pages are really very rough.

Thanks. I see you've read through all the previous failures on this path...

>> We encountered similar issues when developing the multimaster support
>> in OpenLDAP. Our solution was simply to assign unique server IDs to
>> each master, and record the last known modification for each known
>> server ID. This way local modifications don't overwrite the (lack of)
>> state from other servers.
>>
>> Another issue to consider when bringing up a new replica - you may
>> want to disallow all queries completely, until the replica has fully
>> sync'd up. If you allow searches during this initial phase, a client
>> is likely to get a partial set of results. In some cases, that may be
>> worse than getting no result...

> The pb is that if you have a lot of replication, it may not be a viable
> approach. Also if you want to implement the 'paging control', as it's up
> to the client to request the next results, you have no good solutions...

> At some point, I think we should consider that a LDAP server is not like
> a RDBMS system with a full transaction system, and you must assume that
> dirty read is a standard mode. That does not mean we shouldn't consider
> blocking incoming requests until a sync-up is done when a server is
> restarted.

In OpenLDAP I've recommended that a replica chain all incoming requests to a 
provider while it's in startup mode. There can be transient inconsistencies at 
later points in its runtime, and we live with those. But if you're bringing up 
a brand new server, it's best to pretend it doesn't even exist until it's at 
least 80-90% sync'd. Obviously you can manage this outside of LDAP (e.g., if 
you have a DNS pool or hardware load-balancer directing clients to servers, 
don't add the new replica to the pool until it's ready) for a new server.

> This is not a simple problem...

Heh heh. So very true.

-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Re: [jira] Created: (DIRSERVER-1098) New replicas may never be synchronised

Posted by Emmanuel Lecharny <el...@gmail.com>.
Howard Chu wrote:
>
> Is there a doc somewhere that describes the replication protocol?
Here you are :
http://cwiki.apache.org/confluence/display/DIRxSRVx11/Mitosis

These pages are really very rough.
>
> We encountered similar issues when developing the multimaster support 
> in OpenLDAP. Our solution was simply to assign unique server IDs to 
> each master, and record the last known modification for each known 
> server ID. This way local modifications don't overwrite the (lack of) 
> state from other servers.
>
> Another issue to consider when bringing up a new replica - you may 
> want to disallow all queries completely, until the replica has fully 
> sync'd up. If you allow searches during this initial phase, a client 
> is likely to get a partial set of results. In some cases, that may be 
> worse than getting no result...
The pb is that if you have a lot of replication, it may not be a viable 
approach. Also if you want to implement the 'paging control', as it's up 
to the client to request the next results, you have no good solutions...

At some point, I think we should consider that a LDAP server is not like 
a RDBMS system with a full transaction system, and you must assume that 
dirty read is a standard mode. That does not mean we shouldn't consider 
blocking incoming requests until a sync-up is done when a server is 
restarted.

This is not a simple problem...


-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

Re: [jira] Created: (DIRSERVER-1098) New replicas may never be synchronised

Posted by Martin Alderson <eq...@planetquake.com>.
Hi Howard,

 > Is there a doc somewhere that describes the replication protocol?

I don't think so.  At the moment we just serialise Java objects and send 
them over the wire.  We plan to add other protocols in the future.

 > Another issue to consider when bringing up a new replica - you may
 > want to disallow all queries completely, until the replica has
 > fully sync'd up. If you allow searches during this initial phase,
 > a client is likely to get a partial set of results. In some cases,
 > that may be worse than getting no result...

I think that since we allow modifications on multiple replicas 
simultaneously we can get that situation at any point anyway.

Martin

Re: [jira] Created: (DIRSERVER-1098) New replicas may never be synchronised

Posted by Howard Chu <hy...@symas.com>.
Martin Alderson (JIRA) wrote:
> New replicas may never be synchronised
> --------------------------------------
> 
>                  Key: DIRSERVER-1098
>                  URL: https://issues.apache.org/jira/browse/DIRSERVER-1098
>              Project: Directory ApacheDS
>           Issue Type: Bug
>           Components: mitosis
>     Affects Versions: 1.5.1
>             Reporter: Martin Alderson
>             Assignee: Martin Alderson
>             Priority: Minor
> 
> 
> When the currently running replicas no longer have the original
> modification
logs and a new replica is added and then modified before it has been brought
up to date then it will never be brought up to date.
> 
> I am currently testing a fix for this which will disallow all
> modifications
to the new replica until it has been brought up to date.

Is there a doc somewhere that describes the replication protocol?

We encountered similar issues when developing the multimaster support in 
OpenLDAP. Our solution was simply to assign unique server IDs to each master, 
and record the last known modification for each known server ID. This way 
local modifications don't overwrite the (lack of) state from other servers.

Another issue to consider when bringing up a new replica - you may want to 
disallow all queries completely, until the replica has fully sync'd up. If you 
allow searches during this initial phase, a client is likely to get a partial 
set of results. In some cases, that may be worse than getting no result...
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

[jira] Updated: (DIRSERVER-1098) New replicas may never be synchronised

Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DIRSERVER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Lecharny updated DIRSERVER-1098:
-----------------------------------------

    Fix Version/s:     (was: 1.5.5)
                   1.5.7

Moved to 1.5.7, where Mitosis will be reviewed

> New replicas may never be synchronised
> --------------------------------------
>
>                 Key: DIRSERVER-1098
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1098
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: mitosis
>    Affects Versions: 1.5.1
>            Reporter: Martin Alderson
>            Assignee: Martin Alderson
>            Priority: Minor
>             Fix For: 1.5.7
>
>
> When the currently running replicas no longer have the original modification logs and a new replica is added and then modified before it has been brought up to date then it will never be brought up to date.
> I am currently testing a fix for this which will disallow all modifications to the new replica until it has been brought up to date.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (DIRSERVER-1098) New replicas may never be synchronised

Posted by "Alex Karasulu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DIRSERVER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Karasulu updated DIRSERVER-1098:
-------------------------------------

    Fix Version/s:     (was: 1.5.3)
                   1.5.5

> New replicas may never be synchronised
> --------------------------------------
>
>                 Key: DIRSERVER-1098
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1098
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: mitosis
>    Affects Versions: 1.5.1
>            Reporter: Martin Alderson
>            Assignee: Martin Alderson
>            Priority: Minor
>             Fix For: 1.5.5
>
>
> When the currently running replicas no longer have the original modification logs and a new replica is added and then modified before it has been brought up to date then it will never be brought up to date.
> I am currently testing a fix for this which will disallow all modifications to the new replica until it has been brought up to date.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (DIRSERVER-1098) New replicas may never be synchronised

Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DIRSERVER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Lecharny updated DIRSERVER-1098:
-----------------------------------------

    Fix Version/s: 1.5.2

Martin, do we have a status for this guy ? Should we postpone it to 1.5.3 ?

> New replicas may never be synchronised
> --------------------------------------
>
>                 Key: DIRSERVER-1098
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1098
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: mitosis
>    Affects Versions: 1.5.1
>            Reporter: Martin Alderson
>            Assignee: Martin Alderson
>            Priority: Minor
>             Fix For: 1.5.2
>
>
> When the currently running replicas no longer have the original modification logs and a new replica is added and then modified before it has been brought up to date then it will never be brought up to date.
> I am currently testing a fix for this which will disallow all modifications to the new replica until it has been brought up to date.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (DIRSERVER-1098) New replicas may never be synchronised

Posted by "Martin Alderson (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DIRSERVER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Alderson updated DIRSERVER-1098:
---------------------------------------

    Fix Version/s:     (was: 1.5.2)
                   1.5.3

Won't be able to get this in on time and not urgent.

> New replicas may never be synchronised
> --------------------------------------
>
>                 Key: DIRSERVER-1098
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1098
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: mitosis
>    Affects Versions: 1.5.1
>            Reporter: Martin Alderson
>            Assignee: Martin Alderson
>            Priority: Minor
>             Fix For: 1.5.3
>
>
> When the currently running replicas no longer have the original modification logs and a new replica is added and then modified before it has been brought up to date then it will never be brought up to date.
> I am currently testing a fix for this which will disallow all modifications to the new replica until it has been brought up to date.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (DIRSERVER-1098) New replicas may never be synchronised

Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DIRSERVER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Lecharny updated DIRSERVER-1098:
-----------------------------------------

    Fix Version/s:     (was: 1.5.7)
                   2.0.0-RC1

Moved to 2.0.0-RC1, we won't release a 1.5.7

> New replicas may never be synchronised
> --------------------------------------
>
>                 Key: DIRSERVER-1098
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1098
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: mitosis
>    Affects Versions: 1.5.1
>            Reporter: Martin Alderson
>            Assignee: Martin Alderson
>            Priority: Minor
>             Fix For: 2.0.0-RC1
>
>
> When the currently running replicas no longer have the original modification logs and a new replica is added and then modified before it has been brought up to date then it will never be brought up to date.
> I am currently testing a fix for this which will disallow all modifications to the new replica until it has been brought up to date.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Closed: (DIRSERVER-1098) New replicas may never be synchronised

Posted by "Kiran Ayyagari (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/DIRSERVER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kiran Ayyagari closed DIRSERVER-1098.
-------------------------------------

      Assignee: Kiran Ayyagari  (was: Martin Alderson)
    Resolution: Won't Fix

Mitosis is no longer present in the server, a new replication system is has been developed based on RC 4533

> New replicas may never be synchronised
> --------------------------------------
>
>                 Key: DIRSERVER-1098
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1098
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: mitosis
>    Affects Versions: 1.5.1
>            Reporter: Martin Alderson
>            Assignee: Kiran Ayyagari
>            Priority: Minor
>             Fix For: 2.0.0-RC1
>
>
> When the currently running replicas no longer have the original modification logs and a new replica is added and then modified before it has been brought up to date then it will never be brought up to date.
> I am currently testing a fix for this which will disallow all modifications to the new replica until it has been brought up to date.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.