You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Emmanuel Lecharny (JIRA)" <ji...@apache.org> on 2011/09/08 12:59:09 UTC
[jira] [Created] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Possible incorrect insertion of modifications in the consumer log
-----------------------------------------------------------------
Key: DIRSERVER-1655
URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
Project: Directory ApacheDS
Issue Type: Bug
Affects Versions: 2.0.0-M2
Reporter: Emmanuel Lecharny
Priority: Critical
Fix For: 2.0.0-M4
The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (DIRSERVER-1655) Possible incorrect
insertion of modifications in the consumer log
Posted by "Selcuk Aya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100268#comment-13100268 ]
Selcuk Aya edited comment on DIRSERVER-1655 at 9/8/11 12:48 PM:
----------------------------------------------------------------
CSN are consecutive and journal keeps the entries in sorted but the log entries are not inserted in sorted order.
I would still go with something like this:
////////////////////////////////
or maye do CSN creation under a lock and notify all consumer logs under this lock
///////////////////////
notify replication even interceptor OF CSN creation ->get log locks
generat csn,
insert an entry with key CSN and value null
notify replication even interceptor OF CSN creation->release log locs
do modification
notify replication event interceptor of modifaction
update the entry with key csn with the real log entry
When consumer log thread reads off this log in increasing CSN order, it will wait for a null entry to be non null.
was (Author: saya):
CSN are consecutive and journal keeps the entries in sorted but the log entries are not inserted in sorted order.
I would still go with something like this:
notify replication even interceptor OF CSN creation ->get log locks
generat csn,
insert an entry with key CSN and value null
notify replication even interceptor OF CSN creation->release log locs
do modification
notify replication event interceptor of modifaction
update the entry with key csn with the real log entry
When consumer log thread reads off this log in increasing CSN order, it will wait for a null entry to be non null.
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Kiran Ayyagari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100245#comment-13100245 ]
Kiran Ayyagari commented on DIRSERVER-1655:
-------------------------------------------
This delay of events can happen at any level before reaching the replication interceptor, IMO the best is to let the journal keep them in sorted order (and currently this CSN based sort order is maintained in the existing JDBM based journal implementation)
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Emmanuel Lecharny resolved DIRSERVER-1655.
------------------------------------------
Resolution: Won't Fix
Fix Version/s: 2.0.0-M8
Not an issue in trunk
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M8, 2.0.0-M7
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Selcuk Aya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100268#comment-13100268 ]
Selcuk Aya commented on DIRSERVER-1655:
---------------------------------------
CSN are consecutive and journal keeps the entries in sorted but the log entries are not inserted in sorted order.
I would still go with something like this:
notify replication even interceptor OF CSN creation ->get log locks
generat csn,
insert an entry with key CSN and value null
notify replication even interceptor OF CSN creation->release log locs
do modification
notify replication event interceptor of modifaction
update the entry with key csn with the real log entry
When consumer log thread reads off this log in increasing CSN order, it will wait for a null entry to be non null.
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Selcuk Aya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100229#comment-13100229 ]
Selcuk Aya commented on DIRSERVER-1655:
---------------------------------------
Have thought about this issue a little bit more after discussing it. The gist of it is that we do not insert the updates to the consumer log in CSN order and this might cause updates to be skipped when sending them to consumer. Inserting the modification to the consumer log using the same thread is not enough to solve this problem because the CSN for a modification seems to be gotten without any synchronization with regard to insertion into the consumer log. So if two threads update two different entries at CSN 9 and CSN 10 for example, these two modifications can be inserted into the consumer log in any order.
I think, for now, the solution would be to guarantee an execution order like this using a log lock:
notify replicationeventlistener for the modification before the csn is gotten and modification is done -> replicationeventlistener gets a lock for the log
get CSN
do the modification
notify replicationeventlistener after modification ->replication event listener inserts into the log and releases the log lock.
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100262#comment-13100262 ]
Emmanuel Lecharny commented on DIRSERVER-1655:
----------------------------------------------
IMO, it's not enough to rely on the fact that the journal is sorted, because CSN are not consecutive. We really need to guarantee that the modifications stored in the log are consecutive.
One solution could be to add a transient field in the CSN clas,s containing the order of creation (incremented by the CsnFactory).
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100278#comment-13100278 ]
Emmanuel Lecharny commented on DIRSERVER-1655:
----------------------------------------------
In fact, the key is to know if we accept the idea that modifications are done sequencially, not in parallel. IMO, it's a very high constraints
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Pierre-Arnaud Marcelot (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pierre-Arnaud Marcelot updated DIRSERVER-1655:
----------------------------------------------
Fix Version/s: (was: 2.0.0-M4)
2.0.0-M5
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M5
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Kiran Ayyagari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100264#comment-13100264 ]
Kiran Ayyagari commented on DIRSERVER-1655:
-------------------------------------------
CSN should be consecutive, and am sure it is, cause we include the timestamp and not only that we support the changeCount just incase if we create another CSN
at the same time.
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100277#comment-13100277 ]
Emmanuel Lecharny commented on DIRSERVER-1655:
----------------------------------------------
CSN aren't consecutive, you may insert a CSN between two existing ones.
By consecutive, I mean you can immediately know if a CSN follows the preceding one, or not.
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRSERVER-1655) Possible incorrect insertion of
modifications in the consumer log
Posted by "Emmanuel Lecharny (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRSERVER-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100240#comment-13100240 ]
Emmanuel Lecharny commented on DIRSERVER-1655:
----------------------------------------------
the problem is that the CSN is created way before the modification is inserted. Locking the log could last for a (relative) long time. The other problem is that we have no clue about which log we should lock, before processing the replication filter, so we may have to lock blindly *all* the consumer logs.
Can't we use a mechanism where each thread acquire a unique number, which will be used by the consumer log when it will process the mdoifications ? Something like :
get an order number from the log
do the modification
post modification, push the modififcation to the log system
the modification is inserted into the log if all the previous numbers have been processed
ie if a mod has a number N, then it can only be inserted into the log if the N-1, N-2, ... mods have already been processed. The log will keep the latest N it has processed.
> Possible incorrect insertion of modifications in the consumer log
> -----------------------------------------------------------------
>
> Key: DIRSERVER-1655
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1655
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 2.0.0-M2
> Reporter: Emmanuel Lecharny
> Priority: Critical
> Fix For: 2.0.0-M4
>
>
> The way we process modifications in the EventInterceptor, creating a new thread, make it possible that the modifications may be inserted into a consumer log in the wrong order.
> A possoble solution could be to use the same thread to insert modifications in the log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira