[jira] [Commented] (COUCHDB-2965) Race condition in replicator rescan logic

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/COUCHDB-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186198#comment-15186198 ] 

ASF GitHub Bot commented on COUCHDB-2965:
-----------------------------------------

GitHub user nickva opened a pull request:

    https://github.com/apache/couchdb-couch-replicator/pull/31

    After a rescan prevent checkpoints from a previous epoch

    Add an `epoch` ref to State. On rescan update
    the epoch. Thread epoch through the change feed process
    and callbacks, then only allow checkpoints from current
    epoch.
    
    JIRA: COUCHDB-2965

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cloudant/couchdb-couch-replicator couchdb-2965

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/couchdb-couch-replicator/pull/31.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #31
    
----
commit 3e8a75184fb36d5dfece189e7b9dd22032ba1b74
Author: Nick Vatamaniuc <va...@gmail.com>
Date:   2016-03-09T00:37:17Z

    After a rescan prevent checkpoints from a previous epoch
    
    Add an `epoch` ref to State. On rescan update
    the epoch. Thread epoch through the change feed process
    and callbacks, then only allow checkpoints from current
    epoch.
    
    JIRA: COUCHDB-2965

----


> Race condition in replicator rescan logic
> -----------------------------------------
>
>                 Key: COUCHDB-2965
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-2965
>             Project: CouchDB
>          Issue Type: Bug
>          Components: Replication
>            Reporter: Nick Vatamaniuc
>
> There is race condition between the full rescan and regular change feed processing in the couch_replicator_manger code.
> This race condition would lead to replication docs left in untriggered state when a rescan of all the docs is performed. The rescan might happen when nodes connect and disconnect. The likelihood of this race condition appear goes up if a lot of documents are updated and there is a back-up of messages in the replicator manager's mailbox.
> The race condition happens in the following way:
> * A full rescan is initiated here: https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_manager.erl#L424 It clears the db_to_seq ets table which holds the latest change sequence for each replicator database. Then launches a scan_all_dbs process.
>  * scan_all_dbs will find all replicator-looking-like database and for each send a \{resume_scan, DbName\} message to the main couch_replicator_manager process.
>  * \{resume_scan, DbName\} message is handled here:  https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_manager.erl#L233 The expectation is because db_to_seq was reset it ends up not finding a sequence checkpoint in db_to_seq, so start 0 and spawns a new change feed, which will rescan all documents (since we need to determine ownership for them).  
> But the race condition occurs because when change feeds stop, they call  replicator manager with \{ rep_db_checkpoint, DbName \} message. That updates db_to_seq ets table with the latest change sequence: https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_manager.erl#L225 Which means this sequence of operations could happen:
>  * db_to_seq is reset to 0, scan_all_dbs is spawned
>  * change feed stops at sequence 1042, it calls \{rep_db_checkpoint, <<"_replicator">>\}
>  * \{rep_db_checkpoint, <<"_replicator">>\} call is handled, now latest db_to_seq for _replicator is 1042
>  * \{resume, <<"_replicator">>\} is sent from scan_all_dbs process and received by replicator manager. It sees that db_to_seq has _replicator with latest sequence 1042, so it will either start from that instead of 0, thus skipping updates from 0 to 1042.
> This was seen by running the experiment with1000 replication documents were being updated. Around document 700 or so , node1 was killed (pkill -f node1) . node2 experienced the race condition on rescan and never picked up a bunch of document that should have belong to it. didn't.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)