[GitHub] nickva opened a new pull request #1012: Fix replicator cluster stability race condition

[GitBox <gi...@apache.org>]

nickva opened a new pull request #1012: Fix replicator cluster stability race condition
URL: https://github.com/apache/couchdb/pull/1012
 
 
   Replicator clustering module is in charge of keeping track of when the cluster
   is stable or unstable. A cluster is said to be "stable" if there weren't any
   new nodes added or removed for some period of time.
   
   Replicator document processor uses cluster state to determine what to do with
   document updates as they come in. If a document update comes during a time when
   cluster is unstable, the update is skipped. The idea is that when cluster
   becomes stable again, all the documents will be rescanned anyway.
   
   In order for document updates to not be dropped, there is an implicit
   constraint when cluster becomes stable again (couch_replicator_notifier
   gen_event broadcasts `{cluster, stable}` event), any subsequent calls to
   `couch_replicator_clustering:is_stable()` would return `true`. If that's not
   the case then this sequence of events is possible:
   
   1. mem3_cluster process calls the cluster_stable event
   
   2. couch_replicator_notifier broadcasts {cluster, stable} event
   
   3. couch_replicator_doc_processor start processing documents
   
   4. On first document update couch_replicator_clustering:is_stable() is false,
   because that gen_server wasn't notifier yet.
   
   5. Document update is dropped.
   
   6. There won't be any rescans until cluster membership is changed again.
   
   To fix this, switch to setting stable state first via a gen_server call. This
   way after the `{cluster, stable}` has been called, is_stable() is guaranteed to
   return true.
   
   Note: This issue is mostly theoretical. It was noticed by examinig the code
   related to another bug. The chance of the clustering process going to sleep
   immediately after gen_event broadcast, then not handling the cast long enough
   for a document to be processed by doc processor is pretty low.
   
   #### Checklist
   
   - [x] Code is written and works correctly;
   - [x] Changes are covered by tests;
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services