[jira] [Comment Edited] (SOLR-7109) Indexing threads stuck during network partition can put leader into down state

["Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343551#comment-14343551 ] 

Shalin Shekhar Mangar edited comment on SOLR-7109 at 3/2/15 6:47 PM:
---------------------------------------------------------------------

Here's a patch which uses ZooKeeper 'multi' transactions to make sure that the LIR state can be set only when the requesting leader node is still alive. This ensures that regardless of how long the network partition lasts (long GC, whatever), the node setting the LIR state must be the leader or else the LIR state cannot be set.

Initially I attempted to use the shard leader path as the 'exists' check in the 'multi' command but this doesn't work because the leader path is always created fresh which means that it's version is always 0 and the check always succeeds regardless of who the current leader is. This is why we must use the election's leader sequence path.

This is just a first cut of this approach. I intend to refactor some of these LIR methods -- they have become too big. I will also write a test which exercises these new transactional semantics and reproduces the failure.

Edit - I also remove the replicaUrl parameter from ZkController.ensureReplicaInLeaderInitiatedRecovery because replicaProps were already being passed as a parameter and the replica url can be derived from it.


was (Author: shalinmangar):
Here's a patch which uses ZooKeeper 'multi' transactions to make sure that the LIR state can be set only when the requesting leader node is still alive. This ensures that regardless of how long the network partition lasts (long GC, whatever), the node setting the LIR state must be the leader or else the LIR state cannot be set.

Initially I attempted to use the shard leader path as the 'exists' check in the 'multi' command but this doesn't work because the leader path is always created fresh which means that it's version is always 0 and the check always succeeds regardless of who the current leader is. This is why we must use the election's leader sequence path.

This is just a first cut of this approach. I intend to refactor some of these LIR methods -- they have become too big. I will also write a test which exercises these new transactional semantics and reproduces the failure.

> Indexing threads stuck during network partition can put leader into down state
> ------------------------------------------------------------------------------
>
>                 Key: SOLR-7109
>                 URL: https://issues.apache.org/jira/browse/SOLR-7109
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>    Affects Versions: 4.10.3, 5.0
>            Reporter: Shalin Shekhar Mangar
>             Fix For: Trunk, 5.1
>
>         Attachments: SOLR-7109.patch
>
>
> I found this recently while running some Jepsen tests. I found that some threads get stuck on zk operations for a long time in ZkController.updateLeaderInitiatedRecoveryState method and when they wake up they go ahead with setting the LIR state to down. But in the mean time, new leader has been elected and sometimes you'd get into a state where the leader itself is put into recovery causing the shard to reject all writes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org