[jira] [Updated] (SOLR-8173) CLONE - Leader recovery process can select the wrong leader if all replicas for a shard are down and trying to recover as well as lose updates that should have been recovered.

["Matteo Grolla (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-8173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matteo Grolla updated SOLR-8173:
--------------------------------
    Fix Version/s:     (was: 4.6.1)
                       (was: 4.7)
                       (was: Trunk)
                   5.2.1
      Description: 
I'm doing this test
collection test is replicated on two solr nodes running on 8983, 8984
using external zk

1)turn on solr 8984
2)add,commit a doc x con solr 8983
3)turn off solr 8983
4)turn on solr 8984
5)shortly after (leader still not elected) turn on solr 8983
6)8984 is elected as leader
7)doc x is present on 8983 but not on 8984 (check issuing a query)


  was:
One particular issue that leads to out-of-sync shards, related to SOLR-4260

Here's what I know so far, which admittedly isn't much:
As cloud85 (replica before it crashed) is initializing, it enters the wait process in ShardLeaderElectionContext#waitForReplicasToComeUp; this is expected and a good thing.
Some short amount of time in the future, cloud84 (leader before it crashed) begins initializing and gets to a point where it adds itself as a possible leader for the shard (by creating a znode under /collections/cloud/leaders_elect/shard1/election), which leads to cloud85 being able to return from waitForReplicasToComeUp and try to determine who should be the leader.
cloud85 then tries to run the SyncStrategy, which can never work because in this scenario the Jetty HTTP listener is not active yet on either node, so all replication work that uses HTTP requests fails on both nodes ... PeerSync treats these failures as indicators that the other replicas in the shard are unavailable (or whatever) and assumes success. Here's the log message:
2013-12-11 11:43:25,936 [coreLoadExecutor-3-thread-1] WARN solr.update.PeerSync - PeerSync: core=cloud_shard1_replica1 url=http://cloud85:8985/solr couldn't connect to http://cloud84:8984/solr/cloud_shard1_replica2/, counting as success
The Jetty HTTP listener doesn't start accepting connections until long after this process has completed and already selected the wrong leader.
>From what I can see, we seem to have a leader recovery process that is based partly on HTTP requests to the other nodes, but the HTTP listener on those nodes isn't active yet. We need a leader recovery process that doesn't rely on HTTP requests. Perhaps, leader recovery for a shard w/o a current leader may need to work differently than leader election in a shard that has replicas that can respond to HTTP requests? All of what I'm seeing makes perfect sense for leader election when there are active replicas and the current leader fails.
All this aside, I'm not asserting that this is the only cause for the out-of-sync issues reported in this ticket, but it definitely seems like it could happen in a real cluster.


> CLONE - Leader recovery process can select the wrong leader if all replicas for a shard are down and trying to recover as well as lose updates that should have been recovered.
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-8173
>                 URL: https://issues.apache.org/jira/browse/SOLR-8173
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>            Reporter: Matteo Grolla
>            Assignee: Mark Miller
>            Priority: Critical
>              Labels: leader, recovery
>             Fix For: 5.2.1
>
>
> I'm doing this test
> collection test is replicated on two solr nodes running on 8983, 8984
> using external zk
> 1)turn on solr 8984
> 2)add,commit a doc x con solr 8983
> 3)turn off solr 8983
> 4)turn on solr 8984
> 5)shortly after (leader still not elected) turn on solr 8983
> 6)8984 is elected as leader
> 7)doc x is present on 8983 but not on 8984 (check issuing a query)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org