You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Adrian Fitzpatrick (JIRA)" <ji...@apache.org> on 2015/08/20 12:43:45 UTC
[jira] [Commented] (SOLR-7021) Leader will not publish core as
active without recovering first, but never recovers
[ https://issues.apache.org/jira/browse/SOLR-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704663#comment-14704663 ]
Adrian Fitzpatrick commented on SOLR-7021:
------------------------------------------
Also have seen this issue on Solr 4.10.3, on a 3 node cluster. Issue affected one of 3 collections only, and each of the 3 collections configured with 5 shards and 3 replicas. In the affected collection, for each of the 5 shards, the leader was on the same node (hadoopnode02) and was showing as down for all shards. Other replicas for each shard were reporting that were waiting for leader (eg "I was asked to wait on state recovering for shard3 in the_collection_20150818161800 on hadoopnode01:8983_solr but I still do not see the requested state. I see state: recovering live:true leader from ZK: http://hadoopnode02:8983/solr/the_collection_20150818161800_shard3_replica2")
Something like the work-around suggested by Andrey worked - we shut down the whole cluster, brought back up all nodes except the one which was reporting leader errors (hadoopnode02). This seemed to trigger a leader election but without a quorum. Then brought up hadoopnode02 - election then completed successfully and cluster state returned to normal.
> Leader will not publish core as active without recovering first, but never recovers
> -----------------------------------------------------------------------------------
>
> Key: SOLR-7021
> URL: https://issues.apache.org/jira/browse/SOLR-7021
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.10
> Reporter: James Hardwick
> Priority: Critical
> Labels: recovery, solrcloud, zookeeper
>
> A little background: 1 core solr-cloud cluster across 3 nodes, each with its own shard and each shard with a single replica hence each replica is itself a leader.
> For reasons we won't get into, we witnessed a shard go down in our cluster. We restarted the cluster but our core/shards still did not come back up. After inspecting the logs, we found this:
> {code}
> 015-01-21 15:51:56,494 [coreZkRegister-1-thread-2] INFO cloud.ZkController - We are http://xxx.xxx.xxx.35:8081/solr/xyzcore/ and leader is http://xxx.xxx.xxx.35:8081/solr/xyzcore/
> 2015-01-21 15:51:56,496 [coreZkRegister-1-thread-2] INFO cloud.ZkController - No LogReplay needed for core=xyzcore baseURL=http://xxx.xxx.xxx.35:8081/solr
> 2015-01-21 15:51:56,496 [coreZkRegister-1-thread-2] INFO cloud.ZkController - I am the leader, no recovery necessary
> 2015-01-21 15:51:56,496 [coreZkRegister-1-thread-2] INFO cloud.ZkController - publishing core=xyzcore state=active collection=xyzcore
> 2015-01-21 15:51:56,497 [coreZkRegister-1-thread-2] INFO cloud.ZkController - numShards not found on descriptor - reading it from system property
> 2015-01-21 15:51:56,498 [coreZkRegister-1-thread-2] INFO cloud.ZkController - publishing core=xyzcore state=down collection=xyzcore
> 2015-01-21 15:51:56,498 [coreZkRegister-1-thread-2] INFO cloud.ZkController - numShards not found on descriptor - reading it from system property
> 2015-01-21 15:51:56,501 [coreZkRegister-1-thread-2] ERROR core.ZkContainer - :org.apache.solr.common.SolrException: Cannot publish state of core 'xyzcore' as active without recovering first!
> at org.apache.solr.cloud.ZkController.publish(ZkController.java:1075)
> {code}
> And at this point the necessary shards never recover correctly and hence our core never returns to a functional state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org