SolrCloud RemoteSolrException: We are not the leader

[Alexey Kozhemiakin <Al...@epam.com>]

Dear All,

We are facing strange issue with SolrCloud (4.4 with Embedded Zookeeper). Cluster consists of 2 shards and 4 nodes. 4th node cannot be added to cluster and stays in "recovering" state with following error in logs. Picture from admin cloud interface http://imageshack.us/photo/my-images/577/0jus.png/.

Do you have any ideas what's wrong?


137467 [main-EventThread] INFO  org.apache.solr.common.cloud.ZkStateReader  û A cluster state change: WatchedEvent state:Syn
cConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
224734 [qtp155768696-18] INFO  org.apache.solr.servlet.SolrDispatchFilter  û [admin] webapp=null path=/admin/cores params={i
ndexInfo=false&_=1375696898299&wt=json} status=0 QTime=0
224765 [qtp155768696-18] INFO  org.apache.solr.core.SolrCore  û [prices] webapp=/solr path=/admin/system params={_=137569689
8327&wt=json} status=0 QTime=0
266666 [RecoveryThread] INFO  org.apache.solr.cloud.ZkController  û publishing core=prices state=recovering
266666 [RecoveryThread] INFO  org.apache.solr.cloud.ZkController  û numShards not found on descriptor - reading it from syst
em property
266666 [RecoveryThread] INFO  org.apache.solr.client.solrj.impl.HttpClientUtil  û Creating new http client, config:maxConnec
tions=128&maxConnectionsPerHost=32&followRedirects=false
266713 [RecoveryThread] ERROR org.apache.solr.cloud.RecoveryStrategy  û Error while trying to recover. core=prices:org.apach
e.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: We are not the leader
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:424)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:198)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:342)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:219)

266728 [RecoveryThread] ERROR org.apache.solr.cloud.RecoveryStrategy  û Recovery failed - trying again... (7) core=prices
266728 [RecoveryThread] INFO  org.apache.solr.cloud.RecoveryStrategy  û Wait 256.0 seconds before trying to recover again (8
)