Leader election fails in some point.

[yriveiro <ya...@gmail.com>]

Hi,

In this screenshot I have a shard with two replicas without leader,

http://picpaste.com/qf2jdkj8.png

On machine with shard green I found this exception:

INFO  - dat5 - 2013-10-18 22:48:04.775;
org.apache.solr.handler.admin.CoreAdminHandler; Going to wait for
coreNodeName: 192.168.20.106:8983_solr_statistics-13_shard18_replica4,
state: recovering, checkLive: true, onlyIfLeader: true
ERROR - dat5 - 2013-10-18 22:48:04.775;
org.apache.solr.common.SolrException; org.apache.solr.common.SolrException:
We are not the leader
	at
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:824)
	at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)
	at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:655)
	at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
	at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
	at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
	at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
--
	at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
	at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
	at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
	at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Unknown Source)

On the machine with the shard in recovery state I found this exception:

INFO  - dat6 - 2013-10-18 22:48:44.131;
org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader process
for shard shard18
INFO  - dat6 - 2013-10-18 22:48:44.137;
org.apache.solr.cloud.ShardLeaderElectionContext; Checking if I should try
and be the leader.
INFO  - dat6 - 2013-10-18 22:48:44.138;
org.apache.solr.cloud.ShardLeaderElectionContext; My last published State
was recovering, I won't be the leader.
INFO  - dat6 - 2013-10-18 22:48:44.139;
org.apache.solr.cloud.ShardLeaderElectionContext; There may be a better
leader candidate than us - going back into recovery
INFO  - dat6 - 2013-10-18 22:48:44.142;
org.apache.solr.update.DefaultSolrCoreState; Running recovery - first
canceling any ongoing recovery
WARN  - dat6 - 2013-10-18 22:48:44.142;
org.apache.solr.cloud.RecoveryStrategy; Stopping recovery for
zkNodeName=192.168.20.106:8983_solr_statistics-13_shard18_replica4core=statistics-13_shard18_replica4
INFO  - dat6 - 2013-10-18 22:48:45.131;
org.apache.solr.cloud.RecoveryStrategy; Finished recovery process.
core=statistics-13_shard18_replica4
INFO  - dat6 - 2013-10-18 22:48:45.131;
org.apache.solr.cloud.RecoveryStrategy; Starting recovery process. 
core=statistics-13_shard18_replica4 recoveringAfterStartup=false
INFO  - dat6 - 2013-10-18 22:48:45.131; org.apache.solr.cloud.ZkController;
publishing core=statistics-13_shard18_replica4 state=recovering
INFO  - dat6 - 2013-10-18 22:48:45.132; org.apache.solr.cloud.ZkController;
numShards not found on descriptor - reading it from system property
INFO  - dat6 - 2013-10-18 22:48:45.141;
org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client,
config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
ERROR - dat6 - 2013-10-18 22:48:45.143;
org.apache.solr.common.SolrException; Error while trying to recover.
core=statistics-13_shard18_replica4:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
	at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:424)
	at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
	at
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:198)
	at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:342)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:219)

No leader means we can't index data because a 503 http status code is
returned.

Is this the normal behaviour or a bug?



-----
Best regards
--
View this message in context: http://lucene.472066.n3.nabble.com/Leader-election-fails-in-some-point-tp4096514.html
Sent from the Solr - User mailing list archive at Nabble.com.