Solr 5.2.1 - ReplicationHandler - No route to a host that is long gone

[Eric Torti <er...@gmail.com>]

Hey guys!

We have a deploy of SolrCloud 5.2.1 that is composed of 5 to 8 amazon linux
ec2 c3.2xlarge instances. Our main core is composed of 4M docs (6GB) and we
serve an average of 70 req/s per machine.

We are using zookeeper 3.4.6 to provide cluster synchronization. The thing
is we are noticing some weird "No route to host" exceptions on our logs. It
seems that the ReplicationHandler is trying to contact some other server
that used to be the cluster leader but is long gone now.

This behaviour is triggered when accessing this specific core's
"Dashboard".

http://my-server/solr/admin/collections?action=clusterstatus tells me this
former leader is down. So zookeeper knows about it. Any ideas on why the
ReplicationHandler is still trying to contact it? I'll attach the
stacktrace just to illustrate the situation.

Any help will be greatly appreciated.

Thanks!

Best,

Eric

'''

2015-Oct-06 18:18:02,446 [qtp1690716179-12764]
org.apache.solr.handler.ReplicationHandler

  WARN  Exception while invoking 'details' method for replication on master

org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at: http://10.10.10.10:8983/solr/my-core

at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)

at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)

at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)

at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)

at org.apache.solr.handler.IndexFetcher.getDetails(IndexFetcher.java:1563)

at
org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:821)

at
org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:305)

at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)

at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)

at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)

at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)

at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)

at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)

at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)

at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)

at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:497)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.net.NoRouteToHostException: No route to host

at java.net.PlainSocketImpl.socketConnect(Native Method)

at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)

at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)

at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)

at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)

at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)

at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)

at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)

at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)

at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)

at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)

... 32 more'''