You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by philippa griggs <ph...@hotmail.co.uk> on 2016/10/10 15:43:05 UTC
Zookeeper connection issues
Hello,
Solr Set up
Solr 5.4.1, Zookeeper 3.4.6 (5 zookeeper ensemble)
We have one collection which has multiple shards (two shards for each week). Each shard has a leader and a replica. We only write to the latest week- two shards (four cores) which we refer to a ‘hot cores’. The rest, ‘cold cores’ are for queries. We have multiple solr processes running on an instance- currently 5 each with a 15Gb Heap (there is 122G available memory). As the index grows as the week goes on the heap size starts low and increase to around 9/10Gb. The index size on each core ends up around 8 million docs, 6.5Gb which are stored on 40Gb drives. The zookeeper timeout is 60Secs.
The issue:
We are experiencing issues with connectivity and have started seeing errors messages about being unable to connect to zookeeper. Most of the time solr recovers itself after a while but we are seeing these ‘blips’ more and more often with the last ‘blip’ ending up with manually restarting the hot cores. So far this has only been seen on one shard at a time. All other shards in the cluster don’t have an issue.
There is nothing in the zookeeper log. Below are the solr logs for the last ‘blip’.
I’ve looked at the heap size and its not hitting 15Gb (max around 11Gb). At around the time of the blip the GC is 40sec, which is not over the timeout but is however much larger than we normally see.
These blips are happening towards the end of the week when the index size gets larger.
I’m not sure what is going on, is this a zookeeper issue or solr? What would be causing solr to lose connection with zookeeper if it’s not the timeout? We have checked the network and it doesn’t indicate a network issue.
Any suggests would be useful.
Error Logs for core A
2016-10-08 18:45:36.617 WARN (qtp697960108-32664) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.e.j.h.HttpParser badMessage: java.lang.IllegalStateException: too much data after closed for HttpChannelOverHttp@1b87c2cf{r=1370,c=false,a=IDLE,uri=-}
2016-10-08 18:45:36.717 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.c.ZkController Unable to read /collections/xxx/leader_initiated_recovery/20161003_A/20161003_A54130 due to: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /collections/xxx/leader_initiated_recovery/20161003_A/20161003_A54130
2016-10-08 18:45:44.907 ERROR (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.u.PeerSync PeerSync: core=xxx url=http://x.x.x.x:8987/solr ERROR, update log not in ACTIVE or REPLAY state. FSUpdateLog{state=BUFFERING, tlog=tlog{file=/solrLog_8987/tlog/tlog.0000000000000011469 refcount=1}}
2016-10-08 18:45:44.908 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.u.PeerSync PeerSync: core=xxx url=http://x.x.x.x:8987/solr too many updates received since start - startingUpdates no longer overlaps with our currentUpdates
2016-10-08 18:47:25.772 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1ftq.si did not match. expected checksum is 4254234714 and actual is checksum 2090625558. expected length is 422 and actual length is 422
2016-10-08 18:47:26.286 WARN (zkCallback-3-thread-76-processing-n:x.x.x.x:8987_solr-EventThread) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx coreNodeName=20161003_A54130
2016-10-08 18:47:54.935 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1ftq.si did not match. expected checksum is 4254234714 and actual is checksum 2090625558. expected length is 422 and actual length is 422
2016-10-08 18:47:54.939 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1ftq.cfs did not match. expected checksum is 3006669569 and actual is checksum 3691917. expected length is 35641114 and actual length is 8402832
2016-10-08 18:47:55.084 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1ftq.cfe did not match. expected checksum is 3692862263 and actual is checksum 3783720915. expected length is 289 and actual length is 289
2016-10-08 18:47:55.399 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1fti.cfs did not match. expected checksum is 3751057985 and actual is checksum 396270944. expected length is 9214354 and actual length is 7147563
…
We then start getting errors
2016-10-08 18:51:31.202 WARN (coreZkRegister-1-thread-8-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx coreNodeName=20161003_A54130
2016-10-08 18:51:31.206 ERROR (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.c.RecoveryStrategy Error while trying to recover. core=xxx:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://x.x.x.x:8987/solr
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:607)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:364)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://x.x.x.x:8987/solr
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:589)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:285)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:281)
... 5 more
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
2016-10-08 18:51:42.896 WARN (coreZkRegister-1-thread-9-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx coreNodeName=20161003_A54130
2016-10-08 18:52:16.552 ERROR (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.u.PeerSync PeerSync: core=xxx url=http://x.x.x.x:8987/solr ERROR, update log not in ACTIVE or REPLAY state. FSUpdateLog{state=BUFFERING, tlog=tlog{file=/solrLog_8987/tlog/tlog.0000000000000011469 refcount=1}}
2016-10-08 18:52:16.553 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.u.PeerSync PeerSync: core=xxx url=http://x.x.x.x:8987/solr too many updates received since start - startingUpdates no longer overlaps with our currentUpdates
2016-10-08 19:04:40.098 WARN (zkCallback-3-thread-196-processing-n:x.x.x.x:8987_solr) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx coreNodeName=20161003_A54130
2016-10-08 19:05:11.864 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.c.SolrCore [xxx] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
2016-10-08 19:07:40.446 WARN (zkCallback-3-thread-76-processing-n:x.x.x.x:8987_solr-EventThread) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx coreNodeName=20161003_A54130
2016-10-08 19:10:22.427 ERROR (qtp697960108-32836) [c:xxx s:20161003_A r:20161003_A54130 x:xxx] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: xxx slice: 20161003_A
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:643)
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:629)
Error Logs for core B
2016-10-08 18:45:36.618 ERROR (updateExecutor-2-thread-8085-processing-http:////x.x.x.x:8987//solr//xxx x:xxx r:20161003_A54129 n:x.x.x.x:8987_solr s:20161003_A c:xxx) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.u.StreamingSolrClients error
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:160)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 18:45:36.618 ERROR (updateExecutor-2-thread-8083-processing-http:////x.x.x.x:8987//solr//xxx x:xxx r:20161003_A54129 n:x.x.x.x:8987_solr s:20161003_A c:xxx) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.u.StreamingSolrClients error
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:160)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 18:45:36.619 WARN (qtp2065530879-34668) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Error sending update to http://x.x.x.x:8987/solr
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
2016-10-08 18:45:36.620 WARN (qtp2065530879-34670) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Error sending update to http://x.x.x.x:8987/solr
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:160)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 18:45:36.620 ERROR (qtp2065530879-34668) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://x.x.x.x:8987/solr/xxx/
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:160)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 18:45:36.620 ERROR (qtp2065530879-34670) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://x.x.x.x:8987/solr/xxx/
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
2016-10-08 19:10:22.248 ERROR (qtp2065530879-34820) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: xxx slice: 20161003_A
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:643)
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:629)
at org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:360)
at org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:311)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:661)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:251)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:178)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:95)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:70)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 19:34:05.557 ERROR (coreZkRegister-1-thread-15-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54129) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.c.ZkController Error getting leader from zk
org.apache.solr.common.SolrException: No registered leader was found after waiting for 1560000ms , collection: xxx slice: 20161003_A
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:643)
at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:610)
at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1005)
at org.apache.solr.cloud.ZkController.register(ZkController.java:940)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:234)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 19:34:06.101 ERROR (coreZkRegister-1-thread-14-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A c:xxx r:20161003_A54129) [c:xxx s:20161003_A r:20161003_A54129 x:xxx] o.a.s.c.ZkController Error getting leader from zk
org.apache.solr.common.SolrException: No registered leader was found after waiting for 1560000ms , collection: xxx slice: 20161003_A
Re: Zookeeper connection issues
Posted by Pushkar Raste <pu...@gmail.com>.
If Solr has GC pauses greater than 15 seconds, zookeeper is going to assume
node is down and hence would send it into recovery when node comes out of a
GC pause and reconnects to zookeeper.
You should look into keeping GC pause as short as possible.
Using G1GC with ParallelRefProcEnabled has helped me a lot, but you may
want to come to settings that work best for using trial and error.
If you are not using MMapDirectory try switching to it. It helps by keeping
index off the heap.
Also check if issue SOLR-9310 affects you, since your replicas are going
into recovery, I am afraid it is doing full index copy than just picking up
the delta.
On Oct 10, 2016 11:43 AM, "philippa griggs" <ph...@hotmail.co.uk>
wrote:
> Hello,
>
>
> Solr Set up
>
>
> Solr 5.4.1, Zookeeper 3.4.6 (5 zookeeper ensemble)
>
>
> We have one collection which has multiple shards (two shards for each
> week). Each shard has a leader and a replica. We only write to the latest
> week- two shards (four cores) which we refer to a ‘hot cores’. The rest,
> ‘cold cores’ are for queries. We have multiple solr processes running on an
> instance- currently 5 each with a 15Gb Heap (there is 122G available
> memory). As the index grows as the week goes on the heap size starts low
> and increase to around 9/10Gb. The index size on each core ends up around 8
> million docs, 6.5Gb which are stored on 40Gb drives. The zookeeper timeout
> is 60Secs.
>
>
> The issue:
>
>
> We are experiencing issues with connectivity and have started seeing
> errors messages about being unable to connect to zookeeper. Most of the
> time solr recovers itself after a while but we are seeing these ‘blips’
> more and more often with the last ‘blip’ ending up with manually restarting
> the hot cores. So far this has only been seen on one shard at a time. All
> other shards in the cluster don’t have an issue.
>
>
> There is nothing in the zookeeper log. Below are the solr logs for the
> last ‘blip’.
>
>
> I’ve looked at the heap size and its not hitting 15Gb (max around 11Gb).
> At around the time of the blip the GC is 40sec, which is not over the
> timeout but is however much larger than we normally see.
>
>
> These blips are happening towards the end of the week when the index size
> gets larger.
>
>
> I’m not sure what is going on, is this a zookeeper issue or solr? What
> would be causing solr to lose connection with zookeeper if it’s not the
> timeout? We have checked the network and it doesn’t indicate a network
> issue.
>
>
> Any suggests would be useful.
>
>
>
> Error Logs for core A
>
>
> 2016-10-08 18:45:36.617 WARN (qtp697960108-32664) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.e.j.h.HttpParser badMessage: java.lang.IllegalStateException:
> too much data after closed for HttpChannelOverHttp@1b87c2cf{
> r=1370,c=false,a=IDLE,uri=-}
>
> 2016-10-08 18:45:36.717 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.c.ZkController Unable to read
> /collections/xxx/leader_initiated_recovery/20161003_A/20161003_A54130 due
> to: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /collections/xxx/leader_
> initiated_recovery/20161003_A/20161003_A54130
>
> 2016-10-08 18:45:44.907 ERROR (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.u.PeerSync PeerSync: core=xxx url=
> http://x.x.x.x:8987/solr ERROR, update log not in ACTIVE or REPLAY state.
> FSUpdateLog{state=BUFFERING, tlog=tlog{file=/solrLog_8987/tlog/tlog.0000000000000011469
> refcount=1}}
>
> 2016-10-08 18:45:44.908 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.u.PeerSync PeerSync: core=xxx url=
> http://x.x.x.x:8987/solr too many updates received since start -
> startingUpdates no longer overlaps with our currentUpdates
>
> 2016-10-08 18:47:25.772 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1ftq.si did not
> match. expected checksum is 4254234714 and actual is checksum 2090625558.
> expected length is 422 and actual length is 422
>
> 2016-10-08 18:47:26.286 WARN (zkCallback-3-thread-76-
> processing-n:x.x.x.x:8987_solr-EventThread) [ ]
> o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx
> coreNodeName=20161003_A54130
>
> 2016-10-08 18:47:54.935 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1ftq.si did not
> match. expected checksum is 4254234714 and actual is checksum 2090625558.
> expected length is 422 and actual length is 422
>
> 2016-10-08 18:47:54.939 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1ftq.cfs did not match.
> expected checksum is 3006669569 and actual is checksum 3691917. expected
> length is 35641114 and actual length is 8402832
>
> 2016-10-08 18:47:55.084 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1ftq.cfe did not match.
> expected checksum is 3692862263 and actual is checksum 3783720915. expected
> length is 289 and actual length is 289
>
> 2016-10-08 18:47:55.399 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.h.IndexFetcher File _1fti.cfs did not match.
> expected checksum is 3751057985 and actual is checksum 396270944. expected
> length is 9214354 and actual length is 7147563
>
> …
>
>
> We then start getting errors
>
>
> 2016-10-08 18:51:31.202 WARN (coreZkRegister-1-thread-8-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.c.RecoveryStrategy Stopping recovery for
> core=xxx coreNodeName=20161003_A54130
>
> 2016-10-08 18:51:31.206 ERROR (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.c.RecoveryStrategy Error while trying to
> recover. core=xxx:java.util.concurrent.ExecutionException:
> org.apache.solr.client.solrj.SolrServerException: IOException occured
> when talking to server at: http://x.x.x.x:8987/solr
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
> at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(
> RecoveryStrategy.java:607)
>
> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(
> RecoveryStrategy.java:364)
>
> at org.apache.solr.cloud.RecoveryStrategy.run(
> RecoveryStrategy.java:226)
>
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.solr.client.solrj.SolrServerException: IOException
> occured when talking to server at: http://x.x.x.x:8987/solr
>
> at org.apache.solr.client.solrj.impl.HttpSolrClient.
> executeMethod(HttpSolrClient.java:589)
>
> at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(
> HttpSolrClient.java:285)
>
> at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(
> HttpSolrClient.java:281)
>
> ... 5 more
>
> Caused by: java.net.SocketException: Socket closed
>
> at java.net.SocketInputStream.socketRead0(Native Method)
>
> at java.net.SocketInputStream.socketRead(SocketInputStream.
> java:116)
>
>
> 2016-10-08 18:51:42.896 WARN (coreZkRegister-1-thread-9-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.c.RecoveryStrategy Stopping recovery for
> core=xxx coreNodeName=20161003_A54130
>
> 2016-10-08 18:52:16.552 ERROR (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.u.PeerSync PeerSync: core=xxx url=
> http://x.x.x.x:8987/solr ERROR, update log not in ACTIVE or REPLAY state.
> FSUpdateLog{state=BUFFERING, tlog=tlog{file=/solrLog_8987/tlog/tlog.0000000000000011469
> refcount=1}}
>
> 2016-10-08 18:52:16.553 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.u.PeerSync PeerSync: core=xxx url=
> http://x.x.x.x:8987/solr too many updates received since start -
> startingUpdates no longer overlaps with our currentUpdates
>
> 2016-10-08 19:04:40.098 WARN (zkCallback-3-thread-196-
> processing-n:x.x.x.x:8987_solr) [c:xxx s:20161003_A r:20161003_A54130
> x:xxx] o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx
> coreNodeName=20161003_A54130
>
> 2016-10-08 19:05:11.864 WARN (updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54130) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.c.SolrCore [xxx] PERFORMANCE WARNING:
> Overlapping onDeckSearchers=2
>
> 2016-10-08 19:07:40.446 WARN (zkCallback-3-thread-76-
> processing-n:x.x.x.x:8987_solr-EventThread) [ ]
> o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx
> coreNodeName=20161003_A54130
>
> 2016-10-08 19:10:22.427 ERROR (qtp697960108-32836) [c:xxx s:20161003_A
> r:20161003_A54130 x:xxx] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
> No registered leader was found after waiting for 4000ms , collection: xxx
> slice: 20161003_A
>
> at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(
> ZkStateReader.java:643)
>
> at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(
> ZkStateReader.java:629)
>
>
>
>
>
>
>
> Error Logs for core B
>
>
> 2016-10-08 18:45:36.618 ERROR (updateExecutor-2-thread-8085-
> processing-http:////x.x.x.x:8987//solr//xxx x:xxx r:20161003_A54129
> n:x.x.x.x:8987_solr s:20161003_A c:xxx) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.u.StreamingSolrClients error
>
> org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:143)
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:57)
>
> at org.apache.http.impl.io.AbstractMessageParser.parse(
> AbstractMessageParser.java:261)
>
> at org.apache.http.impl.AbstractHttpClientConnection.
> receiveResponseHeader(AbstractHttpClientConnection.java:283)
>
> at org.apache.http.impl.conn.DefaultClientConnection.
> receiveResponseHeader(DefaultClientConnection.java:251)
>
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.
> receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(
> HttpRequestExecutor.java:272)
>
> at org.apache.http.protocol.HttpRequestExecutor.execute(
> HttpRequestExecutor.java:124)
>
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(
> DefaultRequestDirector.java:685)
>
> at org.apache.http.impl.client.DefaultRequestDirector.execute(
> DefaultRequestDirector.java:487)
>
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(
> AbstractHttpClient.java:882)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:82)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:107)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:55)
>
> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$
> Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
>
> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$
> Runner.run(ConcurrentUpdateSolrClient.java:160)
>
> at org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-10-08 18:45:36.618 ERROR (updateExecutor-2-thread-8083-
> processing-http:////x.x.x.x:8987//solr//xxx x:xxx r:20161003_A54129
> n:x.x.x.x:8987_solr s:20161003_A c:xxx) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.u.StreamingSolrClients error
>
> org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:143)
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:57)
>
> at org.apache.http.impl.io.AbstractMessageParser.parse(
> AbstractMessageParser.java:261)
>
> at org.apache.http.impl.AbstractHttpClientConnection.
> receiveResponseHeader(AbstractHttpClientConnection.java:283)
>
> at org.apache.http.impl.conn.DefaultClientConnection.
> receiveResponseHeader(DefaultClientConnection.java:251)
>
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.
> receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(
> HttpRequestExecutor.java:272)
>
> at org.apache.http.protocol.HttpRequestExecutor.execute(
> HttpRequestExecutor.java:124)
>
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(
> DefaultRequestDirector.java:685)
>
> at org.apache.http.impl.client.DefaultRequestDirector.execute(
> DefaultRequestDirector.java:487)
>
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(
> AbstractHttpClient.java:882)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:82)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:107)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:55)
>
> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$
> Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
>
> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$
> Runner.run(ConcurrentUpdateSolrClient.java:160)
>
> at org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-10-08 18:45:36.619 WARN (qtp2065530879-34668) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Error
> sending update to http://x.x.x.x:8987/solr
>
> org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:143)
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:57)
>
> at org.apache.http.impl.io.AbstractMessageParser.parse(
> AbstractMessageParser.java:261)
>
> 2016-10-08 18:45:36.620 WARN (qtp2065530879-34670) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Error
> sending update to http://x.x.x.x:8987/solr
>
> org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:143)
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:57)
>
> at org.apache.http.impl.io.AbstractMessageParser.parse(
> AbstractMessageParser.java:261)
>
> at org.apache.http.impl.AbstractHttpClientConnection.
> receiveResponseHeader(AbstractHttpClientConnection.java:283)
>
> at org.apache.http.impl.conn.DefaultClientConnection.
> receiveResponseHeader(DefaultClientConnection.java:251)
>
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.
> receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(
> HttpRequestExecutor.java:272)
>
> at org.apache.http.protocol.HttpRequestExecutor.execute(
> HttpRequestExecutor.java:124)
>
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(
> DefaultRequestDirector.java:685)
>
> at org.apache.http.impl.client.DefaultRequestDirector.execute(
> DefaultRequestDirector.java:487)
>
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(
> AbstractHttpClient.java:882)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:82)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:107)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:55)
>
> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$
> Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
>
> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$
> Runner.run(ConcurrentUpdateSolrClient.java:160)
>
> at org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-10-08 18:45:36.620 ERROR (qtp2065530879-34668) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Setting up
> to try to start recovery on replica http://x.x.x.x:8987/solr/xxx/
>
> org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:143)
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:57)
>
> at org.apache.http.impl.io.AbstractMessageParser.parse(
> AbstractMessageParser.java:261)
>
> at org.apache.http.impl.AbstractHttpClientConnection.
> receiveResponseHeader(AbstractHttpClientConnection.java:283)
>
> at org.apache.http.impl.conn.DefaultClientConnection.
> receiveResponseHeader(DefaultClientConnection.java:251)
>
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.
> receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(
> HttpRequestExecutor.java:272)
>
> at org.apache.http.protocol.HttpRequestExecutor.execute(
> HttpRequestExecutor.java:124)
>
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(
> DefaultRequestDirector.java:685)
>
> at org.apache.http.impl.client.DefaultRequestDirector.execute(
> DefaultRequestDirector.java:487)
>
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(
> AbstractHttpClient.java:882)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:82)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:107)
>
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:55)
>
> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$
> Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
>
> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$
> Runner.run(ConcurrentUpdateSolrClient.java:160)
>
> at org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-10-08 18:45:36.620 ERROR (qtp2065530879-34670) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Setting up
> to try to start recovery on replica http://x.x.x.x:8987/solr/xxx/
>
> org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:143)
>
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
> DefaultHttpResponseParser.java:57)
>
> at org.apache.http.impl.io.AbstractMessageParser.parse(
> AbstractMessageParser.java:261)
>
> at org.apache.http.impl.AbstractHttpClientConnection.
> receiveResponseHeader(AbstractHttpClientConnection.java:283)
>
> 2016-10-08 19:10:22.248 ERROR (qtp2065530879-34820) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
> No registered leader was found after waiting for 4000ms , collection: xxx
> slice: 20161003_A
>
> at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(
> ZkStateReader.java:643)
>
> at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(
> ZkStateReader.java:629)
>
> at org.apache.solr.update.processor.DistributedUpdateProcessor.
> setupRequest(DistributedUpdateProcessor.java:360)
>
> at org.apache.solr.update.processor.DistributedUpdateProcessor.
> setupRequest(DistributedUpdateProcessor.java:311)
>
> at org.apache.solr.update.processor.DistributedUpdateProcessor.
> processAdd(DistributedUpdateProcessor.java:661)
>
> at org.apache.solr.handler.loader.XMLLoader.
> processUpdate(XMLLoader.java:251)
>
> at org.apache.solr.handler.loader.XMLLoader.load(
> XMLLoader.java:178)
>
> at org.apache.solr.handler.UpdateRequestHandler$1.load(
> UpdateRequestHandler.java:95)
>
> at org.apache.solr.handler.ContentStreamHandlerBase.
> handleRequestBody(ContentStreamHandlerBase.java:70)
>
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
>
> at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:658)
>
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
>
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:223)
>
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:181)
>
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1652)
>
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:585)
>
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
> at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:577)
>
> at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:223)
>
> at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1127)
>
> at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:515)
>
> at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
> at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1061)
>
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:215)
>
> at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:110)
>
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
>
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:310)
>
> at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:257)
>
> at org.eclipse.jetty.io.AbstractConnection$2.run(
> AbstractConnection.java:540)
>
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:635)
>
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:555)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> 2016-10-08 19:34:05.557 ERROR (coreZkRegister-1-thread-15-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54129) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.c.ZkController Error getting leader from zk
>
> org.apache.solr.common.SolrException: No registered leader was found
> after waiting for 1560000ms , collection: xxx slice: 20161003_A
>
> at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(
> ZkStateReader.java:643)
>
> at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(
> ZkStateReader.java:610)
>
> at org.apache.solr.cloud.ZkController.getLeader(
> ZkController.java:1005)
>
> at org.apache.solr.cloud.ZkController.register(
> ZkController.java:940)
>
> at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(
> ZkController.java:234)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-10-08 19:34:06.101 ERROR (coreZkRegister-1-thread-14-processing-n:x.x.x.x:8987_solr
> x:xxx s:20161003_A c:xxx r:20161003_A54129) [c:xxx s:20161003_A
> r:20161003_A54129 x:xxx] o.a.s.c.ZkController Error getting leader from zk
>
> org.apache.solr.common.SolrException: No registered leader was found
> after waiting for 1560000ms , collection: xxx slice: 20161003_A
>
>
>