Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader

["Bhalla, Rahat" <rb...@HealthPlan.com>]

Hi Solr Users,

I hope this email finds you all in the best of spirits and in a mood where you'd be willing to help a young developer (me :) ) with issues that I'm facing in regards with the Solr Cloud.

At my organization, we are running a Solr Cloud with 5 Nodes for Solr Instances with 13 collections spread across the 5 nodes and an ensemble of 3 zookeeper instances spread across three different nodes.

Over the last one week, our leader node seems to be going down every other day and while we restart the solr instances they still go down within the next 24 Hours or more.

We have tried rebooting the nodes that host the solr instances and that hasn't helped. We plan to clear out the zookeeper logs and data folders before the restart of the zookeeper instances.

As of now, I'm the only one supporting Solr in my organization and any insight from you could help me a great deal to fix the issue. I'm copying the Exception stack trace from this morning. Any recommendations that you might have will be great appreciated.

Below is a snapshot of one of the zoo nodes:

[cid:image001.png@01D2F43C.30CA38E0]

Exception Stacktrace

138127149 [OverseerCollectionConfigSetProcessor-98234161688412161-prod-solr-node01:9080_solr-n_0000000140] [ERROR] 2017-07-03 05:02:55 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01 (SolrException.java:log:148) - org.apache.solr.common.SolrException: Could not load collection from ZK: feedsOutBoundToExchange
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1047)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:610)
        at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:248)
        at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation$20.call(CollectionsHandler.java:674)
        at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:195)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /collections/feedsOutBoundToExchange/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1059)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1045)
        ... 33 more

Thanks,

Rahat Bhalla
HealthPlan Services
Phone: (813) 289-1000  EXT: 7002249
rbhalla@healthplan.com<ma...@healthplan.com>
www.healthplan.com<http://www.healthplan.com/>


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information and/or Protected Health Information (PHI) subject to protection under the law, including the Health Insurance Portability and Accountability Act of 1996, as amended (HIPAA). If you are not the intended recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error and that any use, disclosure, distribution, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error, please notify the sender immediately and destroy all copies of the original message.

Re: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader

[Ere Maijala <er...@helsinki.fi>]

 From the fact that someone has tried to access /etc/passwd file via 
your Solr (see all those WARN messages), it seems you have it exposed to 
the world, unless of course it's a security scanner you use internally. 
Internet is a hostile place, and the very first thing I would do is 
shield Solr from external traffic. Even if it's your own security 
scanning, I wouldn't do it until you have the system stable.

Doing the above you'll reduce noise in the logs and might be able to 
better identify the issue.

Losing the Zookeeper connection is typically a Java garbage collection 
issue. If GC causes too long pauses, the connection may time out. So I 
would recommend you start by reading 
https://wiki.apache.org/solr/SolrPerformanceProblems and 
https://wiki.apache.org/solr/ShawnHeisey. Also make sure that 
Zookeeper's Java settings are good.

--Ere

Bhalla, Rahat kirjoitti 5.7.2017 klo 11.05:
> Hi
> 
> I’m not sure if any of you have had a chance to see this email yet.
> 
> We had a reoccurrence of the Issue Today, and I’m attaching the Logs 
> from today as well inline below.
> 
> Please let me know if any of you have seen this issue before as this 
> would really help me to get to the root of the problem to fix it. I’m a 
> little lost here and not entirely sure what to do.
> 
> Thanks,
> 
> Rahat Bhalla
> 
> 8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20 
> (HttpParser.java:parseNext:1391) - parse exception: 
> java.lang.IllegalArgumentException: No Authority for 
> HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}
> 
> java.lang.IllegalArgumentException: No Authority
> 
>          at 
> org.eclipse.jetty.http.HostPortHttpField.<init>(HostPortHttpField.java:43)
> 
>          at 
> org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
> 
>          at 
> org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
> 
>          at 
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
> 
>          at 
> org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
> 
>          at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
> 
>          at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> 
>          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> 
>          at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> 8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for 
> HttpChannelOverHttp@127bbbb6{r=16,c=false,a=IDLE,uri=/../../../../../../../../../../etc/passwd}
> 
> 8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 No Host for 
> HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}
> 
> 8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for 
> HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd}
> 
> 8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22 
> (URIUtil.java:decodePath:348) - 
> /solr/admin/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd 
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! 
> byte C0 in state 0
> 
> 8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22 
> (URIUtil.java:decodePath:348) - 
> /solr/admin/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd 
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! 
> byte 80 in state 4
> 
> 8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22 
> (URIUtil.java:decodePath:348) - 
> /solr/admin/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd 
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! 
> byte 80 in state 6
> 
> 8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22 
> (URIUtil.java:decodePath:348) - 
> /solr/admin/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/etc/passwd 
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! 
> byte F8 in state 0
> 
> 8706641 [qtp778720569-27] [ WARN] 2017-07-04 01:40:31 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version 
> for HttpChannelOverHttp@7fcd594a{r=54,c=false,a=IDLE,uri=null}
> 
> 8707033 [qtp778720569-20] [ WARN] 2017-07-04 01:40:31 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version 
> for HttpChannelOverHttp@66740d77{r=54,c=false,a=IDLE,uri=null}
> 
> 8719390 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 
> (HttpParser.java:<init>:1740) - Illegal character 0xA in 
> state=HEADER_IN_NAME for buffer 
> HeapByteBuffer@41ed5e30[p=220,l=273,c=8192,r=53]={POST 
> /wp-content/...name="qqfile"\r\n<<<filename="QualysU...ys-Scan: 
> VM\r\n\r\n>>>\n\r\nq=*%3A*&df=tex...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
> 
> 8719391 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Illegal 
> character 0xA for HttpChannelOverHttp@235b51f6{r=2,c=false,a=IDLE,uri=null}
> 
> 8949147 [qtp778720569-27] [ WARN] 2017-07-04 01:44:33 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not 
> supported for HttpChannelOverHttp@62d6261e{r=0,c=false,a=IDLE,uri=null}
> 
> 8949149 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not 
> supported for HttpChannelOverHttp@33f62c0{r=0,c=false,a=IDLE,uri=null}
> 
> 8949163 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not 
> supported for HttpChannelOverHttp@6d58f7d7{r=0,c=false,a=IDLE,uri=null}
> 
> 67338023 
> [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] 
> [ERROR] 2017-07-04 05:57:42 (SolrException.java:log:159) - 
> :org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
> 
>          at 
> org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)
> 
>          at 
> org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:279)
> 
>          at 
> org.apache.solr.cloud.DistributedQueue.peekElements(DistributedQueue.java:328)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:236)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:226)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> 67365997 
> [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] 
> [ERROR] 2017-07-04 05:58:10 (OverseerTaskProcessor.java:amILeader:392) -
> 
> org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> 67381954 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 
> (SolrException.java:log:148) - java.io.IOException: Error opening 
> /configs/feedsInBoundCarrier/db-data-config.xml
> 
>          at 
> org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
> 
>          at 
> org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
> 
>          at 
> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
> 
>          at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> 
>          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
> 
>          at 
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
> 
>          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> 
>          at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> 
>          at org.eclipse.jetty.server.Server.handle(Server.java:518)
> 
>          at 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> 
>          at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> 
>          at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> 
>          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> 
>          at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for 
> /configs/feedsInBoundCarrier/db-data-config.xml
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
> 
>          at 
> org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
> 
>          at 
> org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
> 
>          ... 32 more
> 
> 67381955 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 
> (SolrException.java:log:159) - null:java.io.IOException: Error opening 
> /configs/feedsInBoundCarrier/db-data-config.xml
> 
>          at 
> org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
> 
>          at 
> org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
> 
>          at 
> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
> 
>          at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> 
>          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
> 
>          at 
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
> 
>          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> 
>          at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> 
>          at org.eclipse.jetty.server.Server.handle(Server.java:518)
> 
>          at 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> 
>          at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> 
>          at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> 
>          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> 
>          at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for 
> /configs/feedsInBoundCarrier/db-data-config.xml
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
> 
>          at 
> org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
> 
>          at 
> org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
> 
>          ... 32 more
> 
> 67393217 
> [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] 
> [ERROR] 2017-07-04 05:58:37 (OverseerTaskProcessor.java:amILeader:392) -
> 
> org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> 67420293 
> [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] 
> [ERROR] 2017-07-04 05:59:04 (OverseerTaskProcessor.java:amILeader:392) -
> 
> org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> 67437164 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 
> (SolrException.java:log:148) - java.io.IOException: Error opening 
> /configs/txnInBoundNonEx/db-data-config.xml
> 
>          at 
> org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
> 
>          at 
> org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
> 
>          at 
> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
> 
>          at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> 
>          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
> 
>          at 
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
> 
>          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> 
>          at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> 
>          at org.eclipse.jetty.server.Server.handle(Server.java:518)
> 
>          at 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> 
>          at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> 
>          at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> 
>          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> 
>          at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for 
> /configs/txnInBoundNonEx/db-data-config.xml
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
> 
>          at 
> org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
> 
>          at 
> org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
> 
>          ... 32 more
> 
> 67437165 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 
> (SolrException.java:log:159) - null:java.io.IOException: Error opening 
> /configs/txnInBoundNonEx/db-data-config.xml
> 
>          at 
> org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
> 
>          at 
> org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
> 
>          at 
> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
> 
>          at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> 
>          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
> 
>          at 
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
> 
>          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> 
>          at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> 
>          at org.eclipse.jetty.server.Server.handle(Server.java:518)
> 
>          at 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> 
>          at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> 
>          at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> 
>          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> 
>          at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for 
> /configs/txnInBoundNonEx/db-data-config.xml
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
> 
>          at 
> org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
> 
>          at 
> org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
> 
>          ... 32 more
> 
> *From:* Bhalla, Rahat
> *Sent:* Monday, July 03, 2017 8:38 PM
> *To:* solr-user@lucene.apache.org
> *Subject:* Solr Prod Issue | KeeperErrorCode = ConnectionLoss for 
> /overseer_elect/leader
> *Importance:* High
> 
> Hi Solr Users,
> 
> I hope this email finds you all in the best of spirits and in a mood 
> where you’d be willing to help a young developer (me J ) with issues 
> that I’m facing in regards with the Solr Cloud.
> 
> At my organization, we are running a Solr Cloud with 5 Nodes for Solr 
> Instances with 13 collections spread across the 5 nodes and an ensemble 
> of 3 zookeeper instances spread across three different nodes.
> 
> Over the last one week, our leader node seems to be going down every 
> other day and while we restart the solr instances they still go down 
> within the next 24 Hours or more.
> 
> We have tried rebooting the nodes that host the solr instances and that 
> hasn’t helped. We plan to clear out the zookeeper logs and data folders 
> before the restart of the zookeeper instances.
> 
> As of now, I’m the only one supporting Solr in my organization and any 
> insight from you could help me a great deal to fix the issue. I’m 
> copying the Exception stack trace from this morning. Any recommendations 
> that you might have will be great appreciated.
> 
> *Below is a snapshot of one of the zoo nodes:*
> 
> *Exception Stacktrace*
> 
> 138127149 
> [OverseerCollectionConfigSetProcessor-98234161688412161-prod-solr-node01:9080_solr-n_0000000140] 
> [ERROR] 2017-07-03 05:02:55 (OverseerTaskProcessor.java:amILeader:392) -
> 
> org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
> 
>          at 
> org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> 138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01 
> (SolrException.java:log:148) - org.apache.solr.common.SolrException: 
> Could not load collection from ZK: feedsOutBoundToExchange
> 
>          at 
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1047)
> 
>          at 
> org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:610)
> 
>          at 
> org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:248)
> 
>          at 
> org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation$20.call(CollectionsHandler.java:674)
> 
>          at 
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:195)
> 
>          at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> 
>          at 
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
> 
>          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> 
>          at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> 
>          at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> 
>          at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> 
>          at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> 
>          at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> 
>          at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> 
>          at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> 
>          at org.eclipse.jetty.server.Server.handle(Server.java:518)
> 
>          at 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> 
>          at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> 
>          at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> 
>          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> 
>          at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> 
>          at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> 
>          at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> 
>          at java.lang.Thread.run(Unknown Source)
> 
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
> KeeperErrorCode = ConnectionLoss for 
> /collections/feedsOutBoundToExchange/state.json
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> 
>          at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 
>          at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> 
>          at 
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
> 
>          at 
> org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1059)
> 
>          at 
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1045)
> 
>          ... 33 more
> 
> *Thanks,*
> 
> **
> 
> *Rahat Bhalla*
> 
> *HealthPlan/Services/*
> Phone: (813) 289-1000  EXT: 7002249**
> 
> rbhalla@healthplan.com <ma...@healthplan.com>
> 
> _www.healthplan.com <http://www.healthplan.com/> _
> 
> 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> CONFIDENTIALITY NOTICE: This email message, including any attachments, 
> is for the sole use of the intended recipient(s) and may contain 
> confidential and privileged information and/or Protected Health 
> Information (PHI) subject to protection under the law, including the 
> Health Insurance Portability and Accountability Act of 1996, as amended 
> (HIPAA). If you are not the intended recipient or the person responsible 
> for delivering the email to the intended recipient, be advised that you 
> have received this email in error and that any use, disclosure, 
> distribution, forwarding, printing, or copying of this email is strictly 
> prohibited. If you have received this email in error, please notify the 
> sender immediately and destroy all copies of the original message.

-- 
Ere Maijala
Kansalliskirjasto / The National Library of Finland

Re: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader

[Susheel Kumar <su...@gmail.com>]

PLease send all your attachments via dropbox etc.  Do not send them via
attachment as they will be deleted automatically..

On Wed, Jul 5, 2017 at 6:23 PM, Bhalla, Rahat <rb...@healthplan.com>
wrote:

> Hi Ere/Other Users,
>
>
>
> *@Ere:* Thank you for the response this morning please find more detailed
> logs from this afternoon.
>
>
>
> We are looking into some of the posts that you suggested but they don’t
> seem to be matching completely as per our current analysis. We are still
> working on analyzing the Heap Dumps on how GC is working. We are also
> looking into why packets are being dropped on the zookeeper nodes.
>
>
>
> What we are now seeing are three other things:
>
>
>
> [ERROR] 2017-07-05 01:57:51 (SolrException.java:log:159) - :
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work – *Only
> Once*
>
>
>
> [ERROR] 2017-07-05 01:58:19 (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader – *Multiple
> times for all collections which have shards on that Solr Node.*
>
>
>
> 34811545 [qtp778720569-21] [ERROR] 2017-07-05 02:01:08 (SolrException.java:log:148)
> - java.io.IOException: Error opening /configs/feedsInBoundNonEx/db-
> data-config.xml – *For all collections which have shards on that node.*
>
>
>
> *If any of you have seen this error before and have any suggestions.
> Please let me know.*
>
>
>
> Thanks,
>
> Rahat Bhalla
>
>
> Re: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for
> /overseer_elect/leader
> <http://www.mail-archive.com/solr-user@lucene.apache.org/msg131843.html>
>
> 2017-07-05
> <http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=date:20170705&o=newest&f=1>
>  Thread
> <http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=subject:%22Solr+Prod+Issue+%5C%7C+KeeperErrorCode+%3D+ConnectionLoss+for+%5C%2Foverseer_elect%5C%2Fleader%22&o=newest&f=1>
>  *Ere* *Maijala*
> <http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=from:%22Ere+Maijala%22&o=newest&f=1>
>
>
>
> From the fact that someone has tried to access /etc/passwd file via
>
> your Solr (see all those WARN messages), it seems you have it exposed to
>
> the world, unless of course it's a security scanner you use internally.
>
> Internet is a hostile place, and the very first thing I would do is
>
> shield Solr from external traffic. Even if it's your own security
>
> scanning, I wouldn't do it until you have the system stable.
>
>
>
>
>
> Doing the above you'll reduce noise in the logs and might be able to
>
> better identify the issue.
>
>
>
>
>
> Losing the Zookeeper connection is typically a Java garbage collection
>
> issue. If GC causes too long pauses, the connection may time out. So I
>
> would recommend you start by reading
>
> https://wiki.apache.org/solr/SolrPerformanceProblems and
>
> https://wiki.apache.org/solr/ShawnHeisey. Also make sure that
>
> Zookeeper's Java settings are good.
>
>
>
>
>
> --Ere
>
>
>
>
>
> *From:* Bhalla, Rahat
> *Sent:* Wednesday, July 05, 2017 4:05 AM
> *To:* 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> *Subject:* RE: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for
> /overseer_elect/leader
> *Importance:* High
>
>
>
> Hi
>
>
>
> I’m not sure if any of you have had a chance to see this email yet.
>
>
>
> We had a reoccurrence of the Issue Today, and I’m attaching the Logs from
> today as well inline below.
>
>
>
> Please let me know if any of you have seen this issue before as this would
> really help me to get to the root of the problem to fix it. I’m a little
> lost here and not entirely sure what to do.
>
>
>
> Thanks,
>
> Rahat Bhalla
>
>
>
> 8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20
> (HttpParser.java:parseNext:1391) - parse exception: java.lang.IllegalArgumentException:
> No Authority for HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}
>
> java.lang.IllegalArgumentException: No Authority
>
>         at org.eclipse.jetty.http.HostPortHttpField.<init>(
> HostPortHttpField.java:43)
>
>         at org.eclipse.jetty.http.HttpParser.parsedHeader(
> HttpParser.java:877)
>
>         at org.eclipse.jetty.http.HttpParser.parseHeaders(
> HttpParser.java:1050)
>
>         at org.eclipse.jetty.http.HttpParser.parseNext(
> HttpParser.java:1266)
>
>         at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(
> HttpConnection.java:344)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:227)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for
> HttpChannelOverHttp@127bbbb6{r=16,c=false,a=IDLE,uri=/../..
> /../../../../../../../../etc/passwd}
>
> 8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 No Host for
> HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}
>
> 8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for
> HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-
> solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/
> %2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%
> 2e%2e/%2e%2e/etc/passwd}
> <HttpChannelOverHttp@19a624ec%7br=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd%7d>
>
> 8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22
> (URIUtil.java:decodePath:348) - /solr/admin/%c0%ae%c0%ae/%c0%
> ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%
> ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%
> ae%c0%ae/%c0%ae%c0%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception:
> Not valid UTF8! byte C0 in state 0
>
> 8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22
> (URIUtil.java:decodePath:348) - /solr/admin/%e0%80%ae%e0%80%
> ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%
> ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%
> ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%
> ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> byte 80 in state 4
>
> 8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22
> (URIUtil.java:decodePath:348) - /solr/admin/%f0%80%80%ae%f0%
> 80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%
> ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%
> 80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%
> ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%
> 80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> byte 80 in state 6
>
> 8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22
> (URIUtil.java:decodePath:348) - /solr/admin/%f8%80%80%80%ae%
> f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%
> 80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%
> 80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/
> %f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%
> 80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%
> 80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/etc/passwd
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> byte F8 in state 0
>
> 8706641 [qtp778720569-27] [ WARN] 2017-07-04 01:40:31
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version
> for HttpChannelOverHttp@7fcd594a{r=54,c=false,a=IDLE,uri=null}
>
> 8707033 [qtp778720569-20] [ WARN] 2017-07-04 01:40:31
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version
> for HttpChannelOverHttp@66740d77{r=54,c=false,a=IDLE,uri=null}
>
> 8719390 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44
> (HttpParser.java:<init>:1740) - Illegal character 0xA in
> state=HEADER_IN_NAME for buffer HeapByteBuffer@41ed5e30[p=220,l=273,c=8192,r=53]={POST
> /wp-content/...name="qqfile"\r\n<<<filename="QualysU...ys-Scan:
> VM\r\n\r\n>>>\n\r\nq=*%3A*&df=tex...\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
>
> 8719391 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Illegal character
> 0xA for HttpChannelOverHttp@235b51f6{r=2,c=false,a=IDLE,uri=null}
>
> 8949147 [qtp778720569-27] [ WARN] 2017-07-04 01:44:33
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not
> supported for HttpChannelOverHttp@62d6261e{r=0,c=false,a=IDLE,uri=null}
>
> 8949149 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not
> supported for HttpChannelOverHttp@33f62c0{r=0,c=false,a=IDLE,uri=null}
>
> 8949163 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not
> supported for HttpChannelOverHttp@6d58f7d7{r=0,c=false,a=IDLE,uri=null}
>
> 67338023 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-
> solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:57:42
> (SolrException.java:log:159) - :org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
>
>         at org.apache.solr.common.cloud.SolrZkClient$6.execute(
> SolrZkClient.java:331)
>
>         at org.apache.solr.common.cloud.SolrZkClient$6.execute(
> SolrZkClient.java:328)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getChildren(
> SolrZkClient.java:328)
>
>         at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(
> DistributedQueue.java:299)
>
>         at org.apache.solr.cloud.DistributedQueue.firstChild(
> DistributedQueue.java:279)
>
>         at org.apache.solr.cloud.DistributedQueue.peekElements(
> DistributedQueue.java:328)
>
>         at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(
> OverseerTaskQueue.java:236)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:226)
>
>         at java.lang.Thread.run(Unknown Source)
>
>
>
> 67365997 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-
> solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:58:10
> (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(
> OverseerTaskProcessor.java:384)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:191)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 67381954 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26
> (SolrException.java:log:148) - java.io.IOException: Error opening
> /configs/feedsInBoundCarrier/db-data-config.xml
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:111)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> maybeReloadConfiguration(DataImporter.java:125)
>
>         at org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataImportHandler.java:176)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>
>         at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:657)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:464)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /configs/feedsInBoundCarrier/
> db-data-config.xml
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:314)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:311)
>
>         at org.apache.solr.cloud.ZkController.pathExists(
> ZkController.java:796)
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:89)
>
>         ... 32 more
>
>
>
> 67381955 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26
> (SolrException.java:log:159) - null:java.io.IOException: Error opening
> /configs/feedsInBoundCarrier/db-data-config.xml
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:111)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> maybeReloadConfiguration(DataImporter.java:125)
>
>         at org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataImportHandler.java:176)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>
>         at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:657)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:464)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /configs/feedsInBoundCarrier/
> db-data-config.xml
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:314)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:311)
>
>         at org.apache.solr.cloud.ZkController.pathExists(
> ZkController.java:796)
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:89)
>
>         ... 32 more
>
>
>
> 67393217 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-
> solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:58:37
> (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(
> OverseerTaskProcessor.java:384)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:191)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 67420293 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-
> solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:59:04
> (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(
> OverseerTaskProcessor.java:384)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:191)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 67437164 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21
> (SolrException.java:log:148) - java.io.IOException: Error opening
> /configs/txnInBoundNonEx/db-data-config.xml
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:111)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> maybeReloadConfiguration(DataImporter.java:125)
>
>         at org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataImportHandler.java:176)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>
>         at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:657)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:464)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /configs/txnInBoundNonEx/db-
> data-config.xml
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:314)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:311)
>
>         at org.apache.solr.cloud.ZkController.pathExists(
> ZkController.java:796)
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:89)
>
>         ... 32 more
>
>
>
> 67437165 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21
> (SolrException.java:log:159) - null:java.io.IOException: Error opening
> /configs/txnInBoundNonEx/db-data-config.xml
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:111)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> maybeReloadConfiguration(DataImporter.java:125)
>
>         at org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataImportHandler.java:176)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>
>         at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:657)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:464)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /configs/txnInBoundNonEx/db-
> data-config.xml
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:314)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:311)
>
>         at org.apache.solr.cloud.ZkController.pathExists(
> ZkController.java:796)
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:89)
>
>         ... 32 more
>
>
>
>
>
>
>
> *From:* Bhalla, Rahat
> *Sent:* Monday, July 03, 2017 8:38 PM
> *To:* solr-user@lucene.apache.org
> *Subject:* Solr Prod Issue | KeeperErrorCode = ConnectionLoss for
> /overseer_elect/leader
> *Importance:* High
>
>
>
>
>
> Hi Solr Users,
>
>
>
> I hope this email finds you all in the best of spirits and in a mood where
> you’d be willing to help a young developer (me J ) with issues that I’m
> facing in regards with the Solr Cloud.
>
>
>
> At my organization, we are running a Solr Cloud with 5 Nodes for Solr
> Instances with 13 collections spread across the 5 nodes and an ensemble of
> 3 zookeeper instances spread across three different nodes.
>
>
>
> Over the last one week, our leader node seems to be going down every other
> day and while we restart the solr instances they still go down within the
> next 24 Hours or more.
>
>
>
> We have tried rebooting the nodes that host the solr instances and that
> hasn’t helped. We plan to clear out the zookeeper logs and data folders
> before the restart of the zookeeper instances.
>
>
>
> As of now, I’m the only one supporting Solr in my organization and any
> insight from you could help me a great deal to fix the issue. I’m copying
> the Exception stack trace from this morning. Any recommendations that you
> might have will be great appreciated.
>
>
>
> *Below is a snapshot of one of the zoo nodes:*
>
>
>
>
>
> *Exception Stacktrace*
>
>
>
> 138127149 [OverseerCollectionConfigSetProcessor-98234161688412161-prod-
> solr-node01:9080_solr-n_0000000140] [ERROR] 2017-07-03 05:02:55
> (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(
> OverseerTaskProcessor.java:384)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:191)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01
> (SolrException.java:log:148) - org.apache.solr.common.SolrException:
> Could not load collection from ZK: feedsOutBoundToExchange
>
>         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:1047)
>
>         at org.apache.solr.common.cloud.ZkStateReader$
> LazyCollectionRef.get(ZkStateReader.java:610)
>
>         at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(
> ClusterState.java:248)
>
>         at org.apache.solr.handler.admin.CollectionsHandler$
> CollectionOperation$20.call(CollectionsHandler.java:674)
>
>         at org.apache.solr.handler.admin.CollectionsHandler.
> handleRequestBody(CollectionsHandler.java:195)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
> HttpSolrCall.java:663)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:445)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /collections/
> feedsOutBoundToExchange/state.json
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkStateReader.
> fetchCollectionState(ZkStateReader.java:1059)
>
>         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:1045)
>
>         ... 33 more
>
>
>
> *Thanks,*
>
>
>
> *Rahat Bhalla*
>
> *HealthPlan Services*
> Phone: (813) 289-1000  EXT: 7002249 <(813)%20289-1000>
>
> rbhalla@healthplan.com
>
> *www.healthplan.com <http://www.healthplan.com/> *
>
>
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> CONFIDENTIALITY NOTICE: This email message, including any attachments, is
> for the sole use of the intended recipient(s) and may contain confidential
> and privileged information and/or Protected Health Information (PHI)
> subject to protection under the law, including the Health Insurance
> Portability and Accountability Act of 1996, as amended (HIPAA). If you are
> not the intended recipient or the person responsible for delivering the
> email to the intended recipient, be advised that you have received this
> email in error and that any use, disclosure, distribution, forwarding,
> printing, or copying of this email is strictly prohibited. If you have
> received this email in error, please notify the sender immediately and
> destroy all copies of the original message.
>

RE: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader

["Bhalla, Rahat" <rb...@HealthPlan.com>]

Hi Ere/Other Users,

@Ere: Thank you for the response this morning please find more detailed logs from this afternoon.

We are looking into some of the posts that you suggested but they don't seem to be matching completely as per our current analysis. We are still working on analyzing the Heap Dumps on how GC is working. We are also looking into why packets are being dropped on the zookeeper nodes.

What we are now seeing are three other things:

[ERROR] 2017-07-05 01:57:51 (SolrException.java:log:159) - :org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work - Only Once

[ERROR] 2017-07-05 01:58:19 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader - Multiple times for all collections which have shards on that Solr Node.

34811545 [qtp778720569-21] [ERROR] 2017-07-05 02:01:08 (SolrException.java:log:148) - java.io.IOException: Error opening /configs/feedsInBoundNonEx/db-data-config.xml - For all collections which have shards on that node.

If any of you have seen this error before and have any suggestions. Please let me know.

Thanks,
Rahat Bhalla

Re: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader<http://www.mail-archive.com/solr-user@lucene.apache.org/msg131843.html>
2017-07-05<http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=date:20170705&o=newest&f=1> Thread<http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=subject:%22Solr+Prod+Issue+%5C%7C+KeeperErrorCode+%3D+ConnectionLoss+for+%5C%2Foverseer_elect%5C%2Fleader%22&o=newest&f=1> Ere Maijala<http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=from:%22Ere+Maijala%22&o=newest&f=1>

From the fact that someone has tried to access /etc/passwd file via
your Solr (see all those WARN messages), it seems you have it exposed to
the world, unless of course it's a security scanner you use internally.
Internet is a hostile place, and the very first thing I would do is
shield Solr from external traffic. Even if it's your own security
scanning, I wouldn't do it until you have the system stable.


Doing the above you'll reduce noise in the logs and might be able to
better identify the issue.


Losing the Zookeeper connection is typically a Java garbage collection
issue. If GC causes too long pauses, the connection may time out. So I
would recommend you start by reading
https://wiki.apache.org/solr/SolrPerformanceProblems and
https://wiki.apache.org/solr/ShawnHeisey. Also make sure that
Zookeeper's Java settings are good.


--Ere


From: Bhalla, Rahat
Sent: Wednesday, July 05, 2017 4:05 AM
To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
Subject: RE: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
Importance: High

Hi

I'm not sure if any of you have had a chance to see this email yet.

We had a reoccurrence of the Issue Today, and I'm attaching the Logs from today as well inline below.

Please let me know if any of you have seen this issue before as this would really help me to get to the root of the problem to fix it. I'm a little lost here and not entirely sure what to do.

Thanks,
Rahat Bhalla

8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20 (HttpParser.java:parseNext:1391) - parse exception: java.lang.IllegalArgumentException: No Authority for HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}
java.lang.IllegalArgumentException: No Authority
        at org.eclipse.jetty.http.HostPortHttpField.<init>(HostPortHttpField.java:43)
        at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
        at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
        at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for HttpChannelOverHttp@127bbbb6{r=16,c=false,a=IDLE,uri=/../../../../../../../../../../etc/passwd}<mailto:HttpChannelOverHttp@127bbbb6%7br=16,c=false,a=IDLE,uri=/../../../../../../../../../../etc/passwd%7d>
8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 No Host for HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}
8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd}<mailto:HttpChannelOverHttp@19a624ec%7br=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd%7d>
8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte C0 in state 0
8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte 80 in state 4
8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte 80 in state 6
8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte F8 in state 0
8706641 [qtp778720569-27] [ WARN] 2017-07-04 01:40:31 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version for HttpChannelOverHttp@7fcd594a{r=54,c=false,a=IDLE,uri=null}
8707033 [qtp778720569-20] [ WARN] 2017-07-04 01:40:31 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version for HttpChannelOverHttp@66740d77{r=54,c=false,a=IDLE,uri=null}
8719390 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 (HttpParser.java:<init>:1740) - Illegal character 0xA in state=HEADER_IN_NAME for buffer HeapByteBuffer@41ed5e30[p=220,l=273,c=8192,r=53]={POST /wp-content/...name="qqfile"\r\n<<<filename="QualysU...ys-Scan: VM\r\n\r\n>>>\n\r\nq=*%3A*&df=tex...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
8719391 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Illegal character 0xA for HttpChannelOverHttp@235b51f6{r=2,c=false,a=IDLE,uri=null}
8949147 [qtp778720569-27] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@62d6261e{r=0,c=false,a=IDLE,uri=null}
8949149 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@33f62c0{r=0,c=false,a=IDLE,uri=null}
8949163 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@6d58f7d7{r=0,c=false,a=IDLE,uri=null}
67338023 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:57:42 (SolrException.java:log:159) - :org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
        at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)
        at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:279)
        at org.apache.solr.cloud.DistributedQueue.peekElements(DistributedQueue.java:328)
        at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:236)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:226)
        at java.lang.Thread.run(Unknown Source)

67365997 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:58:10 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67381954 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 (SolrException.java:log:148) - java.io.IOException: Error opening /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67381955 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 (SolrException.java:log:159) - null:java.io.IOException: Error opening /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67393217 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:58:37 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67420293 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:59:04 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67437164 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 (SolrException.java:log:148) - java.io.IOException: Error opening /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67437165 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 (SolrException.java:log:159) - null:java.io.IOException: Error opening /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more



From: Bhalla, Rahat
Sent: Monday, July 03, 2017 8:38 PM
To: solr-user@lucene.apache.org<ma...@lucene.apache.org>
Subject: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
Importance: High


Hi Solr Users,

I hope this email finds you all in the best of spirits and in a mood where you'd be willing to help a young developer (me :) ) with issues that I'm facing in regards with the Solr Cloud.

At my organization, we are running a Solr Cloud with 5 Nodes for Solr Instances with 13 collections spread across the 5 nodes and an ensemble of 3 zookeeper instances spread across three different nodes.

Over the last one week, our leader node seems to be going down every other day and while we restart the solr instances they still go down within the next 24 Hours or more.

We have tried rebooting the nodes that host the solr instances and that hasn't helped. We plan to clear out the zookeeper logs and data folders before the restart of the zookeeper instances.

As of now, I'm the only one supporting Solr in my organization and any insight from you could help me a great deal to fix the issue. I'm copying the Exception stack trace from this morning. Any recommendations that you might have will be great appreciated.

Below is a snapshot of one of the zoo nodes:

[cid:image001.png@01D2F5BB.D79E7020]

Exception Stacktrace

138127149 [OverseerCollectionConfigSetProcessor-98234161688412161-prod-solr-node01:9080_solr-n_0000000140] [ERROR] 2017-07-03 05:02:55 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01 (SolrException.java:log:148) - org.apache.solr.common.SolrException: Could not load collection from ZK: feedsOutBoundToExchange
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1047)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:610)
        at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:248)
        at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation$20.call(CollectionsHandler.java:674)
        at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:195)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /collections/feedsOutBoundToExchange/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1059)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1045)
        ... 33 more

Thanks,

Rahat Bhalla
HealthPlan Services
Phone: (813) 289-1000  EXT: 7002249
rbhalla@healthplan.com<ma...@healthplan.com>
www.healthplan.com<http://www.healthplan.com/>


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information and/or Protected Health Information (PHI) subject to protection under the law, including the Health Insurance Portability and Accountability Act of 1996, as amended (HIPAA). If you are not the intended recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error and that any use, disclosure, distribution, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error, please notify the sender immediately and destroy all copies of the original message.

RE: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader

["Bhalla, Rahat" <rb...@HealthPlan.com>]

Hi

I'm not sure if any of you have had a chance to see this email yet.

We had a reoccurrence of the Issue Today, and I'm attaching the Logs from today as well inline below.

Please let me know if any of you have seen this issue before as this would really help me to get to the root of the problem to fix it. I'm a little lost here and not entirely sure what to do.

Thanks,
Rahat Bhalla

8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20 (HttpParser.java:parseNext:1391) - parse exception: java.lang.IllegalArgumentException: No Authority for HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}
java.lang.IllegalArgumentException: No Authority
        at org.eclipse.jetty.http.HostPortHttpField.<init>(HostPortHttpField.java:43)
        at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
        at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
        at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for HttpChannelOverHttp@127bbbb6{r=16,c=false,a=IDLE,uri=/../../../../../../../../../../etc/passwd}
8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 No Host for HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}
8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd}
8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte C0 in state 0
8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte 80 in state 4
8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte 80 in state 6
8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte F8 in state 0
8706641 [qtp778720569-27] [ WARN] 2017-07-04 01:40:31 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version for HttpChannelOverHttp@7fcd594a{r=54,c=false,a=IDLE,uri=null}
8707033 [qtp778720569-20] [ WARN] 2017-07-04 01:40:31 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version for HttpChannelOverHttp@66740d77{r=54,c=false,a=IDLE,uri=null}
8719390 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 (HttpParser.java:<init>:1740) - Illegal character 0xA in state=HEADER_IN_NAME for buffer HeapByteBuffer@41ed5e30[p=220,l=273,c=8192,r=53]={POST /wp-content/...name="qqfile"\r\n<<<filename="QualysU...ys-Scan: VM\r\n\r\n>>>\n\r\nq=*%3A*&df=tex...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
8719391 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Illegal character 0xA for HttpChannelOverHttp@235b51f6{r=2,c=false,a=IDLE,uri=null}
8949147 [qtp778720569-27] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@62d6261e{r=0,c=false,a=IDLE,uri=null}
8949149 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@33f62c0{r=0,c=false,a=IDLE,uri=null}
8949163 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@6d58f7d7{r=0,c=false,a=IDLE,uri=null}
67338023 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:57:42 (SolrException.java:log:159) - :org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
        at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)
        at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:279)
        at org.apache.solr.cloud.DistributedQueue.peekElements(DistributedQueue.java:328)
        at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:236)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:226)
        at java.lang.Thread.run(Unknown Source)

67365997 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:58:10 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67381954 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 (SolrException.java:log:148) - java.io.IOException: Error opening /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67381955 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 (SolrException.java:log:159) - null:java.io.IOException: Error opening /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67393217 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:58:37 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67420293 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:59:04 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67437164 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 (SolrException.java:log:148) - java.io.IOException: Error opening /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67437165 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 (SolrException.java:log:159) - null:java.io.IOException: Error opening /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more



From: Bhalla, Rahat
Sent: Monday, July 03, 2017 8:38 PM
To: solr-user@lucene.apache.org
Subject: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
Importance: High


Hi Solr Users,

I hope this email finds you all in the best of spirits and in a mood where you'd be willing to help a young developer (me :) ) with issues that I'm facing in regards with the Solr Cloud.

At my organization, we are running a Solr Cloud with 5 Nodes for Solr Instances with 13 collections spread across the 5 nodes and an ensemble of 3 zookeeper instances spread across three different nodes.

Over the last one week, our leader node seems to be going down every other day and while we restart the solr instances they still go down within the next 24 Hours or more.

We have tried rebooting the nodes that host the solr instances and that hasn't helped. We plan to clear out the zookeeper logs and data folders before the restart of the zookeeper instances.

As of now, I'm the only one supporting Solr in my organization and any insight from you could help me a great deal to fix the issue. I'm copying the Exception stack trace from this morning. Any recommendations that you might have will be great appreciated.

Below is a snapshot of one of the zoo nodes:

[cid:image001.png@01D2F543.E5447AD0]

Exception Stacktrace

138127149 [OverseerCollectionConfigSetProcessor-98234161688412161-prod-solr-node01:9080_solr-n_0000000140] [ERROR] 2017-07-03 05:02:55 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01 (SolrException.java:log:148) - org.apache.solr.common.SolrException: Could not load collection from ZK: feedsOutBoundToExchange
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1047)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:610)
        at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:248)
        at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation$20.call(CollectionsHandler.java:674)
        at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:195)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /collections/feedsOutBoundToExchange/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1059)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1045)
        ... 33 more

Thanks,

Rahat Bhalla
HealthPlan Services
Phone: (813) 289-1000  EXT: 7002249
rbhalla@healthplan.com<ma...@healthplan.com>
www.healthplan.com<http://www.healthplan.com/>


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information and/or Protected Health Information (PHI) subject to protection under the law, including the Health Insurance Portability and Accountability Act of 1996, as amended (HIPAA). If you are not the intended recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error and that any use, disclosure, distribution, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error, please notify the sender immediately and destroy all copies of the original message.