You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Modassar Ather <mo...@gmail.com> on 2015/04/16 12:57:50 UTC
Nno servers hosting shard.
Hi,
I have a setup of 5 node SolrCloud (Lucene/Solr version 5.1.0) without
replicas. When I am executing complex and large queries with wild-cards
after some time I am getting following exceptions.
The index size on each of the node is around 170GB and the memory is set to
-Xms20g -Xmx24g on each node.
Empty shard!
org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:214)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:184)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
There is no OutofMemory or any other major lead for me to understand what
had caused it. May be I am missing something. There are following other
exceptions:
SEVERE: null:org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while
waiting response from server at: http://<server>:8080/solr/collection
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:342)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARNING: listener throws error
org.apache.solr.common.SolrException:
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /configs/collection/params.json
at
org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:163)
at
org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:919)
at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2500)
at org.apache.solr.cloud.ZkController$4.run(ZkController.java:2366)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /configs/collection/params.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at
org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:294)
at
org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:291)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
at
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:291)
at
org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:153)
... 3 more
The Zookeeper session timeout is set to 30000. In the log file I can see
logs of the following pattern for all the queries I fired.
INFO: [collection] webapp=/solr path=/search_handler
params={sort=score+desc&start=0&q=(ft:search term)} status=0 QTime=<time>
If I am not wrong they are getting executed but somehow as the shard is
gone down which I can see in /clusterstate.json under the log, the search
is failing.
Kindly help me debug/solve this problem.
Regards,
Modassar
Re: Nno servers hosting shard.
Posted by Ugo Matrangolo <ug...@gmail.com>.
Hi,
sounds like you hit a Full GC. Check your GC.log.
Ugo
On 17 Apr 2015 08:24, "Modassar Ather" <mo...@gmail.com> wrote:
> Hi,
>
> Any suggestion will be really helpful. Kindly provide your inputs.
>
> Thanks,
> Modassar
>
> On Thu, Apr 16, 2015 at 4:27 PM, Modassar Ather <mo...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have a setup of 5 node SolrCloud (Lucene/Solr version 5.1.0) without
> > replicas. When I am executing complex and large queries with wild-cards
> > after some time I am getting following exceptions.
> > The index size on each of the node is around 170GB and the memory is set
> > to -Xms20g -Xmx24g on each node.
> >
> > Empty shard!
> > org.apache.solr.common.SolrException log
> > SEVERE: org.apache.solr.common.SolrException: no servers hosting shard:
> > at
> >
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:214)
> > at
> >
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:184)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > There is no OutofMemory or any other major lead for me to understand what
> > had caused it. May be I am missing something. There are following other
> > exceptions:
> >
> > SEVERE: null:org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: Timeout occurred while
> > waiting response from server at: http://<server>:8080/solr/collection
> > at
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:342)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> > at
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> > at
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> > at
> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > at
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> > at
> > org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
> > at
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> > at
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > WARNING: listener throws error
> > org.apache.solr.common.SolrException:
> > org.apache.zookeeper.KeeperException$SessionExpiredException:
> > KeeperErrorCode = Session expired for /configs/collection/params.json
> > at
> >
> org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:163)
> > at
> > org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:919)
> > at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2500)
> > at org.apache.solr.cloud.ZkController$4.run(ZkController.java:2366)
> > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> > KeeperErrorCode = Session expired for /configs/collection/params.json
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > at
> >
> org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:294)
> > at
> >
> org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:291)
> > at
> >
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
> > at
> > org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:291)
> > at
> >
> org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:153)
> > ... 3 more
> >
> > The Zookeeper session timeout is set to 30000. In the log file I can see
> > logs of the following pattern for all the queries I fired.
> > INFO: [collection] webapp=/solr path=/search_handler
> > params={sort=score+desc&start=0&q=(ft:search term)} status=0 QTime=<time>
> > If I am not wrong they are getting executed but somehow as the shard is
> > gone down which I can see in /clusterstate.json under the log, the search
> > is failing.
> >
> > Kindly help me debug/solve this problem.
> >
> > Regards,
> > Modassar
> >
>
Re: Nno servers hosting shard.
Posted by Modassar Ather <mo...@gmail.com>.
Hi,
Any suggestion will be really helpful. Kindly provide your inputs.
Thanks,
Modassar
On Thu, Apr 16, 2015 at 4:27 PM, Modassar Ather <mo...@gmail.com>
wrote:
> Hi,
>
> I have a setup of 5 node SolrCloud (Lucene/Solr version 5.1.0) without
> replicas. When I am executing complex and large queries with wild-cards
> after some time I am getting following exceptions.
> The index size on each of the node is around 170GB and the memory is set
> to -Xms20g -Xmx24g on each node.
>
> Empty shard!
> org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: no servers hosting shard:
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:214)
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:184)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> There is no OutofMemory or any other major lead for me to understand what
> had caused it. May be I am missing something. There are following other
> exceptions:
>
> SEVERE: null:org.apache.solr.common.SolrException:
> org.apache.solr.client.solrj.SolrServerException: Timeout occurred while
> waiting response from server at: http://<server>:8080/solr/collection
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:342)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> WARNING: listener throws error
> org.apache.solr.common.SolrException:
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /configs/collection/params.json
> at
> org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:163)
> at
> org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:919)
> at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2500)
> at org.apache.solr.cloud.ZkController$4.run(ZkController.java:2366)
> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /configs/collection/params.json
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> at
> org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:294)
> at
> org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:291)
> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
> at
> org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:291)
> at
> org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:153)
> ... 3 more
>
> The Zookeeper session timeout is set to 30000. In the log file I can see
> logs of the following pattern for all the queries I fired.
> INFO: [collection] webapp=/solr path=/search_handler
> params={sort=score+desc&start=0&q=(ft:search term)} status=0 QTime=<time>
> If I am not wrong they are getting executed but somehow as the shard is
> gone down which I can see in /clusterstate.json under the log, the search
> is failing.
>
> Kindly help me debug/solve this problem.
>
> Regards,
> Modassar
>