You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by elvis鱼人 <q4...@163.com> on 2015/12/25 12:08:58 UTC
( no servers hosting shard ) very strange
my environment is solrcloud 5.2.0, 3 shards, every shard has 2 replica,one
leader,one replica
[11-18 06:58:42.515][ERROR]
org.apache.solr.common.SolrException.log(SolrException.java:139) -
org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:211)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[11-18 06:58:42.515][INFO]
org.apache.solr.core.SolrCore.execute(SolrCore.java:2069) - [feed]
webapp=/solr path=/select params={q=uid:2647309&wt=javabin&version=2}
status=503 QTime=4
query to this node,not to solrcloud
--
View this message in context: http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: ( no servers hosting shard ) very strange
Posted by Binoy Dalal <bi...@gmail.com>.
Yes. I'm currently setting up an ensemble but in my case the GC pauses
occur on the machine that is not running the embedded zookeeper.
On Sun, 27 Dec 2015, 23:09 Upayavira <uv...@odoko.co.uk> wrote:
> Firstly, don't use an embedded Zookeeper for anything other than
> experimentation,
>
> Set up your own zookeeper ensemble, otherwise your one box doing GC will
> take down all of your solr instances.
>
> On Fri, Dec 25, 2015, at 01:24 PM, Binoy Dalal wrote:
> > Hi Elvis,
> > We're having a similar problem with one of our cloud setups.
> > I'm currently running Solr 4.10.4 with 2 shards and no replicas. One of
> > our
> > shards also keeps dropping off with the same error message. This happens
> > because the shard executes too many full GCs which completely freezes the
> > machine and causes zookeeper to think that it has dropped off. So I would
> > suggest that you analyze your gc logs and see if you're facing a similar
> > problem.
> > Moreover, I'm currently running the zookeeper that is embedded with solr.
> > Are you doing the same or do you have an external instance/ensemble
> > setup?
> > So far, I've got no solutions but I'll update this thread if I find
> > anything.
> >
> > On Fri, Dec 25, 2015 at 5:01 PM elvis鱼人 <q4...@163.com> wrote:
> >
> > > my environment is solrcloud 5.2.0, 3 shards, every shard has 2
> replica,one
> > > leader,one replica
> > >
> > >
> > > [11-18 06:58:42.515][ERROR]
> > > org.apache.solr.common.SolrException.log(SolrException.java:139) -
> > > org.apache.solr.common.SolrException: no servers hosting shard:
> > > at
> > >
> > >
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:211)
> > > at
> > >
> > >
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > at
> > >
> > >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
> > > at
> > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > [11-18 06:58:42.515][INFO]
> > > org.apache.solr.core.SolrCore.execute(SolrCore.java:2069) - [feed]
> > > webapp=/solr path=/select params={q=uid:2647309&wt=javabin&version=2}
> > > status=503 QTime=4
> > >
> > > query to this node,not to solrcloud
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> > >
> > --
> > Regards,
> > Binoy Dalal
>
--
Regards,
Binoy Dalal
Re: ( no servers hosting shard ) very strange
Posted by Upayavira <uv...@odoko.co.uk>.
Firstly, don't use an embedded Zookeeper for anything other than
experimentation,
Set up your own zookeeper ensemble, otherwise your one box doing GC will
take down all of your solr instances.
On Fri, Dec 25, 2015, at 01:24 PM, Binoy Dalal wrote:
> Hi Elvis,
> We're having a similar problem with one of our cloud setups.
> I'm currently running Solr 4.10.4 with 2 shards and no replicas. One of
> our
> shards also keeps dropping off with the same error message. This happens
> because the shard executes too many full GCs which completely freezes the
> machine and causes zookeeper to think that it has dropped off. So I would
> suggest that you analyze your gc logs and see if you're facing a similar
> problem.
> Moreover, I'm currently running the zookeeper that is embedded with solr.
> Are you doing the same or do you have an external instance/ensemble
> setup?
> So far, I've got no solutions but I'll update this thread if I find
> anything.
>
> On Fri, Dec 25, 2015 at 5:01 PM elvis鱼人 <q4...@163.com> wrote:
>
> > my environment is solrcloud 5.2.0, 3 shards, every shard has 2 replica,one
> > leader,one replica
> >
> >
> > [11-18 06:58:42.515][ERROR]
> > org.apache.solr.common.SolrException.log(SolrException.java:139) -
> > org.apache.solr.common.SolrException: no servers hosting shard:
> > at
> >
> > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:211)
> > at
> >
> > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
> > at
> >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > [11-18 06:58:42.515][INFO]
> > org.apache.solr.core.SolrCore.execute(SolrCore.java:2069) - [feed]
> > webapp=/solr path=/select params={q=uid:2647309&wt=javabin&version=2}
> > status=503 QTime=4
> >
> > query to this node,not to solrcloud
> >
> >
> >
> > --
> > View this message in context:
> > http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> --
> Regards,
> Binoy Dalal
Re: ( no servers hosting shard ) very strange
Posted by elvis鱼人 <q4...@163.com>.
yes ,just may be fixed
--
View this message in context: http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349p4247525.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: ( no servers hosting shard ) very strange
Posted by Binoy Dalal <bi...@gmail.com>.
Has changing the heap size fixed your GC problem?
As for the leader not being elected, I am not very sure about it but if
there was some issue then you'd see it in the solr logs as exceptions so
you should check that.
On Mon, 28 Dec 2015, 15:21 elvis鱼人 <q4...@163.com> wrote:
> yes, i saw massive full GC ,so i change java -Xms10g -Xmx10g
> and there is another problem,
> shard1:
> 192.168.100.210:7001-leader
> 192.168.100.211:7001-replica
>
> shard2:
> 192.168.100.211:7002:leader
> 192.168.100.212:7001:replica
>
> shard3:
> 192.168.100.210:7002:leader
> 192.168.100.212:7002:replica
> i shutdown 2 server,210:7001 and 210:7002, then i saw replica could't
> electing leader.i don't know why?
> and this appearance doesn't appear every time
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349p4247518.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Regards,
Binoy Dalal
Re: ( no servers hosting shard ) very strange
Posted by elvis鱼人 <q4...@163.com>.
yes, i saw massive full GC ,so i change java -Xms10g -Xmx10g
and there is another problem,
shard1:
192.168.100.210:7001-leader
192.168.100.211:7001-replica
shard2:
192.168.100.211:7002:leader
192.168.100.212:7001:replica
shard3:
192.168.100.210:7002:leader
192.168.100.212:7002:replica
i shutdown 2 server,210:7001 and 210:7002, then i saw replica could't
electing leader.i don't know why?
and this appearance doesn't appear every time
--
View this message in context: http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349p4247518.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: ( no servers hosting shard ) very strange
Posted by Binoy Dalal <bi...@gmail.com>.
Are you also seeing massive GC pauses?
Also have you recently changed any heap parameters?
On Mon, 28 Dec 2015, 10:53 elvis鱼人 <q4...@163.com> wrote:
> before this happen, every solr server use -Xms2g -Xmx2g,may be associated
> with it
> i use External ZooKeeper, and this zookeeper connect with dubbo,storm,
> Will not be affected??
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349p4247489.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Regards,
Binoy Dalal
Re: ( no servers hosting shard ) very strange
Posted by elvis鱼人 <q4...@163.com>.
before this happen, every solr server use -Xms2g -Xmx2g,may be associated
with it
i use External ZooKeeper, and this zookeeper connect with dubbo,storm,
Will not be affected??
--
View this message in context: http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349p4247489.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: ( no servers hosting shard ) very strange
Posted by Binoy Dalal <bi...@gmail.com>.
Hi Elvis,
We're having a similar problem with one of our cloud setups.
I'm currently running Solr 4.10.4 with 2 shards and no replicas. One of our
shards also keeps dropping off with the same error message. This happens
because the shard executes too many full GCs which completely freezes the
machine and causes zookeeper to think that it has dropped off. So I would
suggest that you analyze your gc logs and see if you're facing a similar
problem.
Moreover, I'm currently running the zookeeper that is embedded with solr.
Are you doing the same or do you have an external instance/ensemble setup?
So far, I've got no solutions but I'll update this thread if I find
anything.
On Fri, Dec 25, 2015 at 5:01 PM elvis鱼人 <q4...@163.com> wrote:
> my environment is solrcloud 5.2.0, 3 shards, every shard has 2 replica,one
> leader,one replica
>
>
> [11-18 06:58:42.515][ERROR]
> org.apache.solr.common.SolrException.log(SolrException.java:139) -
> org.apache.solr.common.SolrException: no servers hosting shard:
> at
>
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:211)
> at
>
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> [11-18 06:58:42.515][INFO]
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2069) - [feed]
> webapp=/solr path=/select params={q=uid:2647309&wt=javabin&version=2}
> status=503 QTime=4
>
> query to this node,not to solrcloud
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Regards,
Binoy Dalal