You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Vishal Vaibhav <de...@gmail.com> on 2020/06/16 03:04:08 UTC
Proxy Error when cluster went down
Hello all,
I am running on solr 8.5. For some reason entire cluster went down. When i
am trying to bring up the nodes,its not coming up. My health check is
on "/solr/rules/admin/system". I tried forcing a leader election but it
dint help.
so when i run the following commands. Why is it trying to proxy when those
nodes are down. Am i missing something?
*curl "http://localhost:8983/solr/rules/admin/system
<http://localhost:8983/solr/rules/admin/system>" * % Total % Received %
Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left
Speed
100 5582 100 5582 0 0 454k 0 --:--:-- --:--:-- --:--:--
495k
{
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","java.net.UnknownHostException"],
"msg":"Error trying to proxy request for url:
http://search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:8983/solr/rules/admin/system
",
"trace":"org.apache.solr.common.SolrException: Error trying to proxy
request for url:
http://search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:8983/solr/rules/admin/system\n\tat
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:735)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)\n\tat
java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.net.UnknownHostException:
search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
Name or service not known\n\tat
java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)\n\tat
java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown
Source)\n\tat
java.base/java.net.InetAddress.getAddressesFromNameService(Unknown
Source)\n\tat
java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown
Source)\n\tat java.base/java.net.InetAddress.getAllByName0(Unknown
Source)\n\tat java.base/java.net.InetAddress.getAllByName(Unknown
Source)\n\tat java.base/java.net.InetAddress.getAllByName(Unknown
Source)\n\tat
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)\n\tat
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)\n\tat
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)\n\tat
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394)\n\tat
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)\n\tat
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)\n\tat
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)\n\tat
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)\n\tat
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)\n\tat
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)\n\tat
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)\n\tat
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:707)\n\t...
37 more\n",
"code":500}}
Re: Proxy Error when cluster went down
Posted by Vishal Vaibhav <de...@gmail.com>.
So entire cluster was down. I m trying to bring node by node . I restarted
the first node . The solr comes up but add replicas command fails. And then
I tried to check clusterstatus api, it showed shard in active state, no
core as active i.e. all down and one live node which was the one that I
restarted. Also this all connects to one zookeeper ensemble of 3 nodes
On Tue, 16 Jun 2020 at 11:20 PM, Jörn Franke <jo...@gmail.com> wrote:
> Do you have another host with replica alive or are all replicas on the
> host that is down?
>
> Are all SolrCloud hosts in the same ZooKeeper?
>
> > Am 16.06.2020 um 19:29 schrieb Vishal Vaibhav <de...@gmail.com>:
> >
> > Hi thanks . My solr is running in kubernetes. So host name goes away
> with
> > the pod going
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.s
> > vc.cluster.local
> > So in my case the pod with this host has gone and also the hostname
> >
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
> > is no more there.. should not the solr cloud be aware of the fact that
> all
> > the replicas in that solr host is down and should not proxy the request
> to
> > that node..
> >
> >> On Tue, 16 Jun 2020 at 5:06 PM, Shawn Heisey <ap...@elyograg.org>
> wrote:
> >>
> >>> On 6/15/2020 9:04 PM, Vishal Vaibhav wrote:
> >>> I am running on solr 8.5. For some reason entire cluster went down.
> When
> >> i
> >>> am trying to bring up the nodes,its not coming up. My health check is
> >>> on "/solr/rules/admin/system". I tried forcing a leader election but it
> >>> dint help.
> >>> so when i run the following commands. Why is it trying to proxy when
> >> those
> >>> nodes are down. Am i missing something?
> >>
> >> <snip>
> >>
> >>> java.net.UnknownHostException:
> >>>
> >>
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
> >>
> >> It is trying to proxy because it's SolrCloud. SolrCloud has an internal
> >> load balancer that spreads queries across multiple replicas when
> >> possible. Your cluster must be aware of multiple servers where the
> >> "rules" collection can be queried.
> >>
> >> The underlying problem behind this error message is that the following
> >> hostname is being looked up, and it doesn't exist:
> >>
> >>
> >>
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
> >>
> >> This hostname is most likely coming from /etc/hosts on one of your
> >> systems when that system starts Solr and it registers with the cluster,
> >> and that /etc/hosts file is the ONLY place that the hostname exists, so
> >> when SolrCloud tries to forward the request to that server, it is
> failing.
> >>
> >> Thanks,
> >> Shawn
> >>
>
Re: Proxy Error when cluster went down
Posted by Jörn Franke <jo...@gmail.com>.
Do you have another host with replica alive or are all replicas on the host that is down?
Are all SolrCloud hosts in the same ZooKeeper?
> Am 16.06.2020 um 19:29 schrieb Vishal Vaibhav <de...@gmail.com>:
>
> Hi thanks . My solr is running in kubernetes. So host name goes away with
> the pod going search-rules-solr-v1-2.search-rules-solr-v1.search-digital.s
> vc.cluster.local
> So in my case the pod with this host has gone and also the hostname
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
> is no more there.. should not the solr cloud be aware of the fact that all
> the replicas in that solr host is down and should not proxy the request to
> that node..
>
>> On Tue, 16 Jun 2020 at 5:06 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>>
>>> On 6/15/2020 9:04 PM, Vishal Vaibhav wrote:
>>> I am running on solr 8.5. For some reason entire cluster went down. When
>> i
>>> am trying to bring up the nodes,its not coming up. My health check is
>>> on "/solr/rules/admin/system". I tried forcing a leader election but it
>>> dint help.
>>> so when i run the following commands. Why is it trying to proxy when
>> those
>>> nodes are down. Am i missing something?
>>
>> <snip>
>>
>>> java.net.UnknownHostException:
>>>
>> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
>>
>> It is trying to proxy because it's SolrCloud. SolrCloud has an internal
>> load balancer that spreads queries across multiple replicas when
>> possible. Your cluster must be aware of multiple servers where the
>> "rules" collection can be queried.
>>
>> The underlying problem behind this error message is that the following
>> hostname is being looked up, and it doesn't exist:
>>
>>
>> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
>>
>> This hostname is most likely coming from /etc/hosts on one of your
>> systems when that system starts Solr and it registers with the cluster,
>> and that /etc/hosts file is the ONLY place that the hostname exists, so
>> when SolrCloud tries to forward the request to that server, it is failing.
>>
>> Thanks,
>> Shawn
>>
Re: Proxy Error when cluster went down
Posted by Vishal Vaibhav <de...@gmail.com>.
Hi thanks . My solr is running in kubernetes. So host name goes away with
the pod going search-rules-solr-v1-2.search-rules-solr-v1.search-digital.s
vc.cluster.local
So in my case the pod with this host has gone and also the hostname
search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
is no more there.. should not the solr cloud be aware of the fact that all
the replicas in that solr host is down and should not proxy the request to
that node..
On Tue, 16 Jun 2020 at 5:06 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 6/15/2020 9:04 PM, Vishal Vaibhav wrote:
> > I am running on solr 8.5. For some reason entire cluster went down. When
> i
> > am trying to bring up the nodes,its not coming up. My health check is
> > on "/solr/rules/admin/system". I tried forcing a leader election but it
> > dint help.
> > so when i run the following commands. Why is it trying to proxy when
> those
> > nodes are down. Am i missing something?
>
> <snip>
>
> > java.net.UnknownHostException:
> >
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
>
> It is trying to proxy because it's SolrCloud. SolrCloud has an internal
> load balancer that spreads queries across multiple replicas when
> possible. Your cluster must be aware of multiple servers where the
> "rules" collection can be queried.
>
> The underlying problem behind this error message is that the following
> hostname is being looked up, and it doesn't exist:
>
>
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
>
> This hostname is most likely coming from /etc/hosts on one of your
> systems when that system starts Solr and it registers with the cluster,
> and that /etc/hosts file is the ONLY place that the hostname exists, so
> when SolrCloud tries to forward the request to that server, it is failing.
>
> Thanks,
> Shawn
>
Re: Proxy Error when cluster went down
Posted by Shawn Heisey <ap...@elyograg.org>.
On 6/15/2020 9:04 PM, Vishal Vaibhav wrote:
> I am running on solr 8.5. For some reason entire cluster went down. When i
> am trying to bring up the nodes,its not coming up. My health check is
> on "/solr/rules/admin/system". I tried forcing a leader election but it
> dint help.
> so when i run the following commands. Why is it trying to proxy when those
> nodes are down. Am i missing something?
<snip>
> java.net.UnknownHostException:
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
It is trying to proxy because it's SolrCloud. SolrCloud has an internal
load balancer that spreads queries across multiple replicas when
possible. Your cluster must be aware of multiple servers where the
"rules" collection can be queried.
The underlying problem behind this error message is that the following
hostname is being looked up, and it doesn't exist:
search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
This hostname is most likely coming from /etc/hosts on one of your
systems when that system starts Solr and it registers with the cluster,
and that /etc/hosts file is the ONLY place that the hostname exists, so
when SolrCloud tries to forward the request to that server, it is failing.
Thanks,
Shawn