You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris Ulicny <cu...@iq.media> on 2018/10/17 12:32:59 UTC
Overseer could not get tags
Hi all,
Recently in a 7.4.0 test cluster, we ran into SOLR-12814
<https://issues.apache.org/jira/browse/SOLR-12814> which we fixed by
slightly increasing the request header size. However, there were some other
log messages along with the "URI size >8192" message which we thought were
related, but have not abated since increasing the header size. A full
shutdown of the solr processes and bringing them back up one at a time did
not solve the issue.
The overseer node seems to not be authenticating any of the requests to
/solr/admin/metrics on any node (including itself). Every minute, there are
two warning per node
10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not
get tags from node host1:port1_solr
10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not
get tags from node host1:port1_solr
10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
get tags from node host2:port2_solr
10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
get tags from node host2:port2_solr
10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
get tags from node host3:port3_solr
10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
get tags from node host3:port3_solr
There are two slightly different stack traces that appear with each pair:
https://pastebin.com/2Z1C5rXr
The warning message possibly comes from
solrj.impl.SolrClientNodeStateProvider.fetchMetrics which both of the
attempted requests call in their stack trace.
However, we already have a 7.4.0 production cluster running that also has
security enabled with similar replica density where we have not seen this
issue.
*Test:*
-- 10 collections (9 with 2 shards, 1 with 43 shards)
-- replication factor of 2 for all collections
-- 3 hosts with 40 or 41 replicas each
*Production:*
-- 9 collections with 14 shards
-- replication factor of 2 for all collections
-- 7 hosts with 36 replicas each
I've enabled TRACE logging in our test environment on most options related
to metrics and authentication. So far the only new message I've gotten is
the challenge from the target server for the necessary credentials right
before the warning and stack trace.
2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
o.a.h.i.a.HttpAuthenticator Authentication required
2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
o.a.h.i.a.HttpAuthenticator host3:port3 requested authentication
I suspect the creation and balancing of the large collection on test might
have something to do with it since the problem started happening after
that.
Are there any other specific log settings I should turn on that might
produce some useful information?
Thanks,
Chris
Re: Overseer could not get tags
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
I am seeing the same issue in Solr 7.7.0
Apparently there is already a JIRA at
https://issues.apache.org/jira/browse/SOLR-12860.
Regards,
Edwin
On Wed, 27 Feb 2019 at 08:03, dshih <ds...@tableau.com> wrote:
> Opened SOLR-13274
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Re: Overseer could not get tags
Posted by dshih <ds...@tableau.com>.
Opened SOLR-13274
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Overseer could not get tags
Posted by dshih <ds...@tableau.com>.
We are seeing the same issue running 7.4.0. Increasing the request and
response header size did not resolve the issue. Should we open a JIRA
ticket if one does not already exist?
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Overseer could not get tags
Posted by Chris Ulicny <cu...@iq.media>.
Vadim,
Thanks for the suggestion.
For the test case, of spinning up solr 7.5 locally with the security.json
upload, I went ahead and tried to change both the request and response
header sizes as you mentioned and restarting.
Unfortunately, it does not seem to fix the issue for me. The error that is
being displayed in the logs is still the 401 error when accessing
/solr/admin/metrics. To me, it seems that this internal request is just
ignoring the security altogether.
On Wed, Oct 31, 2018 at 4:54 PM Vadim Ivanov <
vadim.ivanov@spb.ntk-intourist.ru> wrote:
> Hi, Chris
> I had the same messages in solr log while testing 7.4 and 7.5
> The only remedy I've found - increasing header size:
> /opt/solr/server/etc/jetty.xml
> <Set name="requestHeaderSize"><Property
> name="solr.jetty.request.header.size" default="81920" /></Set>
> <Set name="responseHeaderSize"><Property
> name="solr.jetty.response.header.size" default="81920" /></Set>
> After solr restart - no more annoying messages
>
> > -----Original Message-----
> > From: Chris Ulicny [mailto:culicny@iq.media]
> > Sent: Wednesday, October 31, 2018 7:40 PM
> > To: solr-user
> > Subject: Re: Overseer could not get tags
> >
> > I've managed to replicate this issue with the 7.5.0 release as well by
> > starting up a single instance of solr in cloud mode (on windows) and
> > uploading the security.json file below to it.
> >
> > After a short while, the "could not get tags from node..." messages start
> > coming through every 60 seconds. The accompanying logged error and
> > expecting stacktrace are also included below.
> >
> > Is there a JIRA ticket for this issue (or a directly related one)? I
> > couldn't seem to find one.
> >
> > Thanks,
> > Chris
> >
> > *security.json:*
> > {
> > "authentication":{"blockUnknown":true,"class":"solr.BasicAuthPlugin",
> > "credentials":{
> > "solradmin":"...",
> > "solrreader":"...",
> > "solrwriter":"..."}
> > },
> > "authorization":{"class":"solr.RuleBasedAuthorizationPlugin",
> > "permissions":[
> > {"name":"read","role":"reader"},
> > {"name":"security-read","role":"reader"},
> > {"name":"schema-read","role":"reader"},
> > {"name":"config-read","role":"reader"},
> > {"name":"core-admin-read","role":"reader"},
> > {"name":"collection-admin-read","role":"reader"},
> > {"name":"update","role":"writer"},
> > {"name":"security-edit","role":"admin"},
> > {"name":"schema-edit","role":"admin"},
> > {"name":"config-edit","role":"admin"},
> > {"name":"core-admin-edit","role":"admin"},
> > {"name":"collection-admin-edit","role":"admin"},
> > {"name":"all","role":"admin"}],
> > "user-role":{
> > "solradmin":["reader","writer","admin"],
> > "solrreader":["reader"],
> > "solrwriter":["reader","writer"]}
> > }
> > }
> >
> > *StackTrace:*
> > 2018-10-31 16:20:01.994 WARN (MetricsHistoryHandler-12-thread-1) [ ]
> > o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node
> > ip:8080_solr
> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error
> > from server at http://ip:8080/solr: Expected mime type
> > application/octet-stream but got text/html. <html>
> > <head>
> > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> > <title>Error 401 require authentication</title>
> > </head>
> > <body><h2>HTTP ERROR 401</h2>
> > <p>Problem accessing /solr/admin/metrics. Reason:
> > <pre> require authentication</pre></p>
> > </body>
> > </html>
> >
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.ja
> > va:607)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.in
> > voke(SolrClientNodeStateProvider.java:342)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetri
> > cs(SolrClientNodeStateProvider.java:195)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitc
> > h.getRemoteInfo(SolrClientNodeStateProvider.java:241)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:7
> > 6)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(S
> > olrClientNodeStateProvider.java:139)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(S
> > olrClientNodeStateProvider.java:128)
> > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:58]
> > at
> >
> org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(Me
> > tricsHistoryHandler.java:498)
> > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:55]
> > at
> >
> org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHi
> > storyHandler.java:371)
> > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:55]
> > at
> > org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(Metrics
> > HistoryHandler.java:231)
> > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> > jimczi - 2018-09-18 13:07:55]
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > [?:1.8.0_121]
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > [?:1.8.0_121]
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acces
> > s$301(ScheduledThreadPoolExecutor.java:180)
> > [?:1.8.0_121]
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(S
> > cheduledThreadPoolExecutor.java:294)
> > [?:1.8.0_121]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> > 1142)
> > [?:1.8.0_121]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
> > :617)
> > [?:1.8.0_121]
> > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
> >
> >
> > On Wed, Oct 17, 2018 at 8:32 AM Chris Ulicny <cu...@iq.media> wrote:
> >
> > > Hi all,
> > >
> > > Recently in a 7.4.0 test cluster, we ran into SOLR-12814
> > > <https://issues.apache.org/jira/browse/SOLR-12814> which we fixed by
> > > slightly increasing the request header size. However, there were some
> other
> > > log messages along with the "URI size >8192" message which we thought
> > were
> > > related, but have not abated since increasing the header size. A full
> > > shutdown of the solr processes and bringing them back up one at a time
> did
> > > not solve the issue.
> > >
> > > The overseer node seems to not be authenticating any of the requests to
> > > /solr/admin/metrics on any node (including itself). Every minute,
> there are
> > > two warning per node
> > >
> > > 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could
> not
> > > get tags from node host1:port1_solr
> > > 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could
> not
> > > get tags from node host1:port1_solr
> > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could
> not
> > > get tags from node host2:port2_solr
> > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could
> not
> > > get tags from node host2:port2_solr
> > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could
> not
> > > get tags from node host3:port3_solr
> > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could
> not
> > > get tags from node host3:port3_solr
> > >
> > > There are two slightly different stack traces that appear with each
> pair:
> > > https://pastebin.com/2Z1C5rXr
> > >
> > > The warning message possibly comes from
> > > solrj.impl.SolrClientNodeStateProvider.fetchMetrics which both of the
> > > attempted requests call in their stack trace.
> > >
> > > However, we already have a 7.4.0 production cluster running that also
> has
> > > security enabled with similar replica density where we have not seen
> this
> > > issue.
> > >
> > > *Test:*
> > > -- 10 collections (9 with 2 shards, 1 with 43 shards)
> > > -- replication factor of 2 for all collections
> > > -- 3 hosts with 40 or 41 replicas each
> > >
> > > *Production:*
> > > -- 9 collections with 14 shards
> > > -- replication factor of 2 for all collections
> > > -- 7 hosts with 36 replicas each
> > >
> > > I've enabled TRACE logging in our test environment on most options
> related
> > > to metrics and authentication. So far the only new message I've gotten
> is
> > > the challenge from the target server for the necessary credentials
> right
> > > before the warning and stack trace.
> > >
> > > 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
> > > o.a.h.i.a.HttpAuthenticator Authentication required
> > > 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
> > > o.a.h.i.a.HttpAuthenticator host3:port3 requested authentication
> > >
> > > I suspect the creation and balancing of the large collection on test
> might
> > > have something to do with it since the problem started happening after
> > > that.
> > >
> > > Are there any other specific log settings I should turn on that might
> > > produce some useful information?
> > >
> > > Thanks,
> > > Chris
> > >
>
>
RE: Overseer could not get tags
Posted by Vadim Ivanov <va...@spb.ntk-intourist.ru>.
Hi, Chris
I had the same messages in solr log while testing 7.4 and 7.5
The only remedy I've found - increasing header size:
/opt/solr/server/etc/jetty.xml
<Set name="requestHeaderSize"><Property name="solr.jetty.request.header.size" default="81920" /></Set>
<Set name="responseHeaderSize"><Property name="solr.jetty.response.header.size" default="81920" /></Set>
After solr restart - no more annoying messages
> -----Original Message-----
> From: Chris Ulicny [mailto:culicny@iq.media]
> Sent: Wednesday, October 31, 2018 7:40 PM
> To: solr-user
> Subject: Re: Overseer could not get tags
>
> I've managed to replicate this issue with the 7.5.0 release as well by
> starting up a single instance of solr in cloud mode (on windows) and
> uploading the security.json file below to it.
>
> After a short while, the "could not get tags from node..." messages start
> coming through every 60 seconds. The accompanying logged error and
> expecting stacktrace are also included below.
>
> Is there a JIRA ticket for this issue (or a directly related one)? I
> couldn't seem to find one.
>
> Thanks,
> Chris
>
> *security.json:*
> {
> "authentication":{"blockUnknown":true,"class":"solr.BasicAuthPlugin",
> "credentials":{
> "solradmin":"...",
> "solrreader":"...",
> "solrwriter":"..."}
> },
> "authorization":{"class":"solr.RuleBasedAuthorizationPlugin",
> "permissions":[
> {"name":"read","role":"reader"},
> {"name":"security-read","role":"reader"},
> {"name":"schema-read","role":"reader"},
> {"name":"config-read","role":"reader"},
> {"name":"core-admin-read","role":"reader"},
> {"name":"collection-admin-read","role":"reader"},
> {"name":"update","role":"writer"},
> {"name":"security-edit","role":"admin"},
> {"name":"schema-edit","role":"admin"},
> {"name":"config-edit","role":"admin"},
> {"name":"core-admin-edit","role":"admin"},
> {"name":"collection-admin-edit","role":"admin"},
> {"name":"all","role":"admin"}],
> "user-role":{
> "solradmin":["reader","writer","admin"],
> "solrreader":["reader"],
> "solrwriter":["reader","writer"]}
> }
> }
>
> *StackTrace:*
> 2018-10-31 16:20:01.994 WARN (MetricsHistoryHandler-12-thread-1) [ ]
> o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node
> ip:8080_solr
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at http://ip:8080/solr: Expected mime type
> application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 401 require authentication</title>
> </head>
> <body><h2>HTTP ERROR 401</h2>
> <p>Problem accessing /solr/admin/metrics. Reason:
> <pre> require authentication</pre></p>
> </body>
> </html>
>
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.ja
> va:607)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.in
> voke(SolrClientNodeStateProvider.java:342)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetri
> cs(SolrClientNodeStateProvider.java:195)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitc
> h.getRemoteInfo(SolrClientNodeStateProvider.java:241)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:7
> 6)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(S
> olrClientNodeStateProvider.java:139)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(S
> olrClientNodeStateProvider.java:128)
> ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:58]
> at
> org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(Me
> tricsHistoryHandler.java:498)
> ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:55]
> at
> org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHi
> storyHandler.java:371)
> ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:55]
> at
> org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(Metrics
> HistoryHandler.java:231)
> ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
> jimczi - 2018-09-18 13:07:55]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_121]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_121]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acces
> s$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_121]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(S
> cheduledThreadPoolExecutor.java:294)
> [?:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1142)
> [?:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
> :617)
> [?:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
>
>
> On Wed, Oct 17, 2018 at 8:32 AM Chris Ulicny <cu...@iq.media> wrote:
>
> > Hi all,
> >
> > Recently in a 7.4.0 test cluster, we ran into SOLR-12814
> > <https://issues.apache.org/jira/browse/SOLR-12814> which we fixed by
> > slightly increasing the request header size. However, there were some other
> > log messages along with the "URI size >8192" message which we thought
> were
> > related, but have not abated since increasing the header size. A full
> > shutdown of the solr processes and bringing them back up one at a time did
> > not solve the issue.
> >
> > The overseer node seems to not be authenticating any of the requests to
> > /solr/admin/metrics on any node (including itself). Every minute, there are
> > two warning per node
> >
> > 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not
> > get tags from node host1:port1_solr
> > 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not
> > get tags from node host1:port1_solr
> > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> > get tags from node host2:port2_solr
> > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> > get tags from node host2:port2_solr
> > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> > get tags from node host3:port3_solr
> > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> > get tags from node host3:port3_solr
> >
> > There are two slightly different stack traces that appear with each pair:
> > https://pastebin.com/2Z1C5rXr
> >
> > The warning message possibly comes from
> > solrj.impl.SolrClientNodeStateProvider.fetchMetrics which both of the
> > attempted requests call in their stack trace.
> >
> > However, we already have a 7.4.0 production cluster running that also has
> > security enabled with similar replica density where we have not seen this
> > issue.
> >
> > *Test:*
> > -- 10 collections (9 with 2 shards, 1 with 43 shards)
> > -- replication factor of 2 for all collections
> > -- 3 hosts with 40 or 41 replicas each
> >
> > *Production:*
> > -- 9 collections with 14 shards
> > -- replication factor of 2 for all collections
> > -- 7 hosts with 36 replicas each
> >
> > I've enabled TRACE logging in our test environment on most options related
> > to metrics and authentication. So far the only new message I've gotten is
> > the challenge from the target server for the necessary credentials right
> > before the warning and stack trace.
> >
> > 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
> > o.a.h.i.a.HttpAuthenticator Authentication required
> > 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
> > o.a.h.i.a.HttpAuthenticator host3:port3 requested authentication
> >
> > I suspect the creation and balancing of the large collection on test might
> > have something to do with it since the problem started happening after
> > that.
> >
> > Are there any other specific log settings I should turn on that might
> > produce some useful information?
> >
> > Thanks,
> > Chris
> >
Re: Overseer could not get tags
Posted by Chris Ulicny <cu...@iq.media>.
I've managed to replicate this issue with the 7.5.0 release as well by
starting up a single instance of solr in cloud mode (on windows) and
uploading the security.json file below to it.
After a short while, the "could not get tags from node..." messages start
coming through every 60 seconds. The accompanying logged error and
expecting stacktrace are also included below.
Is there a JIRA ticket for this issue (or a directly related one)? I
couldn't seem to find one.
Thanks,
Chris
*security.json:*
{
"authentication":{"blockUnknown":true,"class":"solr.BasicAuthPlugin",
"credentials":{
"solradmin":"...",
"solrreader":"...",
"solrwriter":"..."}
},
"authorization":{"class":"solr.RuleBasedAuthorizationPlugin",
"permissions":[
{"name":"read","role":"reader"},
{"name":"security-read","role":"reader"},
{"name":"schema-read","role":"reader"},
{"name":"config-read","role":"reader"},
{"name":"core-admin-read","role":"reader"},
{"name":"collection-admin-read","role":"reader"},
{"name":"update","role":"writer"},
{"name":"security-edit","role":"admin"},
{"name":"schema-edit","role":"admin"},
{"name":"config-edit","role":"admin"},
{"name":"core-admin-edit","role":"admin"},
{"name":"collection-admin-edit","role":"admin"},
{"name":"all","role":"admin"}],
"user-role":{
"solradmin":["reader","writer","admin"],
"solrreader":["reader"],
"solrwriter":["reader","writer"]}
}
}
*StackTrace:*
2018-10-31 16:20:01.994 WARN (MetricsHistoryHandler-12-thread-1) [ ]
o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node
ip:8080_solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://ip:8080/solr: Expected mime type
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 require authentication</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/admin/metrics. Reason:
<pre> require authentication</pre></p>
</body>
</html>
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:342)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:241)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:76)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:139)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:128)
~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:58]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(MetricsHistoryHandler.java:498)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:55]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHistoryHandler.java:371)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:55]
at
org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(MetricsHistoryHandler.java:231)
~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df -
jimczi - 2018-09-18 13:07:55]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_121]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_121]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
On Wed, Oct 17, 2018 at 8:32 AM Chris Ulicny <cu...@iq.media> wrote:
> Hi all,
>
> Recently in a 7.4.0 test cluster, we ran into SOLR-12814
> <https://issues.apache.org/jira/browse/SOLR-12814> which we fixed by
> slightly increasing the request header size. However, there were some other
> log messages along with the "URI size >8192" message which we thought were
> related, but have not abated since increasing the header size. A full
> shutdown of the solr processes and bringing them back up one at a time did
> not solve the issue.
>
> The overseer node seems to not be authenticating any of the requests to
> /solr/admin/metrics on any node (including itself). Every minute, there are
> two warning per node
>
> 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not
> get tags from node host1:port1_solr
> 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not
> get tags from node host1:port1_solr
> 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> get tags from node host2:port2_solr
> 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> get tags from node host2:port2_solr
> 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> get tags from node host3:port3_solr
> 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not
> get tags from node host3:port3_solr
>
> There are two slightly different stack traces that appear with each pair:
> https://pastebin.com/2Z1C5rXr
>
> The warning message possibly comes from
> solrj.impl.SolrClientNodeStateProvider.fetchMetrics which both of the
> attempted requests call in their stack trace.
>
> However, we already have a 7.4.0 production cluster running that also has
> security enabled with similar replica density where we have not seen this
> issue.
>
> *Test:*
> -- 10 collections (9 with 2 shards, 1 with 43 shards)
> -- replication factor of 2 for all collections
> -- 3 hosts with 40 or 41 replicas each
>
> *Production:*
> -- 9 collections with 14 shards
> -- replication factor of 2 for all collections
> -- 7 hosts with 36 replicas each
>
> I've enabled TRACE logging in our test environment on most options related
> to metrics and authentication. So far the only new message I've gotten is
> the challenge from the target server for the necessary credentials right
> before the warning and stack trace.
>
> 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
> o.a.h.i.a.HttpAuthenticator Authentication required
> 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ]
> o.a.h.i.a.HttpAuthenticator host3:port3 requested authentication
>
> I suspect the creation and balancing of the large collection on test might
> have something to do with it since the problem started happening after
> that.
>
> Are there any other specific log settings I should turn on that might
> produce some useful information?
>
> Thanks,
> Chris
>