You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Daniel Hofer (JIRA)" <ji...@apache.org> on 2019/02/05 10:44:00 UTC
[jira] [Updated] (SOLR-13218) MetricsHistoryHandler failes to
collect metrics in BasicAuthPlugin enabled setup
[ https://issues.apache.org/jira/browse/SOLR-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Hofer updated SOLR-13218:
--------------------------------
Description:
After Updating solr to version 7.6.0 (from 6.6.5) running the setup
* SolrCloud with 3 Zookeepers and 2 Solr Servers
* HTTPS enabled
* BasicAuthentication Enabled
{code:java}
{
"authorization": {
"user-role": {"Solr": "admin"},
"permissions": [{
"role": "admin",
"name": "security-edit"
}],
"class": "solr.RuleBasedAuthorizationPlugin"
},
"authentication": {
"credentials": {"Solr": "1XkpuFRODp3WJJeMqvyGXcu8QSeo0juRvR2g4B1pCvQ= RETXcj9xeYDQ9V+kuDSV3hF2gEUEj79e7Uo+HehPfAE="},
"blockUnknown": true,
"class": "solr.BasicAuthPlugin"
}
}{code}
the following errors started showing up in the logs at regular intervals
{code:java}
11:22:35 [MetricsHistoryHandler-12-thread-1] WARN o.a.s.c.s.i.SolrClientNodeStateProvider - could not get tags from node localhost:8011_solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://localhost:8011/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 require authentication</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/admin/metrics. Reason:
<pre> require authentication</pre></p>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:349)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:242)
at org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:76)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:138)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:127)
at org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(MetricsHistoryHandler.java:498)
at org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHistoryHandler.java:371)
at org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(MetricsHistoryHandler.java:231)
{code}
After some debugging it showed that the error only shows up during the (seemingly) periodic metrics collection on the leader node. The following stack is seen in this cases
!image-2019-02-05-11-29-22-569.png!
While trying to debug it a bit if this can be resolved by some additional settings - for which no solution was found - this method was also called from another location during collection creation where the authentication seems to work just fine
!image-2019-02-05-11-30-33-948.png!
So something in this chain must enable the correct authentication settings it seems.
Appart from periodic error logs generally functionality seems to work but most likely something related to the cluster metrics is not fully working.
Disabling basic authentication or setting ""blockUnknown": false" makes the issue disappear as expected.
was:
After Updating solr to version 7.6.0 (from 6.6.5) running the setup
* SolrCloud with 3 Zookeepers and 2 Solr Servers
* HTTPS enabled
* BasicAuthentication Enabled
{code:java}
{
"authorization": {
"user-role": {"Solr": "admin"},
"permissions": [{
"role": "admin",
"name": "security-edit"
}],
"class": "solr.RuleBasedAuthorizationPlugin"
},
"authentication": {
"credentials": {"Solr": "1XkpuFRODp3WJJeMqvyGXcu8QSeo0juRvR2g4B1pCvQ= RETXcj9xeYDQ9V+kuDSV3hF2gEUEj79e7Uo+HehPfAE="},
"blockUnknown": true,
"class": "solr.BasicAuthPlugin"
}
}{code}
the following errors started showing up in the logs at regular intervals
{code:java}
11:22:35 [MetricsHistoryHandler-12-thread-1] WARN o.a.s.c.s.i.SolrClientNodeStateProvider - could not get tags from node localhost:8011_solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://localhost:8011/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 require authentication</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/admin/metrics. Reason:
<pre> require authentication</pre></p>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:349)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:242)
at org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:76)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:138)
at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:127)
at org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(MetricsHistoryHandler.java:498)
at org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHistoryHandler.java:371)
at org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(MetricsHistoryHandler.java:231)
{code}
After some debugging it showed that the error only shows up during the (seemingly) periodic metrics collection on the leader node. The following stack is seen in this cases
!image-2019-02-05-11-29-22-569.png!
While trying to debug it a bit if this can be resolved by some additional settings - for which no solution was found - this method was also called from another location during collection creation where the authentication seems to work just fine
!image-2019-02-05-11-30-33-948.png!
So something in this chain must enable the correct authentication settings it seems.
Appart from periodic error logs generally functionality seems to work but most likely something related to the cluster metrics is not fully working.
> MetricsHistoryHandler failes to collect metrics in BasicAuthPlugin enabled setup
> --------------------------------------------------------------------------------
>
> Key: SOLR-13218
> URL: https://issues.apache.org/jira/browse/SOLR-13218
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Authentication, metrics
> Affects Versions: 7.6
> Environment: Rhel6 Servers
> Windows 10
> Reporter: Daniel Hofer
> Priority: Major
> Labels: BasicAuth, SolrCloud, metrics
> Attachments: image-2019-02-05-11-29-22-569.png, image-2019-02-05-11-30-33-948.png
>
>
> After Updating solr to version 7.6.0 (from 6.6.5) running the setup
> * SolrCloud with 3 Zookeepers and 2 Solr Servers
> * HTTPS enabled
> * BasicAuthentication Enabled
> {code:java}
> {
> "authorization": {
> "user-role": {"Solr": "admin"},
> "permissions": [{
> "role": "admin",
> "name": "security-edit"
> }],
> "class": "solr.RuleBasedAuthorizationPlugin"
> },
> "authentication": {
> "credentials": {"Solr": "1XkpuFRODp3WJJeMqvyGXcu8QSeo0juRvR2g4B1pCvQ= RETXcj9xeYDQ9V+kuDSV3hF2gEUEj79e7Uo+HehPfAE="},
> "blockUnknown": true,
> "class": "solr.BasicAuthPlugin"
> }
> }{code}
> the following errors started showing up in the logs at regular intervals
> {code:java}
> 11:22:35 [MetricsHistoryHandler-12-thread-1] WARN o.a.s.c.s.i.SolrClientNodeStateProvider - could not get tags from node localhost:8011_solr
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://localhost:8011/solr: Expected mime type application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 401 require authentication</title>
> </head>
> <body><h2>HTTP ERROR 401</h2>
> <p>Problem accessing /solr/admin/metrics. Reason:
> <pre> require authentication</pre></p>
> </body>
> </html>
> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
> at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:349)
> at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
> at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitch.getRemoteInfo(SolrClientNodeStateProvider.java:242)
> at org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:76)
> at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(SolrClientNodeStateProvider.java:138)
> at org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(SolrClientNodeStateProvider.java:127)
> at org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(MetricsHistoryHandler.java:498)
> at org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHistoryHandler.java:371)
> at org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(MetricsHistoryHandler.java:231)
> {code}
> After some debugging it showed that the error only shows up during the (seemingly) periodic metrics collection on the leader node. The following stack is seen in this cases
> !image-2019-02-05-11-29-22-569.png!
> While trying to debug it a bit if this can be resolved by some additional settings - for which no solution was found - this method was also called from another location during collection creation where the authentication seems to work just fine
> !image-2019-02-05-11-30-33-948.png!
> So something in this chain must enable the correct authentication settings it seems.
> Appart from periodic error logs generally functionality seems to work but most likely something related to the cluster metrics is not fully working.
> Disabling basic authentication or setting ""blockUnknown": false" makes the issue disappear as expected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org