You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/11/10 06:15:10 UTC
[GitHub] [solr-operator] vipul-06 opened a new issue, #497: solr prometheus exporter crashloopbackoff
vipul-06 opened a new issue, #497:
URL: https://github.com/apache/solr-operator/issues/497
I have deployed solr prometheus exporter for monitoring purpose. It is running on gke but my exporter pod is gettting error of crashloopbackoff. The logs of my exporter pod are like this
INFO - 2022-11-10 05:45:56.943; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
INFO - 2022-11-10 05:45:56.991; org.apache.solr.common.cloud.ConnectionManager; zkClient has connected
INFO - 2022-11-10 05:45:56.991; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
INFO - 2022-11-10 05:45:57.003; org.apache.solr.common.cloud.ZkStateReader; Updated live nodes from ZooKeeper... (0) -> (5)
INFO - 2022-11-10 05:45:57.021; org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at experro-new-solrcloud-zookeeper-0.experro-new-solrcloud-zookeeper-headless.dev-backend.svc.cluster.local:2181,experro-new-solrcloud-zookeeper-1.experro-new-solrcloud-zookeeper-headless.dev-backend.svc.cluster.local:2181,experro-new-solrcloud-zookeeper-2.experro-new-solrcloud-zookeeper-headless.dev-backend.svc.cluster.local:2181,experro-new-solrcloud-zookeeper-3.experro-new-solrcloud-zookeeper-headless.dev-backend.svc.cluster.local:2181,experro-new-solrcloud-zookeeper-4.experro-new-solrcloud-zookeeper-headless.dev-backend.svc.cluster.local:2181 ready
INFO - 2022-11-10 05:45:57.037; org.apache.solr.prometheus.exporter.SolrExporter; Starting Solr Prometheus Exporting
INFO - 2022-11-10 05:45:57.039; org.apache.solr.prometheus.collector.SchedulerMetricsCollector; Beginning metrics collection
INFO - 2022-11-10 05:45:57.055; org.apache.solr.prometheus.exporter.SolrExporter; Solr Prometheus Exporter is running
WARN - 2022-11-10 05:46:00.154; org.apache.solr.prometheus.scraper.SolrScraper; Error occurred during metrics collection => org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://experro-new-solrcloud-0.experro-new-solrcloud-headless.dev-backend:8983/solr: Forbidden
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://experro-new-solrcloud-0.experro-new-solrcloud-headless.dev-backend:8983/solr: Forbidden
request: http://experro-new-solrcloud-0.experro-new-solrcloud-headless.dev-backend:8983/solr/ylxg9u69_merchandising_rules_field_data_shard1_replica_n1/admin/ping?_stateVer_=z7rven5j_merchandising_rules_field_data:6&wt=json&version=2.2
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:681) ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:42]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:42]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:42]
at org.apache.solr.prometheus.scraper.SolrScraper.request(SolrScraper.java:117) ~[solr-prometheus-exporter-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:35]
at org.apache.solr.prometheus.scraper.SolrCloudScraper.lambda$pingAllCores$1(SolrCloudScraper.java:77) ~[solr-prometheus-exporter-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:35]
at org.apache.solr.prometheus.scraper.SolrScraper.lambda$null$0(SolrScraper.java:86) ~[solr-prometheus-exporter-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:35]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:42]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
WARN - 2022-11-10 05:46:00.174; org.apache.solr.prometheus.scraper.SolrScraper; Error occurred during metrics collection => org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://experro-new-solrcloud-0.experro-new-solrcloud-headless.dev-backend:8983/solr: Forbidden
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://experro-new-solrcloud-0.experro-new-solrcloud-headless.dev-backend:8983/solr: Forbidden
request: http://experro-new-solrcloud-0.experro-new-solrcloud-headless.dev-backend:8983/solr/ylxg9u69_merchandising_rules_field_data_shard2_replica_n10/admin/ping?_stateVer_=z7rven5j_merchandising_rules_field_data:6&wt=json&version=2.2
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:681) ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:42]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:42]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:42]
at org.apache.solr.prometheus.scraper.SolrScraper.request(SolrScraper.java:117) ~[solr-prometheus-exporter-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:35]
at org.apache.solr.prometheus.scraper.SolrCloudScraper.lambda$pingAllCores$1(SolrCloudScraper.java:77) ~[solr-prometheus-exporter-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:35]
at org.apache.solr.prometheus.scraper.SolrScraper.lambda$null$0(SolrScraper.java:86) ~[solr-prometheus-exporter-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:35]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) ~[solr-solrj-8.9.0.jar:8.9.0 05c8a6f0163fe4c330e93775e8e91f3ab66a3f80 - mayyasharipova - 2021-06-10 17:54:42]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] HoustonPutman commented on issue #497: solr prometheus exporter crashloopbackoff
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on issue #497:
URL: https://github.com/apache/solr-operator/issues/497#issuecomment-1332632924
is that the same basic auth secret that the SolrCloud is setup to use? have you changed the solrcloud security after creating the cloud? The [default roles provided by the Solr Operator](https://github.com/apache/solr-operator/blob/v0.6.0/controllers/util/solr_security_util.go#L354) should allow for calling the ping handler.
Can you check your solr cloud security json and see if the user in that basic auth secret is allowed to use the ping handler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] HoustonPutman commented on issue #497: solr prometheus exporter crashloopbackoff
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on issue #497:
URL: https://github.com/apache/solr-operator/issues/497#issuecomment-1310741599
Can you share the yamls for your solrcloud and solrprometheusexporters? Hard to debug without that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] vipul-06 commented on issue #497: solr prometheus exporter crashloopbackoff
Posted by GitBox <gi...@apache.org>.
vipul-06 commented on issue #497:
URL: https://github.com/apache/solr-operator/issues/497#issuecomment-1321544461
Solr is deployed through helm and exporter yaml is below here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr-operator] fliphess commented on issue #497: solr prometheus exporter crashloopbackoff
Posted by "fliphess (via GitHub)" <gi...@apache.org>.
fliphess commented on issue #497:
URL: https://github.com/apache/solr-operator/issues/497#issuecomment-1540600036
I ran into a similar issue where the prometheus exporter kept crashing: Changing the CPU limits on the prometheus exporter did the trick for me: Otherwise every collection round it crashes because it's being throttled by the CPU.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org