You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/04/22 06:50:00 UTC
[jira] [Commented] (SOLR-13234) Prometheus Metric Exporter Not
Threadsafe
[ https://issues.apache.org/jira/browse/SOLR-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822903#comment-16822903 ]
ASF subversion and git services commented on SOLR-13234:
--------------------------------------------------------
Commit 4571a2d66687cca6670885a94414c7a8c02c0bbc in lucene-solr's branch refs/heads/master from Shalin Shekhar Mangar
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4571a2d ]
SOLR-13392: Add all solr dependencies to prometheus exporter classpath to make sure that it can start.
SOLR-13234 broke prometheus exporter startup from the startup scripts because there was a mismatch between the dependency list in ant/ivy and those actually added to the classpath by the script. This commit changes the script to add all solr dependencies to the classpath.
> Prometheus Metric Exporter Not Threadsafe
> -----------------------------------------
>
> Key: SOLR-13234
> URL: https://issues.apache.org/jira/browse/SOLR-13234
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: metrics
> Affects Versions: 7.6, 8.0
> Reporter: Danyal Prout
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Labels: metric-collector
> Fix For: 7.7.2, 8.1, master (9.0)
>
> Attachments: SOLR-13234-branch_7x.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The Solr Prometheus Exporter collects metrics when it receives a HTTP request from Prometheus. Prometheus sends this request, on its [scrape interval|https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config]. When the time taken to collect the Solr metrics is greater than the scrape interval of the Prometheus server, this results in concurrent metric collection occurring in this [method|https://github.com/apache/lucene-solr/blob/master/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/collector/SolrCollector.java#L86]. This method doesn’t appear to be thread safe, for instance you could have concurrent modifications of a [map|https://github.com/apache/lucene-solr/blob/master/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/collector/SolrCollector.java#L119]. After a while the Solr Exporter processes becomes nondeterministic, we've observed NPE and loss of metrics.
> To address this, I'm proposing the following fixes:
> 1. Read/parse the configuration at startup and make it immutable.
> 2. Collect metrics from Solr on an interval which is controlled by the Solr Exporter and cache the metric samples to return during Prometheus scraping. Metric collection can be expensive, for example executing arbitrary Solr searches, it's not ideal to allow for concurrent metric collection and on an interval which is not defined by the Solr Exporter.
> There are also a few other performance improvements that we've made while fixing this, for example using the ClusterStateProvider instead of sending multiple HTTP requests to each Solr node to lookup all the cores.
> I'm currently finishing up these changes which I'll submit as a PR.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org