You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ab...@apache.org on 2018/05/30 15:36:38 UTC
[2/4] lucene-solr:branch_7x: SOLR-11779: Basic long-term collection
of aggregated metrics.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/090159f9/solr/solr-ref-guide/src/metrics-history.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/metrics-history.adoc b/solr/solr-ref-guide/src/metrics-history.adoc
new file mode 100644
index 0000000..5dc1c3c
--- /dev/null
+++ b/solr/solr-ref-guide/src/metrics-history.adoc
@@ -0,0 +1,411 @@
+= Metrics History
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+== Design
+=== Round-robin databases
+When Solr runs in "cloud" mode it collects long-term history of certain key metrics. This information
+can be used for very simple monitoring and troubleshooting, but also some Solr Cloud components
+(eg. autoscaling) can use this data for making informed decisions based on long-term
+trends of selected metrics.
+
+[IMPORTANT]
+====
+Metrics history is available ONLY in SolrCloud mode, it's not supported in standalone Solr. Also,
+the `.system` collection must exist if metrics history should be persisted.
+====
+
+This data is maintained as multi-resolution time series, with a fixed total number of data points
+per metric history (a fixed size window). Multi-resolution refers to the fact that data from the most detailed
+time series is periodically resampled to create coarser-grained time series, which in turn
+are periodically resampled again to build even coarser-grained series.
+
+In the default configuration selected metrics are sampled every 60 seconds, and the following
+time series are built:
+
+* 240 samples, every 60 sec (4 hours)
+* 288 samples, every 600 sec (48 hours)
+* 336 samples, every 1h (2 weeks)
+* 180 samples, every 4h (2 months)
+* 365 samples, every 1 day (1 year)
+
+This means that the total number of samples in all data series is constant, and consequently
+the size of this data structure is also constant (because the size of the moving window is fixed, and
+older samples are replaced by newer ones). This arrangement is referred to as a
+round-robin database, and Solr uses implementation of this concept provided by RRD4j library.
+
+=== Storage
+Databases created with RRD4j are compact - for the time series specified above the total size
+of data is ca. 11kB for each of the primary time series, including its resampled data. Each database may contain
+several primary time series ("datasources" in RRD4j parlance) and their re-sampled versions (called
+"archives").
+
+This data is updated in memory and then periodically stored in the `.system`
+collection in the form of Solr documents with a binary `data_bin` field, each document
+containing data of one full database. This method of storage is much more compact and generates less
+update operations than storing each data point in a separate Solr document. Metrics history API allows retrieving
+detailed data from each database, including retrieval of all individual datapoints.
+
+Databases are identified primarily by their corresponding metric registry name, so for databases that
+keep track of aggregated metrics this will be eg. `solr.jvm`, `solr.node`, `solr.collection.gettingstarted`,
+and for databases with non-aggregated metrics this will be eg. `solr.jvm.localhost:8983_solr`,
+`solr.node.localhost:7574_solr`, `solr.core.gettingstarted.shard1.replica_n1`.
+
+=== Collected metrics
+Currently the following selected metrics are tracked:
+
+* `solr.core` and `solr.collection` metrics:
+** `QUERY./select.requests`
+** `UPDATE./update.requests`
+** `INDEX.sizeInBytes`
+** `numShards` (aggregated, active shards)
+** `numReplicas` (aggregated, active replicas)
+
+* `solr.node` metrics:
+** `CONTAINER.fs.coreRoot.usableSpace`
+** `numNodes` (aggregated, number of live nodes)
+* `solr.jvm` metrics:
+** `memory.heap.used`
+** `os.processCpuLoad`
+** `os.systemLoadAverage`
+
+Separate databases are created for each of these groups, and each database keeps data for
+all metrics listed in that group.
+
+=== SolrRrdBackendFactory
+This component is responsible for managing in-memory databases and periodically saving them
+to the `.system` collection. If the `.system` collection is not available the updates to the
+databases will be kept in memory, until the time when `.system` collection becomes available.
+
+If the `.system` collection is permanently unavailable then data will not be saved and it will
+be lost when the Solr node is shut down.
+
+=== MetricsHistoryHandler
+This component provides a REST API for accessing the metrics history. It is also responsible for
+collecting and periodically updating the in-memory databases.
+
+This handler also performs aggregation of metrics on per-collection level, and on a cluster level.
+By default only these aggregated metrics are tracked - historic data from each node and each replica
+in each collection is not collected separately. Aggregated databases are managed on the Overseer leader
+node.
+
+The handler assumes that a simple aggregation (sum of partial metric values from each resource) is
+sufficient. This happens to make sense for the default built-in sets of metrics. Future extensions will
+provide other aggregation strategies (average, max, min, ...).
+
+== Metrics History Configuration
+There are two mechanisms for configuring this subsystem:
+
+* `/clusterprops.json` - this is the primary mechanism. It uses the cluster properties JSON
+file in ZooKeeper. Configuration is stored in the `/metrics/history` element in a JSON map.
+
+* `solr.xml` - this is the secondary mechanism, which is not recommended but provided for consistency
+with the existing metrics configuration section in this file. Configuration is stored in the
+`/solr/metrics/history` element of this file.
+
+Currently the following configuration options are supported:
+
+`enable`:: boolean, default is true. If this if false then metrics history is not collected
+but can still be retrieved from existing databases. When this is true then metrics are
+periodically collected, aggregated and saved.
+
+`enableReplicas`:: boolean, default is false. When this is true non-aggregated history will be
+collected for each replica in each collection. When this is false then only aggregated history
+is collected for each collection.
+
+`enableNodes`:: boolean, default is false. When this is true then non-aggregated history will be
+collected separately for each node (for node and JVM metrics), with database names consisting of
+base registry name with appended node name, eg. `solr.jvm.localhost:8983_solr`. When this is false
+then only aggregated history will be collected in a single `solr.jvm` and `solr.node` cluster-wide
+databases.
+
+`collectPeriod`:: integer, in seconds, default is 60. Metrics values will be collected and respective
+databases updated every `collectPeriod` seconds.
+[IMPORTANT]
+====
+Value of `collectPeriod` must be at least 1, and if it's changed then all previously existing databases
+with their historic data must be manually removed (new databases will be created automatically).
+====
+
+`syncPeriod`:: integer, in seconds, default is 60. Data from modified databases will be saved to Solr
+every `syncPeriod` seconds. When accessing the databases via REST API the visibility of most recent
+data depends on this period, because requests accessing the data from other nodes see only the
+version of the data that is stored in the `.system` collection.
+
+=== Example configuration
+Example `/clusterprops.json` file with metrics history configuration that turns on the collection of
+per-node metrics history for node and JVM metrics. Note: typically this file will also contain other
+properties unrelated to metrics history API.
+
+[source,json]
+----
+{
+ "metrics" : {
+ "history" : {
+ "enable" : true,
+ "enableNodes" : true,
+ "syncPeriod" : 300
+ }
+ }
+}
+----
+
+== Metrics History API
+Main entry point for accessing metrics history is `/admin/metrics/history` (or `/api/cluster/metrics/history` for
+v2 API).
+
+The following sections describe actions available in this API. All calls have at least one
+required parameter `action`.
+
+=== List databases (`action=list`)
+This call produces a list of available databases. It supports the following parameters:
+
+`rows`:: optional integer, default is 500. Maximum number of results to return
+
+Example:
+[source,bash]
+----
+curl http://localhost:8983/solr/admin/metrics/history?action=list&rows=10
+----
+[source,json]
+----
+{
+ "responseHeader": {
+ "status": 0,
+ "QTime": 16
+ },
+ "metrics": [
+ "solr.collection..system",
+ "solr.collection.gettingstarted",
+ "solr.jvm",
+ "solr.node"
+ ]
+}
+----
+
+=== Database status (`action=status`)
+This call provides detailed status of the selected database.
+
+The following parameters are supported:
+
+`name`:: string, required: database name
+
+Example:
+[source,bash]
+----
+curl http://localhost:8983/solr/admin/metrics/history?action=status&name=solr.collection.gettingstarted
+----
+[source,json]
+----
+{
+ "responseHeader": {
+ "status": 0,
+ "QTime": 38
+ },
+ "metrics": [
+ "solr.collection.gettingstarted",
+ [
+ "status",
+ {
+ "lastModified": 1527268438,
+ "step": 60,
+ "datasourceCount": 5,
+ "archiveCount": 5,
+ "datasourceNames": [
+ "numShards",
+ "numReplicas",
+ "QUERY./select.requests",
+ "UPDATE./update.requests",
+ "INDEX.sizeInBytes"
+ ],
+ "datasources": [
+ {
+ "datasource": "DS:numShards:GAUGE:120:U:U",
+ "lastValue": 2
+ },
+ {
+ "datasource": "DS:QUERY./select.requests:COUNTER:120:U:U",
+ "lastValue": 8786
+ },
+ ...
+ ],
+ "archives": [
+ {
+ "archive": "RRA:AVERAGE:0.5:1:240",
+ "steps": 1,
+ "consolFun": "AVERAGE",
+ "xff": 0.5,
+ "startTime": 1527254040,
+ "endTime": 1527268380,
+ "rows": 240
+ },
+ {
+ "archive": "RRA:AVERAGE:0.5:10:288",
+ "steps": 10,
+ "consolFun": "AVERAGE",
+ "xff": 0.5,
+ "startTime": 1527096000,
+ "endTime": 1527268200,
+ "rows": 288
+ },
+ ...
+ ]
+ }
+ ]
+ ]
+}
+----
+
+=== Get database data (`action=get`)
+This call retrieves all data collected in the specified database.
+
+The following parameters are supported:
+
+`name`:: string, required: database name
+`format`:: string, optional, default is `list`. Format of the data. Currently the
+following formats are supported:
+
+* `list` - each datapoint is returned as separate JSON element. For efficiency, for each
+datasource in a database for each time series the timestamps are provided separately from
+values (because points from all datasources in a given time series share the same timestamps).
+* `string` - all datapoint values and timestamps are returned as strings, with values separated by new line character.
+* `graph` - data is returned as PNG images, Base64-encoded, containing graphs of each time series values over time.
+
+In each case the response is structured in a similar way: archive identifiers are keys in a JSON map,
+and timestamps / datapoints / graphs are values.
+
+==== Examples
+This is the output using the default `list` format:
+[source,bash]
+----
+curl http://localhost:8983/solr/admin/metrics/history?action=get&name=solr.collection.gettingstarted
+----
+[source,json]
+----
+{
+ "responseHeader": {
+ "status": 0,
+ "QTime": 36
+ },
+ "metrics": [
+ "solr.collection.gettingstarted",
+ [
+ "data",
+ {
+ "RRA:AVERAGE:0.5:1:240": {
+ "timestamps":1527254460,
+ "timestamps":1527254520,
+ "timestamps":1527254580,
+ ...
+ "values": {
+ "numShards": "NaN",
+ "numShards": 2.0,
+ "numShards": 2.0,
+ ...
+ "numReplicas": "NaN",
+ "numReplicas": 4.0,
+ "numReplicas": 4.0,
+ ...
+ "QUERY./select.requests": "NaN",
+ "QUERY./select.requests": 123,
+ "QUERY./select.requests": 456,
+ ...
+ }
+ },
+ "RRA:AVERAGE:0.5:10:288": {
+...
+----
+
+This is the output when using the `string` format:
+[source,bash]
+----
+curl http://localhost:8983/solr/admin/metrics/history?action=get&name=solr.collection.gettingstarted&format=string
+----
+[source,json]
+----
+{
+ "responseHeader": {
+ "status": 0,
+ "QTime": 11
+ },
+ "metrics": [
+ "solr.collection.gettingstarted",
+ [
+ "data",
+ {
+ "RRA:AVERAGE:0.5:1:240": {
+ "timestamps": "1527254820\n1527254880\n1527254940\n...",
+ "values": {
+ "numShards": "NaN\n2.0\n2.0\n2.0\n2.0\n2.0\n2.0\n...",
+ "numReplicas": "NaN\n4.0\n4.0\n4.0\n4.0\n4.0\n4.0\n...",
+ "QUERY./select.requests": "NaN\n123\n456\n789\n...",
+ ...
+ }
+ },
+ "RRA:AVERAGE:0.5:10:288": {
+ ...
+----
+
+This is the output when using the `graph` format:
+[source,bash]
+----
+curl http://localhost:8983/solr/admin/metrics/history?action=get&name=solr.collection.gettingstarted&format=graph
+----
+[source,json]
+----
+{
+ "responseHeader": {
+ "status": 0,
+ "QTime": 2275
+ },
+ "metrics": [
+ "solr.collection.gettingstarted",
+ [
+ "data",
+ {
+ "RRA:AVERAGE:0.5:1:240": {
+ "values": {
+ "numShards": "iVBORw0KGgoAAAANSUhEUgAAAkQAAA...",
+ "numReplicas": "iVBORw0KGgoAAAANSUhEUgAAAkQA...",
+ "QUERY./select.requests": "iVBORw0KGgoAAAANS...",
+ ...
+ }
+ },
+ "RRA:AVERAGE:0.5:10:288": {
+ "values": {
+ "numShards": "iVBORw0KGgoAAAANSUhEUgAAAkQAAA...",
+ ...
+ },
+ ...
+----
+
+.Example 60 sec resolution history graph for `QUERY./select.requests` metric
+image::images/metrics-history/query-graph-60s.png[image]
+
+
+.Example 10 min resolution history graph for `QUERY./select.requests` metric
+image::images/metrics-history/query-graph-10min.png[image]
+
+
+.Example 60 sec resolution history graph for `UPDATE./update.requests` metric
+image::images/metrics-history/update-graph-60s.png[image]
+
+.Example 60 sec resolution history graph for `memory.heap.used` metric
+image::images/metrics-history/memHeap-60s.png[image]
+
+.Example 60 sec resolution history graph for `os.systemLoadAverage` metric
+image::images/metrics-history/loadAvg-60s.png[image]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/090159f9/solr/solr-ref-guide/src/monitoring-solr.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/monitoring-solr.adoc b/solr/solr-ref-guide/src/monitoring-solr.adoc
index 2fb7077..2a8e1e8 100644
--- a/solr/solr-ref-guide/src/monitoring-solr.adoc
+++ b/solr/solr-ref-guide/src/monitoring-solr.adoc
@@ -1,5 +1,5 @@
= Monitoring Solr
-:page-children: metrics-reporting, mbean-request-handler, configuring-logging, using-jmx-with-solr, monitoring-solr-with-prometheus-and-grafana, performance-statistics-reference
+:page-children: metrics-reporting, metrics-history, mbean-request-handler, configuring-logging, using-jmx-with-solr, monitoring-solr-with-prometheus-and-grafana, performance-statistics-reference
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -24,6 +24,8 @@ Common administrative tasks include:
<<metrics-reporting.adoc#metrics-reporting,Metrics Reporting>>: Details of Solr's metrics registries and Metrics API.
+<<metrics-history.adoc#metrics-history,Metrics History>>: Metrics history collection, configuration and API.
+
<<mbean-request-handler.adoc#mbean-request-handler,MBean Request Handler>>: How to use Solr's MBeans for programmatic access to the system plugins and stats.
<<configuring-logging.adoc#configuring-logging,Configuring Logging>>: Describes how to configure logging for Solr.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/090159f9/solr/solrj/src/java/org/apache/solr/common/params/CommonParams.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/common/params/CommonParams.java b/solr/solrj/src/java/org/apache/solr/common/params/CommonParams.java
index 6fe2950..3ce491b 100644
--- a/solr/solrj/src/java/org/apache/solr/common/params/CommonParams.java
+++ b/solr/solrj/src/java/org/apache/solr/common/params/CommonParams.java
@@ -182,6 +182,7 @@ public interface CommonParams {
String AUTHC_PATH = "/admin/authentication";
String ZK_PATH = "/admin/zookeeper";
String METRICS_PATH = "/admin/metrics";
+ String METRICS_HISTORY_PATH = "/admin/metrics/history";
String AUTOSCALING_PATH = "/admin/autoscaling";
String AUTOSCALING_HISTORY_PATH = "/admin/autoscaling/history";
String AUTOSCALING_DIAGNOSTICS_PATH = "/admin/autoscaling/diagnostics";
@@ -194,6 +195,7 @@ public interface CommonParams {
AUTHC_PATH,
AUTHZ_PATH,
METRICS_PATH,
+ METRICS_HISTORY_PATH,
AUTOSCALING_PATH,
AUTOSCALING_HISTORY_PATH,
AUTOSCALING_DIAGNOSTICS_PATH,
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/090159f9/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java b/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java
index 3fc6157..58c39a3 100644
--- a/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java
+++ b/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java
@@ -101,6 +101,8 @@ public abstract class CoreAdminParams
public static final String DELETE_INSTANCE_DIR = "deleteInstanceDir";
+ public static final String DELETE_METRICS_HISTORY = "deleteMetricsHistory";
+
public static final String LOAD_ON_STARTUP = "loadOnStartup";
public static final String TRANSIENT = "transient";
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/090159f9/solr/solrj/src/resources/apispec/metrics.history.json
----------------------------------------------------------------------
diff --git a/solr/solrj/src/resources/apispec/metrics.history.json b/solr/solrj/src/resources/apispec/metrics.history.json
new file mode 100644
index 0000000..975d775
--- /dev/null
+++ b/solr/solrj/src/resources/apispec/metrics.history.json
@@ -0,0 +1,23 @@
+{
+ "documentation": "https://lucene.apache.org/solr/guide/metrics-reporting.html",
+ "description": "Metrics history handler allows retrieving samples of past metrics recorded in the .system collection.",
+ "methods": [
+ "GET"
+ ],
+ "url": {
+ "paths": [
+ "/cluster/metrics/history"
+ ],
+ "params": {
+ "action": {
+ "type": "string",
+ "description": "one of supported actions",
+ "default": "list"
+ },
+ "name": {
+ "type": "string",
+ "description": "registry name"
+ }
+ }
+ }
+}