You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by sh...@apache.org on 2017/06/07 14:11:25 UTC
[21/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-10509,
SOLR-10649: add docs for new metric features; add to solr.xml docs
SOLR-10509, SOLR-10649: add docs for new metric features; add <metrics> to solr.xml docs
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/9e99a23f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/9e99a23f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/9e99a23f
Branch: refs/heads/feature/autoscaling
Commit: 9e99a23f31b8d3508526ea473b944beb13303334
Parents: a477ebb
Author: Cassandra Targett <ct...@apache.org>
Authored: Fri Jun 2 14:50:54 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Fri Jun 2 14:52:02 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/format-of-solr-xml.adoc | 27 +++---
solr/solr-ref-guide/src/metrics-reporting.adoc | 97 +++++++++++++++++---
2 files changed, 99 insertions(+), 25 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9e99a23f/solr/solr-ref-guide/src/format-of-solr-xml.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/format-of-solr-xml.adoc b/solr/solr-ref-guide/src/format-of-solr-xml.adoc
index 4e55033..02e72bd 100644
--- a/solr/solr-ref-guide/src/format-of-solr-xml.adoc
+++ b/solr/solr-ref-guide/src/format-of-solr-xml.adoc
@@ -22,7 +22,6 @@ The `solr.xml` file defines some global configuration options that apply to all
This section will describe the default `solr.xml` file included with Solr and how to modify it for your needs. For details on how to configure `core.properties`, see the section <<defining-core-properties.adoc#defining-core-properties,Defining core.properties>>.
-[[Formatofsolr.xml-Definingsolr.xml]]
== Defining solr.xml
You can find `solr.xml` in your `$SOLR_HOME` directory (usually `server/solr`) in standalone mode or in Zookeeper when using SolrCloud. The default `solr.xml` file looks like this:
@@ -50,10 +49,9 @@ You can find `solr.xml` in your `$SOLR_HOME` directory (usually `server/solr`) i
As you can see, the discovery Solr configuration is "SolrCloud friendly". However, the presence of the `<solrcloud>` element does _not_ mean that the Solr instance is running in SolrCloud mode. Unless the `-DzkHost` or `-DzkRun` are specified at startup time, this section is ignored.
-[[Formatofsolr.xml-Solr.xmlParameters]]
-=== Solr.xml Parameters
+== Solr.xml Parameters
-==== The `<solr>` Element
+=== The <solr> Element
There are no attributes that you can specify in the `<solr>` tag, which is the root element of `solr.xml`. The tables below list the child nodes of each XML element in `solr.xml`.
@@ -74,7 +72,7 @@ There are no attributes that you can specify in the `<solr>` tag, which is the r
|`configSetBaseDir` |The directory under which configsets for Solr cores can be found. Defaults to `$SOLR_HOME/configsets`.
|===
-==== The `<solrcloud>` Element
+=== The <solrcloud> Element
This element defines several parameters that relate so SolrCloud. This section is ignored unless theSolr instance is started with either `-DzkRun` or `-DzkHost`
@@ -96,7 +94,7 @@ This element defines several parameters that relate so SolrCloud. This section i
|`zkCredentialsProvider` & ` zkACLProvider` |Optional parameters that can be specified if you are using <<zookeeper-access-control.adoc#zookeeper-access-control,ZooKeeper Access Control>>.
|===
-==== The `<logging>` Element
+=== The <logging> Element
// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
@@ -107,7 +105,7 @@ This element defines several parameters that relate so SolrCloud. This section i
|`enabled` |true/false - whether to enable logging or not.
|===
-===== The `<logging><watcher>` Element
+==== The <logging><watcher> Element
// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
@@ -118,7 +116,7 @@ This element defines several parameters that relate so SolrCloud. This section i
|`threshold` |The logging level above which your particular logging implementation will record. For example when using log4j one might specify DEBUG, WARN, INFO, etc.
|===
-==== The `<shardHandlerFactory>` Element
+=== The <shardHandlerFactory> Element
Custom shard handlers can be defined in `solr.xml` if you wish to create a custom shard handler.
@@ -146,14 +144,19 @@ Since this is a custom shard handler, sub-elements are specific to the implement
|`fairnessPolicy` |A boolean to configure if the threadpool favours fairness over throughput. Default is false to favor throughput.
|===
-[[Formatofsolr.xml-SubstitutingJVMSystemPropertiesinsolr.xml]]
+=== The <metrics> Element
+
+The `<metrics>` element in `solr.xml` allows you to customize the metrics reported by Solr. You can define system properties that should not be returned, or define custom suppliers and reporters.
+
+In a default `solr.xml` you will not see any `<metrics>` configuration. If you would like to customize the metrics for your installation, see the section <<metrics-reporting.adoc#metrics-configuration,Metrics Configuration>>.
+
== Substituting JVM System Properties in solr.xml
-Solr supports variable substitution of JVM system property values in `solr.xml`, which allows runtime specification of various configuration options. The syntax is `${propertyname[:option default value]`}. This allows defining a default that can be overridden when Solr is launched. If a default value is not specified, then the property must be specified at runtime or the `solr.xml` file will generate an error when parsed.
+Solr supports variable substitution of JVM system property values in `solr.xml`, which allows runtime specification of various configuration options. The syntax is `${propertyname[:option default value]}`. This allows defining a default that can be overridden when Solr is launched. If a default value is not specified, then the property must be specified at runtime or the `solr.xml` file will generate an error when parsed.
-Any JVM system properties usually specified using the -D flag when starting the JVM, can be used as variables in the `solr.xml` file.
+Any JVM system properties usually specified using the `-D` flag when starting the JVM, can be used as variables in the `solr.xml` file.
-For example, in the `solr.xml` file shown below, the `socketTimeout` and `connTimeout` values are each set to "0". However, if you start Solr using '`bin/solr -DsocketTimeout=1000`', the `socketTimeout` option of the `HttpShardHandlerFactory` to be overridden using a value of 1000ms, while the `connTimeout` option will continue to use the default property value of "0".
+For example, in the `solr.xml` file shown below, the `socketTimeout` and `connTimeout` values are each set to "0". However, if you start Solr using `bin/solr -DsocketTimeout=1000`, the `socketTimeout` option of the `HttpShardHandlerFactory` to be overridden using a value of 1000ms, while the `connTimeout` option will continue to use the default property value of "0".
[source,xml]
----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9e99a23f/solr/solr-ref-guide/src/metrics-reporting.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/metrics-reporting.adoc b/solr/solr-ref-guide/src/metrics-reporting.adoc
index 63fdbf3..4fa049e 100644
--- a/solr/solr-ref-guide/src/metrics-reporting.adoc
+++ b/solr/solr-ref-guide/src/metrics-reporting.adoc
@@ -28,7 +28,7 @@ Internally this feature uses the http://metrics.dropwizard.io[Dropwizard Metrics
* *timers* - measure the number and duration of events. They provide a count and histogram of timings.
* *gauges* - offer instantaneous reading of a current value, e.g., current queue depth, current number of active connections, free heap size.
-Each group of related metrics with unique names is managed in a *metric registry*. Solr maintains several such registries, each corresponding to a high-level group such as: `jvm`, `jetty`, `http`, `node`, and `core` (see <<Metric Registries>> below).
+Each group of related metrics with unique names is managed in a *metric registry*. Solr maintains several such registries, each corresponding to a high-level group such as: `jvm`, `jetty`, `node`, and `core` (see <<Metric Registries>> below).
For each group (and/or for each registry) there can be several *reporters*, which are components responsible for communication of metrics from selected registries to external systems. Currently implemented reporters support emitting metrics via JMX, Ganglia, Graphite and SLF4J.
@@ -42,7 +42,9 @@ Metrics are maintained and accumulated through all lifecycles of components from
These are the major groups of metrics that are collected:
-=== JVM Registry (`solr.jvm`):
+=== JVM Registry
+
+This registry is returned at `solr.jvm` and includes the following information. When making requests with the <<Metrics API>>, you can specify `&group=jvm` to limit to only these metrics.
* direct and mapped buffer pools
* class loading / unloading
@@ -50,26 +52,28 @@ These are the major groups of metrics that are collected:
* GC count and time
* heap, non-heap memory and GC pools
* number of threads, their states and deadlocks
+* System properties such as Java information, various installation directory paths, ports, and similar information. You can control what appears here by modifying `solr.xml`.
+// TODO for 7.0 fix this
+
+=== Node / CoreContainer Registry
-=== Node / CoreContainer Registry (`solr.node`):
+This registry is returned at `solr.node` and includes the following information. When making requests with the <<Metrics API>>, you can specify `&group=node` to limit to only these metrics.
* handler requests (count, timing): collections, info, admin, configSets, etc.
* number of cores (loaded, lazy, unloaded)
-=== Core (SolrCore) Registry:
+=== Core (SolrCore) Registry
-The <<Core Level Metrics,Core (SolrCore) Registry>> includes `solr.core.<collection>`, one for each core.
+The <<Core Level Metrics,Core (SolrCore) Registry>> includes `solr.core.<collection>`, one for each core. When making requests with the <<Metrics API>>, you can specify `&group=core` to limit to only these metrics.
* all common RequestHandler-s report: request timers / counters, timeouts, errors.
* <<Index Merge Metrics,index-level events>>: meters for minor / major merges, number of merged docs, number of deleted docs, gauges for currently running merges and their size.
* shard replication and transaction log replay on replicas (TBD, SOLR-9856)
-* TBD: caches, update handler details, and other relevant SolrInfoMBean-s
-
-=== HTTP Registry (`solr.http`):
-
* open / available / pending connections for shard handler and update handler
-=== Jetty Registry (`solr.jetty`):
+=== Jetty Registry
+
+This registry is returned at `solr.jetty` and includes the following information. When making requests with the <<Metrics API>>, you can specify `&group=jetty` to limit to only these metrics.
* threads and pools,
* connection and request timers,
@@ -77,6 +81,69 @@ The <<Core Level Metrics,Core (SolrCore) Registry>> includes `solr.core.<collect
In the future, metrics will be added for shard leaders and cluster nodes, including aggregations from per-core metrics.
+== Metrics Configuration
+
+The metrics available in your system can be customized by modifying the `<metrics>` element in `solr.xml`.
+
+TIP: See also the section <<format-of-solr-xml.adoc#format-of-solr-xml,Format of Solr.xml>> for more information about the `solr.xml` file, where to find it, and how to edit it.
+
+=== The <metrics><hiddenSysProps> Element
+
+This section of `solr.xml` allows you to define the system properties which are considered system-sensitive and should not be exposed via the Metrics API.
+
+If this section is not defined, the following default configuration is used which hides password and authentication information:
+
+[source,xml]
+<metrics>
+ <hiddenSysProps>
+ <str>javax.net.ssl.keyStorePassword</str>
+ <str>javax.net.ssl.trustStorePassword</str>
+ <str>basicauth</str>
+ <str>zkDigestPassword</str>
+ <str>zkDigestReadonlyPassword</str>
+ </hiddenSysProps>
+</metrics>
+
+=== The <metrics><reporters> Element
+
+Reporters consume the metrics data generated by Solr. See the section <<Reporters>> below for more details on how to configure custom reporters.
+
+=== The <metrics><suppliers> Element
+
+Suppliers help Solr generate metrics data. The `<metrics><suppliers>` section of `solr.xml` allows you to define your own implementations of metrics and configure parameters for them.
+
+Implementation of a custom metrics supplier is beyond the scope of this guide, but there are other customizations possible with the default implementation, via the elements described below.
+
+<counter>:: This element defines the implementation and configuration of a `Counter` supplier. The default implementation does not support any configuration.
+
+<meter>:: This element defines the implementation of a `Meter` supplier. The default implementation supports an additional parameter:
+`<str name="clock">`::: The type of clock to use for calculating EWMA rates. The supported values are:
+* `user`, the default, which uses `System.nanoTime()`
+* `cpu`, which uses the current thread's CPU time
+
+<histogram>:: This element defines the implementation of a `Histogram` supplier. This element also supports the `clock` parameter shown above with the `meter` element, and also:
+`<str name="reservoir">`::: The fully-qualified class name of the `Reservoir` implementation to use. The default is `com.codahale.metrics.ExponentiallyDecayingReservoir` but there are other options available with the http://metrics.dropwizard.io/3.1.0/manual/core/#histograms[Codahale Metrics library] that Solr uses. The following parameters are supported, within the mentioned limitations:
+* `size`, the reservoir size. The default is 1028.
+* `alpha`, the decay parameter. The default is 0.015. This is only valid for the `ExponentiallyDecayingReservoir`.
+* `window`, the window size, in seconds, and only valid for the `SlidingTimeWindowReservoir`. The default is 300 (5 minutes).
+
+<timer>:: This element defines an implementation of a `Timer` supplier. The default implementation supports the `clock` and `reservoir` parameters described above.
+
+As an example of a section of `solr.xml` that defines some of these custom parameters, the following defines the default `Meter` supplier with a non-default `clock` and the default `Timer` is used with a non-default reservoir:
+
+[source,xml]
+<metrics>
+ <suppliers>
+ <meter>
+ <str name="clock">cpu</str>
+ </meter>
+ <timer>
+ <str name="reservoir">com.codahale.metrics.SlidingTimeWindowReservoir</str>
+ <long name="window">600</long>
+ </timer>
+ </suppliers>
+</metrics>
+
== Reporters
Reporter configurations are specified in `solr.xml` file in `<metrics><reporter>` sections, for example:
@@ -101,6 +168,8 @@ Reporter configurations are specified in `solr.xml` file in `<metrics><reporter>
</solr>
----
+This example configures two reporters: <<Graphite Reporter,Graphite>> and <<SLF4J Reporter,SLF4J>>. See below for more details on how to configure reporters.
+
=== Reporter Arguments
Reporter plugins use the following arguments:
@@ -260,15 +329,17 @@ If the boolean flag `mergeDetails` is true then the following additional metrics
The `admin/metrics` endpoint provides access to all the metrics for all metric groups.
-A few query parameters are available to limit the request:
+A few query parameters are available to limit your request to only certain metrics:
group:: The metric group to retrieve. The default is `all` to retrieve all metrics for all groups. Other possible values are: `jvm`, `jetty`, `node`, and `core`. More than one group can be specified in a request; multiple group names should be separated by a comma.
type:: The type of metric to retrieve. The default is `all` to retrieve all metric types. Other possible values are `counter`, `gauge`, `histogram`, `meter`, and `timer`. More than one type can be specified in a request; multiple types should be separated by a comma.
-prefix:: The first characters of metric name that will filter the metrics returned to those starting with the provided string. It can be combined with group and/or type parameters. More than one prefix can be specified in a request; multiple prefixes should be separated by a comma. Prefix matching is also case-sensitive.
+prefix:: The first characters of metric name that will filter the metrics returned to those starting with the provided string. It can be combined with `group` and/or `type` parameters. More than one prefix can be specified in a request; multiple prefixes should be separated by a comma. Prefix matching is also case-sensitive.
+
+property:: Allows requesting only this metric from any compound metric. Multiple `property` parameters can be combined to act as an OR request. For example, to only get the 99th and 999th percentile values from all metric types and groups, you can add `&property=p99_ms&property=p999_ms` to your request. This can be combined with `group`, `type`, and `prefix` as necessary.
-compact:: When true, a more compact format of the response will be returned. Instead of a response like this:
+compact:: When false, a more verbose format of the response will be returned. Instead of a response like this:
+
[source,json]
"metrics": [