You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@solr.apache.org by ja...@apache.org on 2023/11/25 00:42:15 UTC
(solr) branch main updated: SOLR-16974: Documentation and changes entry for global CB (#2069)
This is an automated email from the ASF dual-hosted git repository.
janhoy pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/solr.git
The following commit(s) were added to refs/heads/main by this push:
new 2bbb06aac63 SOLR-16974: Documentation and changes entry for global CB (#2069)
2bbb06aac63 is described below
commit 2bbb06aac63ec2ec94aab73874f09959b39f881f
Author: Jan Høydahl <ja...@apache.org>
AuthorDate: Sat Nov 25 01:42:09 2023 +0100
SOLR-16974: Documentation and changes entry for global CB (#2069)
---
solr/CHANGES.txt | 2 +
solr/bin/solr | 2 -
.../deployment-guide/pages/circuit-breakers.adoc | 51 +++++++++++++++++++++-
.../pages/major-changes-in-solr-9.adoc | 3 ++
4 files changed, 54 insertions(+), 4 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 3bcc8d79a9b..e4bae4d67f8 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -82,6 +82,8 @@ New Features
* SOLR-17006: Collection creation & adding replicas: User-defined properties are persisted to state.json and
applied to new replicas, available for use as property substitution in configuration files. (Vincent Primault)
+* SOLR-16974: Circuit Breakers can now be configured globally (janhoy, Christine Poerschke)
+
* SOLR-17079: Allow to declare replica placement plugins in solr.xml (Vincent Primault)
Improvements
diff --git a/solr/bin/solr b/solr/bin/solr
index 4c8c1393c0f..8837f858d29 100755
--- a/solr/bin/solr
+++ b/solr/bin/solr
@@ -1426,8 +1426,6 @@ if [ -n "${SOLR_CIRCUITBREAKER_QUERY_LOADAVG:-}" ]; then
SOLR_OPTS+=("-Dsolr.circuitbreaker.query.loadavg=$SOLR_CIRCUITBREAKER_QUERY_LOADAVG")
fi
-echo "SOLR_OPTS is now: ${SOLR_OPTS[*]}"
-
: ${SOLR_SERVER_DIR:=$DEFAULT_SERVER_DIR}
if [ ! -e "$SOLR_SERVER_DIR" ]; then
diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc
index c074cc64bf9..d8813dd4140 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc
@@ -32,8 +32,26 @@ Setting the `shards.tolerant=true` parameter on requests can help with graceful
circuit breaker thresholds are reached on some nodes. See the xref:deployment-guide:solrcloud-distributed-requests.adoc#shards-tolerant-parameter[`shards.tolerant` Parameter] for details.
== Circuit Breaker Configurations
-All circuit breaker configurations are listed as independent `<circuitBreaker>` entries in `solrconfig.xml` as shown below.
-A circuit breaker can register itself to trip for query requests and/or update requests. By default, only search requests are affected. A user may register multiple circuit breakers of the same type with different thresholds for each request type.
+Circuit breakers can be configured globally for the entire node, or for each collection individually, or a combination. Per-collection circuit breakers are checked before global circuit breakers, and if there is a conflict, the per-collection circuit breaker takes precedence.
+Typically, any per-collection circuit breaker thresholds are set lower than global thresholds.
+
+A circuit breaker can register itself to be checked for query requests and/or update requests. A user may register circuit breakers of the same type with different thresholds for each request type.
+
+=== Global Circuit Breakers
+Circuit breakers can be configured globally using environment variables, e.g. in `solr.in.sh`, or system properties. The variables available are:
+
+[options="header"]
+|===
+|Name |Environment Variable Name |System Property Name
+|JVM Heap Usage |`SOLR_CIRCUITBREAKER_QUERY_MEM`, `SOLR_CIRCUITBREAKER_UPDATE_MEM` |`solr.circuitbreaker.query.mem`, `solr.circuitbreaker.update.mem`
+|System CPU Usage |`SOLR_CIRCUITBREAKER_QUERY_CPU`, `SOLR_CIRCUITBREAKER_UPDATE_CPU` |`solr.circuitbreaker.query.cpu`, `solr.circuitbreaker.update.cpu`
+|System Load Average |`SOLR_CIRCUITBREAKER_QUERY_LOADAVG`, `SOLR_CIRCUITBREAKER_UPDATE_LOADAVG` |`solr.circuitbreaker.query.loadavg`, `solr.circuitbreaker.update.loadavg`
+|===
+
+For example, you can enable a global CPU circuit breaker that rejects update requests when above 95% CPU load, by setting the following environment variable: `SOLR_CIRCUITBREAKER_UPDATE_CPU=95`.
+
+=== Per Collection Circuit Breakers
+Circuit breakers are configured as independent `<circuitBreaker>` entries in `solrconfig.xml` as shown in the below examples. By default, only search requests are affected.
[TIP]
====
@@ -49,6 +67,7 @@ The main configuration for this circuit breaker is controlling the threshold per
To enable and configure the JVM heap usage based circuit breaker, add the following:
+.Per collection in `solrconfig.xml`
[source,xml]
----
<circuitBreaker class="org.apache.solr.util.circuitbreaker.MemoryCircuitBreaker">
@@ -56,6 +75,12 @@ To enable and configure the JVM heap usage based circuit breaker, add the follow
</circuitBreaker>
----
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_MEM=75
+----
+
The `threshold` is defined as a percentage of the max heap allocated to the JVM.
For the circuit breaker configuration, a value of "0" maps to 0% usage and a value of "100" maps to 100% usage.
@@ -78,6 +103,7 @@ disabled and log an error message. An alternative can then be to use the <<syste
To enable and configure the CPU utilization based circuit breaker:
+.Per collection in `solrconfig.xml`
[source,xml]
----
<circuitBreaker class="org.apache.solr.util.circuitbreaker.CPUCircuitBreaker">
@@ -85,6 +111,12 @@ To enable and configure the CPU utilization based circuit breaker:
</circuitBreaker>
----
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_CPU=75
+----
+
The triggering threshold is defined in percent CPU usage. A value of "0" maps to 0% usage
and a value of "100" maps to 100% usage. The example above will trip when the CPU usage is
equal to or greater than 75%.
@@ -100,6 +132,7 @@ https://en.wikipedia.org/wiki/Load_(computing)[Wikipedia page for Load],
To enable and configure the Load average circuit breaker:
+.Per collection in `solrconfig.xml`
[source,xml]
----
<circuitBreaker class="org.apache.solr.util.circuitbreaker.LoadAverageCircuitBreaker">
@@ -107,6 +140,12 @@ To enable and configure the Load average circuit breaker:
</circuitBreaker>
----
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_LOADAVG=8.0
+----
+
The triggering threshold is a floating point number matching load average.
The example circuit breaker above will trip when the load average is equal to or greater than 8.0.
@@ -120,6 +159,7 @@ The System Load Average Circuit breaker behavior is dependent on the operating s
In this example we will prevent update requests above 80% CPU load, and prevent query requests above 95% CPU load. Supported request types are `query` and `update`.
This would prevent expensive bulk updates from impacting search. Note also the support for short-form class name.
+.Per collection in `solrconfig.xml`
[source,xml]
----
<config>
@@ -139,6 +179,13 @@ This would prevent expensive bulk updates from impacting search. Note also the s
</config>
----
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_UPDATE_CPU=80
+SOLR_CIRCUITBREAKER_QUERY_CPU=95
+----
+
== Performance Considerations
While JVM or CPU circuit breakers do not add any noticeable overhead per request, having too many circuit breakers checked for a single request can cause a performance overhead.
diff --git a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
index fc83e288e41..19a5feaf4b4 100644
--- a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
+++ b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
@@ -71,6 +71,9 @@ Due to changes in Lucene 9, that isn't possible any more.
=== Curator upgraded to 5.5.0 and requires Zookeeper 3.5.x or higher
<<#solr-8-2,Solr 8.2 recommended using Zookeeper 3.5.5>> and now with Curator 5.5.0 requires https://curator.apache.org/docs/breaking-changes/[Zookeeper 3.5.x or higher]. This primarily affects users of `hadoop-auth`, but usage of Curator could affect other parts of Solr.
+=== Global Circuit Breakers
+* Circuit breakers can now be configured globally, not only per collection. See xref:deployment-guide:circuit-breakers.adoc[Configuring Circuit Breakers] for more information.
+
== Solr 9.4
=== The Built-In Config Sets
* The build in ConfigSets (`_default` and `sample_techproducts_configs`), now use a default `autoSoftCommit` time of 3 seconds,