You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@solr.apache.org by ja...@apache.org on 2023/11/25 01:36:02 UTC
(solr) branch branch_9x updated: SOLR-16974: Documentation and changes entry for global CB (9x) (#2094)
This is an automated email from the ASF dual-hosted git repository.
janhoy pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/solr.git
The following commit(s) were added to refs/heads/branch_9x by this push:
new 6b2660edc20 SOLR-16974: Documentation and changes entry for global CB (9x) (#2094)
6b2660edc20 is described below
commit 6b2660edc2052b36ff20ec5e3b972d18a8829d7d
Author: Jan Høydahl <ja...@apache.org>
AuthorDate: Sat Nov 25 02:35:57 2023 +0100
SOLR-16974: Documentation and changes entry for global CB (9x) (#2094)
(cherry picked from commit 2bbb06aac63ec2ec94aab73874f09959b39f881f)
---
solr/CHANGES.txt | 2 +
solr/bin/solr | 2 -
.../deployment-guide/pages/circuit-breakers.adoc | 51 +++++++++++++++++++++-
.../pages/major-changes-in-solr-9.adoc | 3 ++
4 files changed, 54 insertions(+), 4 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index d8d8558ca67..b5d9cfd31bd 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -13,6 +13,8 @@ New Features
* SOLR-17006: Collection creation & adding replicas: User-defined properties are persisted to state.json and
applied to new replicas, available for use as property substitution in configuration files. (Vincent Primault)
+* SOLR-16974: Circuit Breakers can now be configured globally (janhoy, Christine Poerschke)
+
* SOLR-17079: Allow to declare replica placement plugins in solr.xml (Vincent Primault)
Improvements
diff --git a/solr/bin/solr b/solr/bin/solr
index 5a66d7ff60e..2c7a2046002 100755
--- a/solr/bin/solr
+++ b/solr/bin/solr
@@ -1902,8 +1902,6 @@ if [ -n "${SOLR_CIRCUITBREAKER_QUERY_LOADAVG:-}" ]; then
SOLR_OPTS+=("-Dsolr.circuitbreaker.query.loadavg=$SOLR_CIRCUITBREAKER_QUERY_LOADAVG")
fi
-echo "SOLR_OPTS is now: ${SOLR_OPTS[*]}"
-
: ${SOLR_SERVER_DIR:=$DEFAULT_SERVER_DIR}
if [ ! -e "$SOLR_SERVER_DIR" ]; then
diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc
index c640395aaeb..3b9b5600370 100644
--- a/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc
+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc
@@ -32,8 +32,26 @@ Setting the `shards.tolerant=true` parameter on requests can help with graceful
circuit breaker thresholds are reached on some nodes. See the xref:deployment-guide:solrcloud-distributed-requests.adoc#shards-tolerant-parameter[`shards.tolerant` Parameter] for details.
== Circuit Breaker Configurations
-All circuit breaker configurations are listed as independent `<circuitBreaker>` entries in `solrconfig.xml` as shown below.
-A circuit breaker can register itself to trip for query requests and/or update requests. By default only search requests are affected. A user may register multiple circuit breakers of the same type with different thresholds for each request type.
+Circuit breakers can be configured globally for the entire node, or for each collection individually, or a combination. Per-collection circuit breakers are checked before global circuit breakers, and if there is a conflict, the per-collection circuit breaker takes precedence.
+Typically, any per-collection circuit breaker thresholds are set lower than global thresholds.
+
+A circuit breaker can register itself to be checked for query requests and/or update requests. A user may register circuit breakers of the same type with different thresholds for each request type.
+
+=== Global Circuit Breakers
+Circuit breakers can be configured globally using environment variables, e.g. in `solr.in.sh`, or system properties. The variables available are:
+
+[options="header"]
+|===
+|Name |Environment Variable Name |System Property Name
+|JVM Heap Usage |`SOLR_CIRCUITBREAKER_QUERY_MEM`, `SOLR_CIRCUITBREAKER_UPDATE_MEM` |`solr.circuitbreaker.query.mem`, `solr.circuitbreaker.update.mem`
+|System CPU Usage |`SOLR_CIRCUITBREAKER_QUERY_CPU`, `SOLR_CIRCUITBREAKER_UPDATE_CPU` |`solr.circuitbreaker.query.cpu`, `solr.circuitbreaker.update.cpu`
+|System Load Average |`SOLR_CIRCUITBREAKER_QUERY_LOADAVG`, `SOLR_CIRCUITBREAKER_UPDATE_LOADAVG` |`solr.circuitbreaker.query.loadavg`, `solr.circuitbreaker.update.loadavg`
+|===
+
+For example, you can enable a global CPU circuit breaker that rejects update requests when above 95% CPU load, by setting the following environment variable: `SOLR_CIRCUITBREAKER_UPDATE_CPU=95`.
+
+=== Per Collection Circuit Breakers
+Circuit breakers are configured as independent `<circuitBreaker>` entries in `solrconfig.xml` as shown in the below examples. By default, only search requests are affected.
== Currently Supported Circuit Breakers
@@ -52,6 +70,7 @@ The main configuration for this circuit breaker is controlling the threshold per
To enable and configure the JVM heap usage based circuit breaker, add the following:
+.Per collection in `solrconfig.xml`
[source,xml]
----
<circuitBreaker class="org.apache.solr.util.circuitbreaker.MemoryCircuitBreaker">
@@ -59,6 +78,12 @@ To enable and configure the JVM heap usage based circuit breaker, add the follow
</circuitBreaker>
----
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_MEM=75
+----
+
The `threshold` is defined as a percentage of the max heap allocated to the JVM.
For the circuit breaker configuration, a value of "0" maps to 0% usage and a value of "100" maps to 100% usage.
@@ -81,6 +106,7 @@ disabled and log an error message. An alternative can then be to use the <<syste
To enable and configure the CPU utilization based circuit breaker:
+.Per collection in `solrconfig.xml`
[source,xml]
----
<circuitBreaker class="org.apache.solr.util.circuitbreaker.CPUCircuitBreaker">
@@ -88,6 +114,12 @@ To enable and configure the CPU utilization based circuit breaker:
</circuitBreaker>
----
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_CPU=75
+----
+
The triggering threshold is defined in percent CPU usage. A value of "0" maps to 0% usage
and a value of "100" maps to 100% usage. The example above will trip when the CPU usage is
equal to or greater than 75%.
@@ -103,6 +135,7 @@ https://en.wikipedia.org/wiki/Load_(computing)[Wikipedia page for Load],
To enable and configure the CPU utilization based circuit breaker:
+.Per collection in `solrconfig.xml`
[source,xml]
----
<circuitBreaker class="org.apache.solr.util.circuitbreaker.LoadAverageCircuitBreaker">
@@ -110,6 +143,12 @@ To enable and configure the CPU utilization based circuit breaker:
</circuitBreaker>
----
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_LOADAVG=8.0
+----
+
The triggering threshold is a floating point number matching load average.
The example circuit breaker above will trip when the load average is equal to or greater than 8.0.
@@ -118,6 +157,7 @@ The example circuit breaker above will trip when the load average is equal to or
In this example we will prevent update requests above 80% CPU load, and prevent query requests above 95% CPU load. Supported request types are `query` and `update`.
This would prevent expensive bulk updates from impacting search. Note also the support for short-form class name.
+.Per collection in `solrconfig.xml`
[source,xml]
----
<config>
@@ -137,6 +177,13 @@ This would prevent expensive bulk updates from impacting search. Note also the s
</config>
----
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_UPDATE_CPU=80
+SOLR_CIRCUITBREAKER_QUERY_CPU=95
+----
+
== Performance Considerations
While JVM or CPU circuit breakers do not add any noticeable overhead per request, having too many circuit breakers checked for a single request can cause a performance overhead.
diff --git a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
index 886c2e58770..0b3060d16d7 100644
--- a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
+++ b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc
@@ -71,6 +71,9 @@ Due to changes in Lucene 9, that isn't possible any more.
=== Curator upgraded to 5.5.0 and requires Zookeeper 3.5.x or higher
<<#solr-8-2,Solr 8.2 recommended using Zookeeper 3.5.5>> and now with Curator 5.5.0 requires https://curator.apache.org/docs/breaking-changes/[Zookeeper 3.5.x or higher]. This primarily affects users of `hadoop-auth`, but usage of Curator could affect other parts of Solr.
+=== Global Circuit Breakers
+* Circuit breakers can now be configured globally, not only per collection. See xref:deployment-guide:circuit-breakers.adoc[Configuring Circuit Breakers] for more information.
+
== Solr 9.4
=== The Built-In Config Sets
* The build in ConfigSets (`_default` and `sample_techproducts_configs`), now use a default `autoSoftCommit` time of 3 seconds,