You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by at...@apache.org on 2020/10/16 09:56:22 UTC
[lucene-solr] branch branch_8x updated (713b90c -> 327d545)
This is an automated email from the ASF dual-hosted git repository.
atri pushed a change to branch branch_8x
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git.
from 713b90c SOLR-14930: Deprecate rulebased replica placement strategy (#1984)
new 4733110 SOLR-14588: Implement Circuit Breakers (#1626)
new c841f63 SOLR-14615: Implement CPU Utilization Based Circuit Breaker (#1737)
new 5882b42 Update Circuit Breaker configured as a standard plugin (#1785)
new 327d545 Fix NPE in CB Config Resolution
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
solr/CHANGES.txt | 106 +++++++
.../src/java/org/apache/solr/core/SolrConfig.java | 4 +
.../src/java/org/apache/solr/core/SolrCore.java | 17 +-
.../solr/handler/component/SearchHandler.java | 30 +-
.../util/circuitbreaker/CPUCircuitBreaker.java | 115 +++++++
.../solr/util/circuitbreaker/CircuitBreaker.java | 99 ++++++
.../util/circuitbreaker/CircuitBreakerManager.java | 182 +++++++++++
.../util/circuitbreaker/MemoryCircuitBreaker.java | 127 ++++++++
...le.xml => solrconfig-memory-circuitbreaker.xml} | 21 +-
.../org/apache/solr/util/TestCircuitBreaker.java | 332 +++++++++++++++++++++
solr/example/files/conf/solrconfig.xml | 56 ++++
.../solr/configsets/_default/conf/solrconfig.xml | 75 +++++
.../conf/solrconfig.xml | 42 +++
solr/solr-ref-guide/src/circuit-breakers.adoc | 109 +++++++
solr/solr-ref-guide/src/config-api.adoc | 7 +
solr/solr-ref-guide/src/index.adoc | 3 +
.../src/query-settings-in-solrconfig.adoc | 45 +++
17 files changed, 1358 insertions(+), 12 deletions(-)
create mode 100644 solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java
create mode 100644 solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
create mode 100644 solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
create mode 100644 solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
copy solr/core/src/test-files/solr/collection1/conf/{solrconfig-cache-enable-disable.xml => solrconfig-memory-circuitbreaker.xml} (88%)
create mode 100644 solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
create mode 100644 solr/solr-ref-guide/src/circuit-breakers.adoc
[lucene-solr] 04/04: Fix NPE in CB Config Resolution
Posted by at...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
atri pushed a commit to branch branch_8x
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit 327d54548dd30554b18f5afd0530cb04878fdf57
Author: Atri Sharma <at...@gmail.com>
AuthorDate: Thu Aug 27 20:52:12 2020 +0530
Fix NPE in CB Config Resolution
---
.../util/circuitbreaker/CircuitBreakerManager.java | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
index 975ca4b..2f6f93d 100644
--- a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
@@ -137,7 +137,8 @@ public class CircuitBreakerManager implements PluginInfoInitialized {
*/
@SuppressWarnings({"rawtypes"})
public static CircuitBreakerManager build(PluginInfo pluginInfo) {
- CircuitBreakerManager circuitBreakerManager = new CircuitBreakerManager(Boolean.parseBoolean(pluginInfo.attributes.get("enabled")));
+ boolean enabled = pluginInfo == null ? false : Boolean.parseBoolean(pluginInfo.attributes.getOrDefault("enabled", "false"));
+ CircuitBreakerManager circuitBreakerManager = new CircuitBreakerManager(enabled);
circuitBreakerManager.init(pluginInfo);
@@ -147,19 +148,24 @@ public class CircuitBreakerManager implements PluginInfoInitialized {
@VisibleForTesting
@SuppressWarnings({"rawtypes"})
public static CircuitBreaker.CircuitBreakerConfig buildCBConfig(PluginInfo pluginInfo) {
- boolean enabled = Boolean.parseBoolean(pluginInfo.attributes.get("enabled"));
+ boolean enabled = false;
boolean cpuCBEnabled = false;
boolean memCBEnabled = false;
int memCBThreshold = 100;
int cpuCBThreshold = 100;
- NamedList args = pluginInfo.initArgs;
- if (args != null) {
- cpuCBEnabled = args.getBooleanArg("cpuEnabled");
- memCBEnabled = args.getBooleanArg("memEnabled");
- memCBThreshold = Integer.parseInt((String) args.get("memThreshold"));
- cpuCBThreshold = Integer.parseInt((String) args.get("cpuThreshold"));
+ if (pluginInfo != null) {
+ NamedList args = pluginInfo.initArgs;
+
+ enabled = Boolean.parseBoolean(pluginInfo.attributes.getOrDefault("enabled", "false"));
+
+ if (args != null) {
+ cpuCBEnabled = Boolean.parseBoolean(args._getStr("cpuEnabled", "false"));
+ memCBEnabled = Boolean.parseBoolean(args._getStr("memEnabled", "false"));
+ memCBThreshold = Integer.parseInt(args._getStr("memThreshold", "100"));
+ cpuCBThreshold = Integer.parseInt(args._getStr("cpuThreshold", "100"));
+ }
}
return new CircuitBreaker.CircuitBreakerConfig(enabled, memCBEnabled, memCBThreshold, cpuCBEnabled, cpuCBThreshold);
[lucene-solr] 03/04: Update Circuit Breaker configured as a
standard plugin (#1785)
Posted by at...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
atri pushed a commit to branch branch_8x
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit 5882b42e2ecd1db3df97108d03fd578e7a86f358
Author: Atri Sharma <at...@gmail.com>
AuthorDate: Thu Aug 27 14:06:22 2020 +0530
Update Circuit Breaker configured as a standard plugin (#1785)
---
.../src/java/org/apache/solr/core/SolrConfig.java | 32 ++-----------
.../src/java/org/apache/solr/core/SolrCore.java | 3 +-
.../solr/handler/component/SearchHandler.java | 2 +-
.../util/circuitbreaker/CPUCircuitBreaker.java | 9 ++--
.../solr/util/circuitbreaker/CircuitBreaker.java | 47 +++++++++++++++---
.../util/circuitbreaker/CircuitBreakerManager.java | 55 ++++++++++++++++++----
.../util/circuitbreaker/MemoryCircuitBreaker.java | 9 ++--
.../resources/EditableSolrConfigAttributes.json | 11 -----
.../conf/solrconfig-memory-circuitbreaker.xml | 10 ++--
.../test/org/apache/solr/core/SolrCoreTest.java | 3 --
.../org/apache/solr/core/TestConfigOverlay.java | 2 -
.../org/apache/solr/util/TestCircuitBreaker.java | 44 +++++++++++------
solr/example/files/conf/solrconfig.xml | 24 ++++++++--
solr/solr-ref-guide/src/circuit-breakers.adoc | 37 +++++++++++----
.../src/query-settings-in-solrconfig.adoc | 37 ++++++++++++---
15 files changed, 216 insertions(+), 109 deletions(-)
diff --git a/solr/core/src/java/org/apache/solr/core/SolrConfig.java b/solr/core/src/java/org/apache/solr/core/SolrConfig.java
index 707d770..e23fb35 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrConfig.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrConfig.java
@@ -78,6 +78,7 @@ import org.apache.solr.update.UpdateLog;
import org.apache.solr.update.processor.UpdateRequestProcessorChain;
import org.apache.solr.update.processor.UpdateRequestProcessorFactory;
import org.apache.solr.util.DOMUtil;
+import org.apache.solr.util.circuitbreaker.CircuitBreakerManager;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.w3c.dom.Node;
@@ -227,14 +228,6 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
queryResultWindowSize = Math.max(1, getInt("query/queryResultWindowSize", 1));
queryResultMaxDocsCached = getInt("query/queryResultMaxDocsCached", Integer.MAX_VALUE);
enableLazyFieldLoading = getBool("query/enableLazyFieldLoading", false);
-
- useCircuitBreakers = getBool("circuitBreakers/@enabled", false);
- cpuCBEnabled = getBool("circuitBreakers/cpuBreaker/@enabled", false);
- memCBEnabled = getBool("circuitBreakers/memBreaker/@enabled", false);
- memCBThreshold = getInt("circuitBreakers/memBreaker/@threshold", 95);
- cpuCBThreshold = getInt("circuitBreakers/cpuBreaker/@threshold", 95);
-
- validateCircuitBreakerThresholds();
filterCacheConfig = CacheConfig.getConfig(this, "query/filterCache");
queryResultCacheConfig = CacheConfig.getConfig(this, "query/queryResultCache");
@@ -366,6 +359,7 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
.add(new SolrPluginInfo(IndexSchemaFactory.class, "schemaFactory", REQUIRE_CLASS))
.add(new SolrPluginInfo(RestManager.class, "restManager"))
.add(new SolrPluginInfo(StatsCache.class, "statsCache", REQUIRE_CLASS))
+ .add(new SolrPluginInfo(CircuitBreakerManager.class, "circuitBreaker"))
.build();
public static final Map<String, SolrPluginInfo> classVsSolrPluginInfo;
@@ -531,12 +525,6 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
public final int queryResultWindowSize;
public final int queryResultMaxDocsCached;
public final boolean enableLazyFieldLoading;
- // Circuit Breaker Configuration
- public final boolean useCircuitBreakers;
- public final int memCBThreshold;
- public final boolean memCBEnabled;
- public final boolean cpuCBEnabled;
- public final int cpuCBThreshold;
// IndexConfig settings
public final SolrIndexConfig indexConfig;
@@ -817,16 +805,6 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
loader.reloadLuceneSPI();
}
- private void validateCircuitBreakerThresholds() {
- if (useCircuitBreakers) {
- if (memCBEnabled) {
- if (memCBThreshold > 95 || memCBThreshold < 50) {
- throw new IllegalArgumentException("Valid value range of memoryCircuitBreakerThresholdPct is 50 - 95");
- }
- }
- }
- }
-
public int getMultipartUploadLimitKB() {
return multipartUploadLimitKB;
}
@@ -896,11 +874,7 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
m.put("queryResultMaxDocsCached", queryResultMaxDocsCached);
m.put("enableLazyFieldLoading", enableLazyFieldLoading);
m.put("maxBooleanClauses", booleanQueryMaxClauseCount);
- m.put("useCircuitBreakers", useCircuitBreakers);
- m.put("cpuCircuitBreakerEnabled", cpuCBEnabled);
- m.put("memoryCircuitBreakerEnabled", memCBEnabled);
- m.put("memoryCircuitBreakerThresholdPct", memCBThreshold);
- m.put("cpuCircuitBreakerThreshold", cpuCBThreshold);
+
for (SolrPluginInfo plugin : plugins) {
List<PluginInfo> infos = getPluginInfos(plugin.clazz.getName());
if (infos == null || infos.isEmpty()) continue;
diff --git a/solr/core/src/java/org/apache/solr/core/SolrCore.java b/solr/core/src/java/org/apache/solr/core/SolrCore.java
index 7013186..a623cc5 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrCore.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrCore.java
@@ -1179,7 +1179,8 @@ public final class SolrCore implements SolrInfoBean, SolrMetricProducer, Closeab
}
private CircuitBreakerManager initCircuitBreakerManager() {
- CircuitBreakerManager circuitBreakerManager = CircuitBreakerManager.build(solrConfig);
+ final PluginInfo info = solrConfig.getPluginInfo(CircuitBreakerManager.class.getName());
+ CircuitBreakerManager circuitBreakerManager = CircuitBreakerManager.build(info);
return circuitBreakerManager;
}
diff --git a/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java b/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
index 3934721..24cc706 100644
--- a/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
@@ -303,7 +303,7 @@ public class SearchHandler extends RequestHandlerBase implements SolrCoreAware,
final RTimerTree timer = rb.isDebug() ? req.getRequestTimer() : null;
- if (req.getCore().getSolrConfig().useCircuitBreakers) {
+ if (req.getCore().getCircuitBreakerManager().isEnabled()) {
List<CircuitBreaker> trippedCircuitBreakers;
if (timer != null) {
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java
index 45dc1e8..ffef4e5 100644
--- a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java
@@ -21,7 +21,6 @@ import java.lang.invoke.MethodHandles;
import java.lang.management.ManagementFactory;
import java.lang.management.OperatingSystemMXBean;
-import org.apache.solr.core.SolrConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -52,11 +51,11 @@ public class CPUCircuitBreaker extends CircuitBreaker {
private static final ThreadLocal<Double> allowedCPUUsage = ThreadLocal.withInitial(() -> 0.0);
- public CPUCircuitBreaker(SolrConfig solrConfig) {
- super(solrConfig);
+ public CPUCircuitBreaker(CircuitBreakerConfig config) {
+ super(config);
- this.enabled = solrConfig.cpuCBEnabled;
- this.cpuUsageThreshold = solrConfig.cpuCBThreshold;
+ this.enabled = config.getCpuCBEnabled();
+ this.cpuUsageThreshold = config.getCpuCBThreshold();
}
@Override
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
index 63ad7fd..90fda7a 100644
--- a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
@@ -17,8 +17,6 @@
package org.apache.solr.util.circuitbreaker;
-import org.apache.solr.core.SolrConfig;
-
/**
* Default class to define circuit breakers for Solr.
* <p>
@@ -35,16 +33,16 @@ import org.apache.solr.core.SolrConfig;
public abstract class CircuitBreaker {
public static final String NAME = "circuitbreaker";
- protected final SolrConfig solrConfig;
+ protected final CircuitBreakerConfig config;
- public CircuitBreaker(SolrConfig solrConfig) {
- this.solrConfig = solrConfig;
+ public CircuitBreaker(CircuitBreakerConfig config) {
+ this.config = config;
}
// Global config for all circuit breakers. For specific circuit breaker configs, define
// your own config.
protected boolean isEnabled() {
- return solrConfig.useCircuitBreakers;
+ return config.isEnabled();
}
/**
@@ -61,4 +59,41 @@ public abstract class CircuitBreaker {
* Get error message when the circuit breaker triggers
*/
public abstract String getErrorMessage();
+
+ public static class CircuitBreakerConfig {
+ private final boolean enabled;
+ private final boolean memCBEnabled;
+ private final int memCBThreshold;
+ private final boolean cpuCBEnabled;
+ private final int cpuCBThreshold;
+
+ public CircuitBreakerConfig(final boolean enabled, final boolean memCBEnabled, final int memCBThreshold,
+ final boolean cpuCBEnabled, final int cpuCBThreshold) {
+ this.enabled = enabled;
+ this.memCBEnabled = memCBEnabled;
+ this.memCBThreshold = memCBThreshold;
+ this.cpuCBEnabled = cpuCBEnabled;
+ this.cpuCBThreshold = cpuCBThreshold;
+ }
+
+ public boolean isEnabled() {
+ return enabled;
+ }
+
+ public boolean getMemCBEnabled() {
+ return memCBEnabled;
+ }
+
+ public int getMemCBThreshold() {
+ return memCBThreshold;
+ }
+
+ public boolean getCpuCBEnabled() {
+ return cpuCBEnabled;
+ }
+
+ public int getCpuCBThreshold() {
+ return cpuCBThreshold;
+ }
+ }
}
\ No newline at end of file
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
index ed7f62d..975ca4b 100644
--- a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
@@ -21,7 +21,9 @@ import java.util.ArrayList;
import java.util.List;
import com.google.common.annotations.VisibleForTesting;
-import org.apache.solr.core.SolrConfig;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.PluginInfo;
+import org.apache.solr.util.plugin.PluginInfoInitialized;
/**
* Manages all registered circuit breaker instances. Responsible for a holistic view
@@ -37,7 +39,7 @@ import org.apache.solr.core.SolrConfig;
* NOTE: The current way of registering new default circuit breakers is minimal and not a long term
* solution. There will be a follow up with a SIP for a schema API design.
*/
-public class CircuitBreakerManager {
+public class CircuitBreakerManager implements PluginInfoInitialized {
// Class private to potentially allow "family" of circuit breakers to be enabled or disabled
private final boolean enableCircuitBreakerManager;
@@ -47,6 +49,18 @@ public class CircuitBreakerManager {
this.enableCircuitBreakerManager = enableCircuitBreakerManager;
}
+ @Override
+ public void init(PluginInfo pluginInfo) {
+ CircuitBreaker.CircuitBreakerConfig circuitBreakerConfig = buildCBConfig(pluginInfo);
+
+ // Install the default circuit breakers
+ CircuitBreaker memoryCircuitBreaker = new MemoryCircuitBreaker(circuitBreakerConfig);
+ CircuitBreaker cpuCircuitBreaker = new CPUCircuitBreaker(circuitBreakerConfig);
+
+ register(memoryCircuitBreaker);
+ register(cpuCircuitBreaker);
+ }
+
public void register(CircuitBreaker circuitBreaker) {
circuitBreakerList.add(circuitBreaker);
}
@@ -121,20 +135,41 @@ public class CircuitBreakerManager {
*
* Any default circuit breakers should be registered here.
*/
- public static CircuitBreakerManager build(SolrConfig solrConfig) {
- CircuitBreakerManager circuitBreakerManager = new CircuitBreakerManager(solrConfig.useCircuitBreakers);
-
- // Install the default circuit breakers
- CircuitBreaker memoryCircuitBreaker = new MemoryCircuitBreaker(solrConfig);
- CircuitBreaker cpuCircuitBreaker = new CPUCircuitBreaker(solrConfig);
+ @SuppressWarnings({"rawtypes"})
+ public static CircuitBreakerManager build(PluginInfo pluginInfo) {
+ CircuitBreakerManager circuitBreakerManager = new CircuitBreakerManager(Boolean.parseBoolean(pluginInfo.attributes.get("enabled")));
- circuitBreakerManager.register(memoryCircuitBreaker);
- circuitBreakerManager.register(cpuCircuitBreaker);
+ circuitBreakerManager.init(pluginInfo);
return circuitBreakerManager;
}
@VisibleForTesting
+ @SuppressWarnings({"rawtypes"})
+ public static CircuitBreaker.CircuitBreakerConfig buildCBConfig(PluginInfo pluginInfo) {
+ boolean enabled = Boolean.parseBoolean(pluginInfo.attributes.get("enabled"));
+ boolean cpuCBEnabled = false;
+ boolean memCBEnabled = false;
+ int memCBThreshold = 100;
+ int cpuCBThreshold = 100;
+
+ NamedList args = pluginInfo.initArgs;
+
+ if (args != null) {
+ cpuCBEnabled = args.getBooleanArg("cpuEnabled");
+ memCBEnabled = args.getBooleanArg("memEnabled");
+ memCBThreshold = Integer.parseInt((String) args.get("memThreshold"));
+ cpuCBThreshold = Integer.parseInt((String) args.get("cpuThreshold"));
+ }
+
+ return new CircuitBreaker.CircuitBreakerConfig(enabled, memCBEnabled, memCBThreshold, cpuCBEnabled, cpuCBThreshold);
+ }
+
+ public boolean isEnabled() {
+ return enableCircuitBreakerManager;
+ }
+
+ @VisibleForTesting
public List<CircuitBreaker> getRegisteredCircuitBreakers() {
return circuitBreakerList;
}
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
index 797677d..0fc7b01 100644
--- a/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
@@ -21,7 +21,6 @@ import java.lang.invoke.MethodHandles;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
-import org.apache.solr.core.SolrConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -50,10 +49,10 @@ public class MemoryCircuitBreaker extends CircuitBreaker {
private static final ThreadLocal<Long> seenMemory = ThreadLocal.withInitial(() -> 0L);
private static final ThreadLocal<Long> allowedMemory = ThreadLocal.withInitial(() -> 0L);
- public MemoryCircuitBreaker(SolrConfig solrConfig) {
- super(solrConfig);
+ public MemoryCircuitBreaker(CircuitBreakerConfig config) {
+ super(config);
- this.enabled = solrConfig.memCBEnabled;
+ this.enabled = config.getMemCBEnabled();
long currentMaxHeap = MEMORY_MX_BEAN.getHeapMemoryUsage().getMax();
@@ -61,7 +60,7 @@ public class MemoryCircuitBreaker extends CircuitBreaker {
throw new IllegalArgumentException("Invalid JVM state for the max heap usage");
}
- int thresholdValueInPercentage = solrConfig.memCBThreshold;
+ int thresholdValueInPercentage = config.getMemCBThreshold();
double thresholdInFraction = thresholdValueInPercentage / (double) 100;
heapMemoryThreshold = (long) (currentMaxHeap * thresholdInFraction);
diff --git a/solr/core/src/resources/EditableSolrConfigAttributes.json b/solr/core/src/resources/EditableSolrConfigAttributes.json
index 1f49982..03bb1b6 100644
--- a/solr/core/src/resources/EditableSolrConfigAttributes.json
+++ b/solr/core/src/resources/EditableSolrConfigAttributes.json
@@ -55,17 +55,6 @@
"queryResultMaxDocsCached":1,
"enableLazyFieldLoading":1,
"boolTofilterOptimizer":1,
- "circuitBreakers":{
- "enabled":10,
- "memBreaker":{
- "enabled":10,
- "threshold":20
- },
- "cpuBreaker":{
- "enabled":10,
- "threshold":20
- }
- },
"maxBooleanClauses":1},
"jmx":{
"agentId":0,
diff --git a/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml b/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml
index d6adf92..699a7bd 100644
--- a/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml
+++ b/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml
@@ -78,10 +78,12 @@
</query>
- <circuitBreakers enabled="true">
- <cpuBreaker enabled="true" threshold="75"/>
- <memBreaker enabled="true" threshold="75"/>
- </circuitBreakers>
+ <circuitBreaker class="solr.CircuitBreakerManager" enabled="true">
+ <str name="memEnabled">true</str>
+ <str name="memThreshold">75</str>
+ <str name="cpuEnabled">true</str>
+ <str name="cpuThreshold">75</str>
+ </circuitBreaker>
<initParams path="/select">
<lst name="defaults">
diff --git a/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java b/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
index 342fa3e..b42a4aa 100644
--- a/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
+++ b/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
@@ -265,9 +265,6 @@ public class SolrCoreTest extends SolrTestCaseJ4 {
assertEquals("wrong config for maxBooleanClauses", 1024, solrConfig.booleanQueryMaxClauseCount);
assertEquals("wrong config for enableLazyFieldLoading", true, solrConfig.enableLazyFieldLoading);
assertEquals("wrong config for queryResultWindowSize", 10, solrConfig.queryResultWindowSize);
- assertEquals("wrong config for useCircuitBreakers", false, solrConfig.useCircuitBreakers);
- assertEquals("wrong config for memoryCircuitBreakerThresholdPct", 95, solrConfig.memCBThreshold);
- assertEquals("wrong config for cpuCircuitBreakerThreshold", 95, solrConfig.cpuCBThreshold);
}
/**
diff --git a/solr/core/src/test/org/apache/solr/core/TestConfigOverlay.java b/solr/core/src/test/org/apache/solr/core/TestConfigOverlay.java
index cfdc1e3..1c39fa9 100644
--- a/solr/core/src/test/org/apache/solr/core/TestConfigOverlay.java
+++ b/solr/core/src/test/org/apache/solr/core/TestConfigOverlay.java
@@ -46,8 +46,6 @@ public class TestConfigOverlay extends SolrTestCase {
assertTrue(isEditableProp("query.queryResultMaxDocsCached", false, null));
assertTrue(isEditableProp("query.enableLazyFieldLoading", false, null));
assertTrue(isEditableProp("query.boolTofilterOptimizer", false, null));
- assertTrue(isEditableProp("query.useCircuitBreakers", false, null));
- assertTrue(isEditableProp("query.memoryCircuitBreakerThresholdPct", false, null));
assertTrue(isEditableProp("jmx.agentId", false, null));
assertTrue(isEditableProp("jmx.serviceUrl", false, null));
assertTrue(isEditableProp("jmx.rootName", false, null));
diff --git a/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java b/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
index 83e8fc5..6ea7e3a 100644
--- a/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
+++ b/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
@@ -31,10 +31,11 @@ import org.apache.solr.common.SolrException;
import org.apache.solr.common.params.CommonParams;
import org.apache.solr.common.util.ExecutorUtil;
import org.apache.solr.common.util.SolrNamedThreadFactory;
-import org.apache.solr.core.SolrConfig;
+import org.apache.solr.core.PluginInfo;
import org.apache.solr.search.QueryParsing;
import org.apache.solr.util.circuitbreaker.CPUCircuitBreaker;
import org.apache.solr.util.circuitbreaker.CircuitBreaker;
+import org.apache.solr.util.circuitbreaker.CircuitBreakerManager;
import org.apache.solr.util.circuitbreaker.MemoryCircuitBreaker;
import org.junit.After;
import org.junit.BeforeClass;
@@ -89,7 +90,11 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
removeAllExistingCircuitBreakers();
- CircuitBreaker circuitBreaker = new MockCircuitBreaker(h.getCore().getSolrConfig());
+ PluginInfo pluginInfo = h.getCore().getSolrConfig().getPluginInfo(CircuitBreakerManager.class.getName());
+
+ CircuitBreaker.CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerManager.buildCBConfig(pluginInfo);
+
+ CircuitBreaker circuitBreaker = new MockCircuitBreaker(circuitBreakerConfig);
h.getCore().getCircuitBreakerManager().register(circuitBreaker);
@@ -106,7 +111,10 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
removeAllExistingCircuitBreakers();
- CircuitBreaker circuitBreaker = new FakeMemoryPressureCircuitBreaker(h.getCore().getSolrConfig());
+ PluginInfo pluginInfo = h.getCore().getSolrConfig().getPluginInfo(CircuitBreakerManager.class.getName());
+
+ CircuitBreaker.CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerManager.buildCBConfig(pluginInfo);
+ CircuitBreaker circuitBreaker = new FakeMemoryPressureCircuitBreaker(circuitBreakerConfig);
h.getCore().getCircuitBreakerManager().register(circuitBreaker);
@@ -128,7 +136,10 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
try {
removeAllExistingCircuitBreakers();
- CircuitBreaker circuitBreaker = new BuildingUpMemoryPressureCircuitBreaker(h.getCore().getSolrConfig());
+ PluginInfo pluginInfo = h.getCore().getSolrConfig().getPluginInfo(CircuitBreakerManager.class.getName());
+
+ CircuitBreaker.CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerManager.buildCBConfig(pluginInfo);
+ CircuitBreaker circuitBreaker = new BuildingUpMemoryPressureCircuitBreaker(circuitBreakerConfig);
h.getCore().getCircuitBreakerManager().register(circuitBreaker);
@@ -182,7 +193,10 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
try {
removeAllExistingCircuitBreakers();
- CircuitBreaker circuitBreaker = new FakeCPUCircuitBreaker(h.getCore().getSolrConfig());
+ PluginInfo pluginInfo = h.getCore().getSolrConfig().getPluginInfo(CircuitBreakerManager.class.getName());
+
+ CircuitBreaker.CircuitBreakerConfig circuitBreakerConfig = CircuitBreakerManager.buildCBConfig(pluginInfo);
+ CircuitBreaker circuitBreaker = new FakeCPUCircuitBreaker(circuitBreakerConfig);
h.getCore().getCircuitBreakerManager().register(circuitBreaker);
@@ -257,8 +271,8 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
private static class MockCircuitBreaker extends MemoryCircuitBreaker {
- public MockCircuitBreaker(SolrConfig solrConfig) {
- super(solrConfig);
+ public MockCircuitBreaker(CircuitBreakerConfig config) {
+ super(config);
}
@Override
@@ -275,8 +289,8 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
private static class FakeMemoryPressureCircuitBreaker extends MemoryCircuitBreaker {
- public FakeMemoryPressureCircuitBreaker(SolrConfig solrConfig) {
- super(solrConfig);
+ public FakeMemoryPressureCircuitBreaker(CircuitBreakerConfig config) {
+ super(config);
}
@Override
@@ -287,10 +301,12 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
}
private class BuildingUpMemoryPressureCircuitBreaker extends MemoryCircuitBreaker {
- private AtomicInteger count = new AtomicInteger();
+ private AtomicInteger count;
+
+ public BuildingUpMemoryPressureCircuitBreaker(CircuitBreakerConfig config) {
+ super(config);
- public BuildingUpMemoryPressureCircuitBreaker(SolrConfig solrConfig) {
- super(solrConfig);
+ this.count = new AtomicInteger(0);
}
@Override
@@ -304,8 +320,8 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
}
private static class FakeCPUCircuitBreaker extends CPUCircuitBreaker {
- public FakeCPUCircuitBreaker(SolrConfig solrConfig) {
- super(solrConfig);
+ public FakeCPUCircuitBreaker(CircuitBreakerConfig config) {
+ super(config);
}
@Override
diff --git a/solr/example/files/conf/solrconfig.xml b/solr/example/files/conf/solrconfig.xml
index a6f5a58..eea7241 100644
--- a/solr/example/files/conf/solrconfig.xml
+++ b/solr/example/files/conf/solrconfig.xml
@@ -564,7 +564,7 @@
Circuit Breaker Section - This section consists of configurations for
circuit breakers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
- <circuitBreaker>
+ <circuitBreaker class="solr.CircuitBreakerManager" enabled="true">
<!-- Enable Circuit Breakers
Circuit breakers are designed to allow stability and predictable query
@@ -577,7 +577,12 @@
they are free to add their specific configuration but need to ensure that this flag is always
respected - this should have veto over all independent configuration flags.
-->
- <useCircuitBreakers>false</useCircuitBreakers>
+
+ <!-- Memory Circuit Breaker Control Flag
+
+ Use the following flag to control the behaviour of this circuit breaker
+ -->
+ <str name="memEnabled">true</str>
<!-- Memory Circuit Breaker Threshold In Percentage
@@ -594,7 +599,20 @@
in logs and the corresponding error message should tell you what transpired (if the failure
was caused by tripped circuit breakers).
-->
- <memoryCircuitBreakerThresholdPct>100</memoryCircuitBreakerThresholdPct>
+ <str name="memThreshold">75</str>
+
+ <!-- CPU Based Circuit Breaker Control Flag
+
+ Use the following flag to control the behaviour of this circuit breaker
+ -->
+ <str name="cpuEnabled">true</str>
+
+ <!-- CPU Based Circuit Breaker Triggering Threshold
+
+ The triggering threshold is defined in units of CPU utilization. The configuration to control this is as below:
+ -->
+ <str name="cpuThreshold">75</str>
+
</circuitBreaker>
diff --git a/solr/solr-ref-guide/src/circuit-breakers.adoc b/solr/solr-ref-guide/src/circuit-breakers.adoc
index 56451b0..b28bda6 100644
--- a/solr/solr-ref-guide/src/circuit-breakers.adoc
+++ b/solr/solr-ref-guide/src/circuit-breakers.adoc
@@ -27,19 +27,24 @@ are enabled, requests may be rejected under the condition of high node duress wi
It is up to the client to handle this error and potentially build a retrial logic as this should ideally be a transient situation.
== Circuit Breaker Configurations
-The following flag controls the global activation/deactivation of circuit breakers. If this flag is disabled, all circuit breakers
-will be disabled globally. Per circuit breaker configurations are specified in their respective sections later.
+All circuit breaker configurations are listed in the circuitBreaker tags in solrconfig.xml as shown below:
[source,xml]
----
-<circuitBreakers enabled="true">
+<circuitBreaker class="solr.CircuitBreakerManager" enabled="true">
<!-- All specific configs in this section -->
-</circuitBreakers>
+</circuitBreaker>
----
-This flag acts as the highest authority and global controller of circuit breakers. For using specific circuit breakers, each one
+The "enabled" attribute controls the global activation/deactivation of circuit breakers. If this flag is disabled, all circuit breakers
+will be disabled globally. Per circuit breaker configurations are specified in their respective sections later.
+
+This attribute acts as the highest authority and global controller of circuit breakers. For using specific circuit breakers, each one
needs to be individually enabled in addition to this flag being enabled.
+CircuitBreakerManager is the default manager for all circuit breakers that should be defined in the tag unless the user wishes to use
+a custom implementation.
+
== Currently Supported Circuit Breakers
=== JVM Heap Usage Based Circuit Breaker
@@ -51,11 +56,18 @@ Configuration for JVM heap usage based circuit breaker:
[source,xml]
----
-<memBreaker enabled="true" threshold="75"/>
+<str name="memEnabled">true</str>
----
Note that this configuration will be overridden by the global circuit breaker flag -- if circuit breakers are disabled, this flag
-will not help you. Also, the triggering threshold is defined as a percentage of the max heap allocated to the JVM.
+will not help you.
+
+The triggering threshold is defined as a percentage of the max heap allocated to the JVM. It is controlled by the below configuration:
+
+[source,xml]
+----
+<str name="memThreshold">75</str>
+----
It does not logically make sense to have a threshold below 50% and above 95% of the max heap allocated to the JVM. Hence, the range
of valid values for this parameter is [50, 95], both inclusive.
@@ -76,11 +88,18 @@ Configuration for CPU utilization based circuit breaker:
[source,xml]
----
-<cpuBreaker enabled="true" threshold="20"/>
+<str name="cpuEnabled">true</str>
----
Note that this configuration will be overridden by the global circuit breaker flag -- if circuit breakers are disabled, this flag
-will not help you. The triggering threshold is defined in units of CPU utilization.
+will not help you.
+
+The triggering threshold is defined in units of CPU utilization. The configuration to control this is as below:
+
+[source,xml]
+----
+<str name="cpuThreshold">75</str>
+----
== Performance Considerations
It is worth noting that while JVM or CPU circuit breakers do not add any noticeable overhead per query, having too many
diff --git a/solr/solr-ref-guide/src/query-settings-in-solrconfig.adoc b/solr/solr-ref-guide/src/query-settings-in-solrconfig.adoc
index 9a336c3..68b66c02 100644
--- a/solr/solr-ref-guide/src/query-settings-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/query-settings-in-solrconfig.adoc
@@ -185,24 +185,49 @@ This parameter sets the maximum number of documents to cache for any entry in th
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
----
-=== useCircuitBreakers
+=== circuitBreaker
-Global control flag for enabling circuit breakers.
+This set of configurations control the behaviour of circuit breakers.
[source,xml]
----
-<useCircuitBreakers>true</useCircuitBreakers>
+<circuitBreaker class="solr.CircuitBreakerManager" enabled="true">
+ <!-- All specific configs in this section -->
+</circuitBreaker>
----
-=== memoryCircuitBreakerThresholdPct
+To control whether Circuit Breakers are globally enabled, use the "enabled" attribute.
+
+=== Memory Circuit Breaker Settings
+
+To turn memory circuit breaker on/off, use the following flag:
+[source,xml]
+----
+<str name="memEnabled">true</str>
+----
Memory threshold in percentage for JVM heap usage defined in percentage of maximum heap allocated
to the JVM (-Xmx). Ideally, this value should be in the range of 75-80% of maximum heap allocated
-to the JVM.
+to the JVM. The enabled flag can be used to control the specific toggle for this circuit breaker.
+
+[source,xml]
+----
+<str name="memThreshold">75</str>
+----
+
+=== CPU Circuit Breaker Settings
+
+To control turning on/off this feature, use the following flag:
+[source,xml]
+----
+<str name="cpuEnabled">true</str>
+----
+
+Defines the triggering threshold in terms of the average per minute CPU load. The enabled flag can be used to control the specific toggle for this circuit breaker.
[source,xml]
----
-<memoryCircuitBreakerThresholdPct>75</memoryCircuitBreakerThresholdPct>
+<str name="cpuThreshold">75</str>
----
=== useColdSearcher
[lucene-solr] 01/04: SOLR-14588: Implement Circuit Breakers (#1626)
Posted by at...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
atri pushed a commit to branch branch_8x
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit 473311099ad22f50bd5fec5aa72c13b608e4d0c3
Author: Atri Sharma <at...@gmail.com>
AuthorDate: Thu Jul 2 12:43:48 2020 +0530
SOLR-14588: Implement Circuit Breakers (#1626)
* SOLR-14588: Implement Circuit Breakers
This commit consists of two parts: add circuit breakers infrastructure and a "real" JVM heap memory based
circuit breaker which monitors incoming search requests and rejects them with SERVICE_TOO_BUSY error
if the defined threshold is breached, thus giving headroom to existing indexing and search requests
to complete.
---
solr/CHANGES.txt | 7 +-
.../src/java/org/apache/solr/core/SolrConfig.java | 19 ++
.../src/java/org/apache/solr/core/SolrCore.java | 16 +-
.../solr/handler/component/SearchHandler.java | 30 ++-
.../solr/util/circuitbreaker/CircuitBreaker.java | 56 ++++++
.../util/circuitbreaker/CircuitBreakerManager.java | 134 +++++++++++++
.../util/circuitbreaker/MemoryCircuitBreaker.java | 114 +++++++++++
.../resources/EditableSolrConfigAttributes.json | 2 +
.../conf/solrconfig-memory-circuitbreaker.xml | 95 +++++++++
.../test/org/apache/solr/core/SolrCoreTest.java | 2 +
.../org/apache/solr/core/TestConfigOverlay.java | 2 +
.../org/apache/solr/util/TestCircuitBreaker.java | 218 +++++++++++++++++++++
solr/example/files/conf/solrconfig.xml | 38 ++++
.../solr/configsets/_default/conf/solrconfig.xml | 59 ++++++
.../conf/solrconfig.xml | 42 ++++
solr/solr-ref-guide/src/circuit-breakers.adoc | 68 +++++++
solr/solr-ref-guide/src/config-api.adoc | 7 +
solr/solr-ref-guide/src/index.adoc | 3 +
.../src/query-settings-in-solrconfig.adoc | 20 ++
19 files changed, 929 insertions(+), 3 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 3064c33..5f34199 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -21,9 +21,14 @@ Upgrade Notes
New Features
---------------------
-
* SOLR-14151 Make schema components load from packages (noble)
+* SOLR-14588: Introduce Circuit Breaker Infrastructure and a JVM heap usage memory tracking circuit breaker implementation (Atri Sharma)
+
+Improvements
+----------------------
+* LUCENE-8984: MoreLikeThis MLT is biased for uncommon fields (Andy Hind via Anshum Gupta)
+
* SOLR-14681: Introduce ability to delete .jar stored in the Package Store. (MarcusSorealheis , Mike Drob)
* SOLR-14604: Add the ability to uninstall a package from with the Package CLI. (MarcusSorealheis)
diff --git a/solr/core/src/java/org/apache/solr/core/SolrConfig.java b/solr/core/src/java/org/apache/solr/core/SolrConfig.java
index a29fb51..ec81688 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrConfig.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrConfig.java
@@ -227,6 +227,11 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
queryResultWindowSize = Math.max(1, getInt("query/queryResultWindowSize", 1));
queryResultMaxDocsCached = getInt("query/queryResultMaxDocsCached", Integer.MAX_VALUE);
enableLazyFieldLoading = getBool("query/enableLazyFieldLoading", false);
+
+ useCircuitBreakers = getBool("circuitBreaker/useCircuitBreakers", false);
+ memoryCircuitBreakerThresholdPct = getInt("circuitBreaker/memoryCircuitBreakerThresholdPct", 95);
+
+ validateMemoryBreakerThreshold();
filterCacheConfig = CacheConfig.getConfig(this, "query/filterCache");
queryResultCacheConfig = CacheConfig.getConfig(this, "query/queryResultCache");
@@ -523,6 +528,10 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
public final int queryResultWindowSize;
public final int queryResultMaxDocsCached;
public final boolean enableLazyFieldLoading;
+ // Circuit Breaker Configuration
+ public final boolean useCircuitBreakers;
+ public final int memoryCircuitBreakerThresholdPct;
+
// IndexConfig settings
public final SolrIndexConfig indexConfig;
@@ -802,6 +811,14 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
loader.reloadLuceneSPI();
}
+ private void validateMemoryBreakerThreshold() {
+ if (useCircuitBreakers) {
+ if (memoryCircuitBreakerThresholdPct > 95 || memoryCircuitBreakerThresholdPct < 50) {
+ throw new IllegalArgumentException("Valid value range of memoryCircuitBreakerThresholdPct is 50 - 95");
+ }
+ }
+ }
+
public int getMultipartUploadLimitKB() {
return multipartUploadLimitKB;
}
@@ -871,6 +888,8 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
m.put("queryResultMaxDocsCached", queryResultMaxDocsCached);
m.put("enableLazyFieldLoading", enableLazyFieldLoading);
m.put("maxBooleanClauses", booleanQueryMaxClauseCount);
+ m.put("useCircuitBreakers", useCircuitBreakers);
+ m.put("memoryCircuitBreakerThresholdPct", memoryCircuitBreakerThresholdPct);
for (SolrPluginInfo plugin : plugins) {
List<PluginInfo> infos = getPluginInfos(plugin.clazz.getName());
if (infos == null || infos.isEmpty()) continue;
diff --git a/solr/core/src/java/org/apache/solr/core/SolrCore.java b/solr/core/src/java/org/apache/solr/core/SolrCore.java
index a7d7994..7013186 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrCore.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrCore.java
@@ -96,6 +96,7 @@ import org.apache.solr.common.util.IOUtils;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.ObjectReleaseTracker;
import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
import org.apache.solr.common.util.Utils;
import org.apache.solr.core.DirectoryFactory.DirContext;
import org.apache.solr.core.snapshots.SolrSnapshotManager;
@@ -158,13 +159,13 @@ import org.apache.solr.update.processor.RunUpdateProcessorFactory;
import org.apache.solr.update.processor.UpdateRequestProcessorChain;
import org.apache.solr.update.processor.UpdateRequestProcessorChain.ProcessorInfo;
import org.apache.solr.update.processor.UpdateRequestProcessorFactory;
-import org.apache.solr.common.util.SolrNamedThreadFactory;
import org.apache.solr.util.IOFunction;
import org.apache.solr.util.NumberUtils;
import org.apache.solr.util.PropertiesInputStream;
import org.apache.solr.util.PropertiesOutputStream;
import org.apache.solr.util.RefCounted;
import org.apache.solr.util.TestInjection;
+import org.apache.solr.util.circuitbreaker.CircuitBreakerManager;
import org.apache.solr.util.plugin.NamedListInitializedPlugin;
import org.apache.solr.util.plugin.PluginInfoInitialized;
import org.apache.solr.util.plugin.SolrCoreAware;
@@ -229,6 +230,8 @@ public final class SolrCore implements SolrInfoBean, SolrMetricProducer, Closeab
private final MemClassLoader memClassLoader;
//singleton listener for all packages used in schema
+ private final CircuitBreakerManager circuitBreakerManager;
+
private final List<Runnable> confListeners = new CopyOnWriteArrayList<>();
private final ReentrantLock ruleExpiryLock;
@@ -946,6 +949,7 @@ public final class SolrCore implements SolrInfoBean, SolrMetricProducer, Closeab
this.configSetProperties = configSet.getProperties();
// Initialize the metrics manager
this.coreMetricManager = initCoreMetricManager(solrConfig);
+ this.circuitBreakerManager = initCircuitBreakerManager();
solrMetricsContext = coreMetricManager.getSolrMetricsContext();
this.coreMetricManager.loadReporters();
@@ -1174,6 +1178,12 @@ public final class SolrCore implements SolrInfoBean, SolrMetricProducer, Closeab
return coreMetricManager;
}
+ private CircuitBreakerManager initCircuitBreakerManager() {
+ CircuitBreakerManager circuitBreakerManager = CircuitBreakerManager.build(solrConfig);
+
+ return circuitBreakerManager;
+ }
+
@Override
public void initializeMetrics(SolrMetricsContext parentContext, String scope) {
newSearcherCounter = parentContext.counter(this, "new", Category.SEARCHER.toString());
@@ -1508,6 +1518,10 @@ public final class SolrCore implements SolrInfoBean, SolrMetricProducer, Closeab
return updateProcessors;
}
+ public CircuitBreakerManager getCircuitBreakerManager() {
+ return circuitBreakerManager;
+ }
+
// this core current usage count
private final AtomicInteger refCount = new AtomicInteger(1);
diff --git a/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java b/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
index f2087f3..3934721 100644
--- a/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
@@ -53,13 +53,17 @@ import org.apache.solr.security.AuthorizationContext;
import org.apache.solr.security.PermissionNameProvider;
import org.apache.solr.util.RTimerTree;
import org.apache.solr.util.SolrPluginUtils;
+import org.apache.solr.util.circuitbreaker.CircuitBreaker;
+import org.apache.solr.util.circuitbreaker.CircuitBreakerManager;
import org.apache.solr.util.plugin.PluginInfoInitialized;
import org.apache.solr.util.plugin.SolrCoreAware;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import static org.apache.solr.common.params.CommonParams.DISTRIB;
+import static org.apache.solr.common.params.CommonParams.FAILURE;
import static org.apache.solr.common.params.CommonParams.PATH;
+import static org.apache.solr.common.params.CommonParams.STATUS;
/**
@@ -299,6 +303,30 @@ public class SearchHandler extends RequestHandlerBase implements SolrCoreAware,
final RTimerTree timer = rb.isDebug() ? req.getRequestTimer() : null;
+ if (req.getCore().getSolrConfig().useCircuitBreakers) {
+ List<CircuitBreaker> trippedCircuitBreakers;
+
+ if (timer != null) {
+ RTimerTree subt = timer.sub("circuitbreaker");
+ rb.setTimer(subt);
+
+ CircuitBreakerManager circuitBreakerManager = req.getCore().getCircuitBreakerManager();
+ trippedCircuitBreakers = circuitBreakerManager.checkTripped();
+
+ rb.getTimer().stop();
+ } else {
+ CircuitBreakerManager circuitBreakerManager = req.getCore().getCircuitBreakerManager();
+ trippedCircuitBreakers = circuitBreakerManager.checkTripped();
+ }
+
+ if (trippedCircuitBreakers != null) {
+ String errorMessage = CircuitBreakerManager.toErrorMessage(trippedCircuitBreakers);
+ rsp.add(STATUS, FAILURE);
+ rsp.setException(new SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Circuit Breakers tripped " + errorMessage));
+ return;
+ }
+ }
+
final ShardHandler shardHandler1 = getAndPrepShardHandler(req, rb); // creates a ShardHandler object only if it's needed
tagRequestWithRequestId(rb);
@@ -312,7 +340,7 @@ public class SearchHandler extends RequestHandlerBase implements SolrCoreAware,
// debugging prepare phase
RTimerTree subt = timer.sub( "prepare" );
for( SearchComponent c : components ) {
- rb.setTimer( subt.sub( c.getName() ) );
+ rb.setTimer(subt.sub( c.getName() ) );
c.prepare(rb);
rb.getTimer().stop();
}
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
new file mode 100644
index 0000000..f56f81e
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.util.circuitbreaker;
+
+import org.apache.solr.core.SolrConfig;
+
+/**
+ * Default class to define circuit breakers for Solr.
+ * <p>
+ * There are two (typical) ways to use circuit breakers:
+ * 1. Have them checked at admission control by default (use CircuitBreakerManager for the same).
+ * 2. Use the circuit breaker in a specific code path(s).
+ *
+ * TODO: This class should be grown as the scope of circuit breakers grow.
+ * </p>
+ */
+public abstract class CircuitBreaker {
+ public static final String NAME = "circuitbreaker";
+
+ protected final SolrConfig solrConfig;
+
+ public CircuitBreaker(SolrConfig solrConfig) {
+ this.solrConfig = solrConfig;
+ }
+
+ // Global config for all circuit breakers. For specific circuit breaker configs, define
+ // your own config.
+ protected boolean isEnabled() {
+ return solrConfig.useCircuitBreakers;
+ }
+
+ /**
+ * Check if circuit breaker is tripped.
+ */
+ public abstract boolean isTripped();
+
+ /**
+ * Get debug useful info.
+ */
+ public abstract String getDebugInfo();
+}
\ No newline at end of file
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
new file mode 100644
index 0000000..584b933
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.util.circuitbreaker;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.solr.core.SolrConfig;
+
+/**
+ * Manages all registered circuit breaker instances. Responsible for a holistic view
+ * of whether a circuit breaker has tripped or not.
+ *
+ * There are two typical ways of using this class's instance:
+ * 1. Check if any circuit breaker has triggered -- and know which circuit breaker has triggered.
+ * 2. Get an instance of a specific circuit breaker and perform checks.
+ *
+ * It is a good practice to register new circuit breakers here if you want them checked for every
+ * request.
+ *
+ * NOTE: The current way of registering new default circuit breakers is minimal and not a long term
+ * solution. There will be a follow up with a SIP for a schema API design.
+ */
+public class CircuitBreakerManager {
+ // Class private to potentially allow "family" of circuit breakers to be enabled or disabled
+ private final boolean enableCircuitBreakerManager;
+
+ private final List<CircuitBreaker> circuitBreakerList = new ArrayList<>();
+
+ public CircuitBreakerManager(final boolean enableCircuitBreakerManager) {
+ this.enableCircuitBreakerManager = enableCircuitBreakerManager;
+ }
+
+ public void register(CircuitBreaker circuitBreaker) {
+ circuitBreakerList.add(circuitBreaker);
+ }
+
+ public void deregisterAll() {
+ circuitBreakerList.clear();
+ }
+ /**
+ * Check and return circuit breakers that have triggered
+ * @return CircuitBreakers which have triggered, null otherwise.
+ */
+ public List<CircuitBreaker> checkTripped() {
+ List<CircuitBreaker> triggeredCircuitBreakers = null;
+
+ if (enableCircuitBreakerManager) {
+ for (CircuitBreaker circuitBreaker : circuitBreakerList) {
+ if (circuitBreaker.isEnabled() &&
+ circuitBreaker.isTripped()) {
+ if (triggeredCircuitBreakers == null) {
+ triggeredCircuitBreakers = new ArrayList<>();
+ }
+
+ triggeredCircuitBreakers.add(circuitBreaker);
+ }
+ }
+ }
+
+ return triggeredCircuitBreakers;
+ }
+
+ /**
+ * Returns true if *any* circuit breaker has triggered, false if none have triggered.
+ *
+ * <p>
+ * NOTE: This method short circuits the checking of circuit breakers -- the method will
+ * return as soon as it finds a circuit breaker that is enabled and has triggered.
+ * </p>
+ */
+ public boolean checkAnyTripped() {
+ if (enableCircuitBreakerManager) {
+ for (CircuitBreaker circuitBreaker : circuitBreakerList) {
+ if (circuitBreaker.isEnabled() &&
+ circuitBreaker.isTripped()) {
+ return true;
+ }
+ }
+ }
+
+ return false;
+ }
+
+ /**
+ * Construct the final error message to be printed when circuit breakers trip.
+ *
+ * @param circuitBreakerList Input list for circuit breakers.
+ * @return Constructed error message.
+ */
+ public static String toErrorMessage(List<CircuitBreaker> circuitBreakerList) {
+ StringBuilder sb = new StringBuilder();
+
+ for (CircuitBreaker circuitBreaker : circuitBreakerList) {
+ sb.append(circuitBreaker.getClass().getName());
+ sb.append(" ");
+ sb.append(circuitBreaker.getDebugInfo());
+ sb.append("\n");
+ }
+
+ return sb.toString();
+ }
+
+ /**
+ * Register default circuit breakers and return a constructed CircuitBreakerManager
+ * instance which serves the given circuit breakers.
+ *
+ * Any default circuit breakers should be registered here.
+ */
+ public static CircuitBreakerManager build(SolrConfig solrConfig) {
+ CircuitBreakerManager circuitBreakerManager = new CircuitBreakerManager(solrConfig.useCircuitBreakers);
+
+ // Install the default circuit breakers
+ CircuitBreaker memoryCircuitBreaker = new MemoryCircuitBreaker(solrConfig);
+ circuitBreakerManager.register(memoryCircuitBreaker);
+
+ return circuitBreakerManager;
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
new file mode 100644
index 0000000..629d84a
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.util.circuitbreaker;
+
+import java.lang.invoke.MethodHandles;
+import java.lang.management.ManagementFactory;
+import java.lang.management.MemoryMXBean;
+
+import org.apache.solr.core.SolrConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * <p>
+ * Tracks the current JVM heap usage and triggers if it exceeds the defined percentage of the maximum
+ * heap size allocated to the JVM. This circuit breaker is a part of the default CircuitBreakerManager
+ * so is checked for every request -- hence it is realtime. Once the memory usage goes below the threshold,
+ * it will start allowing queries again.
+ * </p>
+ *
+ * <p>
+ * The memory threshold is defined as a percentage of the maximum memory allocated -- see memoryCircuitBreakerThresholdPct
+ * in solrconfig.xml.
+ * </p>
+ */
+
+public class MemoryCircuitBreaker extends CircuitBreaker {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final MemoryMXBean MEMORY_MX_BEAN = ManagementFactory.getMemoryMXBean();
+
+ private final long heapMemoryThreshold;
+
+ // Assumption -- the value of these parameters will be set correctly before invoking getDebugInfo()
+ private final ThreadLocal<Long> seenMemory = new ThreadLocal<>();
+ private final ThreadLocal<Long> allowedMemory = new ThreadLocal<>();
+
+ public MemoryCircuitBreaker(SolrConfig solrConfig) {
+ super(solrConfig);
+
+ long currentMaxHeap = MEMORY_MX_BEAN.getHeapMemoryUsage().getMax();
+
+ if (currentMaxHeap <= 0) {
+ throw new IllegalArgumentException("Invalid JVM state for the max heap usage");
+ }
+
+ int thresholdValueInPercentage = solrConfig.memoryCircuitBreakerThresholdPct;
+ double thresholdInFraction = thresholdValueInPercentage / (double) 100;
+ heapMemoryThreshold = (long) (currentMaxHeap * thresholdInFraction);
+
+ if (heapMemoryThreshold <= 0) {
+ throw new IllegalStateException("Memory limit cannot be less than or equal to zero");
+ }
+ }
+
+ // TODO: An optimization can be to trip the circuit breaker for a duration of time
+ // after the circuit breaker condition is matched. This will optimize for per call
+ // overhead of calculating the condition parameters but can result in false positives.
+ @Override
+ public boolean isTripped() {
+ if (!isEnabled()) {
+ return false;
+ }
+
+ long localAllowedMemory = getCurrentMemoryThreshold();
+ long localSeenMemory = calculateLiveMemoryUsage();
+
+ allowedMemory.set(localAllowedMemory);
+
+ seenMemory.set(localSeenMemory);
+
+ return (localSeenMemory >= localAllowedMemory);
+ }
+
+ @Override
+ public String getDebugInfo() {
+ if (seenMemory.get() == 0L || allowedMemory.get() == 0L) {
+ log.warn("MemoryCircuitBreaker's monitored values (seenMemory, allowedMemory) not set");
+ }
+
+ return "seenMemory=" + seenMemory.get() + " allowedMemory=" + allowedMemory.get();
+ }
+
+ private long getCurrentMemoryThreshold() {
+ return heapMemoryThreshold;
+ }
+
+ /**
+ * Calculate the live memory usage for the system. This method has package visibility
+ * to allow using for testing.
+ * @return Memory usage in bytes.
+ */
+ protected long calculateLiveMemoryUsage() {
+ // NOTE: MemoryUsageGaugeSet provides memory usage statistics but we do not use them
+ // here since it will require extra allocations and incur cost, hence it is cheaper to use
+ // MemoryMXBean directly. Ideally, this call should not add noticeable
+ // latency to a query -- but if it does, please signify on SOLR-14588
+ return MEMORY_MX_BEAN.getHeapMemoryUsage().getUsed();
+ }
+}
diff --git a/solr/core/src/resources/EditableSolrConfigAttributes.json b/solr/core/src/resources/EditableSolrConfigAttributes.json
index 03bb1b6..0ed8333 100644
--- a/solr/core/src/resources/EditableSolrConfigAttributes.json
+++ b/solr/core/src/resources/EditableSolrConfigAttributes.json
@@ -55,6 +55,8 @@
"queryResultMaxDocsCached":1,
"enableLazyFieldLoading":1,
"boolTofilterOptimizer":1,
+ "useCircuitBreakers":10,
+ "memoryCircuitBreakerThresholdPct":20,
"maxBooleanClauses":1},
"jmx":{
"agentId":0,
diff --git a/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml b/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml
new file mode 100644
index 0000000..b6b20ff
--- /dev/null
+++ b/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml
@@ -0,0 +1,95 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<config>
+ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
+ <dataDir>${solr.data.dir:}</dataDir>
+ <xi:include href="solrconfig.snippet.randomindexconfig.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
+ <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.RAMDirectoryFactory}"/>
+ <schemaFactory class="ClassicIndexSchemaFactory"/>
+ <requestHandler name="/select" class="solr.SearchHandler" />
+
+ <query>
+ <!-- Maximum number of clauses in a boolean query... can affect
+ range or wildcard queries that expand to big boolean
+ queries. An exception is thrown if exceeded.
+ -->
+ <maxBooleanClauses>${solr.max.booleanClauses:1024}</maxBooleanClauses>
+
+ <!-- Cache specification for Filters or DocSets - unordered set of *all* documents
+ that match a particular query.
+ -->
+ <filterCache
+ enabled="${filterCache.enabled}"
+ size="512"
+ initialSize="512"
+ autowarmCount="2"/>
+
+ <queryResultCache
+ enabled="${queryResultCache.enabled}"
+ size="512"
+ initialSize="512"
+ autowarmCount="2"/>
+
+ <documentCache
+ enabled="${documentCache.enabled}"
+ size="512"
+ initialSize="512"
+ autowarmCount="0"/>
+
+ <cache
+ name="user_definied_cache_XXX"
+ enabled="${user_definied_cache_XXX.enabled:false}"
+ />
+ <cache
+ name="user_definied_cache_ZZZ"
+ enabled="${user_definied_cache_ZZZ.enabled:false}"
+ />
+
+
+
+ <!-- If true, stored fields that are not requested will be loaded lazily.
+ -->
+ <enableLazyFieldLoading>true</enableLazyFieldLoading>
+
+ <queryResultWindowSize>10</queryResultWindowSize>
+
+ <!-- boolToFilterOptimizer converts boolean clauses with zero boost
+ into cached filters if the number of docs selected by the clause exceeds
+ the threshold (represented as a fraction of the total index)
+ -->
+ <boolTofilterOptimizer enabled="false" cacheSize="32" threshold=".05"/>
+
+ </query>
+
+ <circuitBreaker>
+
+ <useCircuitBreakers>true</useCircuitBreakers>
+
+ <memoryCircuitBreakerThresholdPct>75</memoryCircuitBreakerThresholdPct>
+
+ </circuitBreaker>
+
+ <initParams path="/select">
+ <lst name="defaults">
+ <str name="df">text</str>
+ </lst>
+ </initParams>
+
+</config>
diff --git a/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java b/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
index b42a4aa..dcf593f 100644
--- a/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
+++ b/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
@@ -265,6 +265,8 @@ public class SolrCoreTest extends SolrTestCaseJ4 {
assertEquals("wrong config for maxBooleanClauses", 1024, solrConfig.booleanQueryMaxClauseCount);
assertEquals("wrong config for enableLazyFieldLoading", true, solrConfig.enableLazyFieldLoading);
assertEquals("wrong config for queryResultWindowSize", 10, solrConfig.queryResultWindowSize);
+ assertEquals("wrong config for useCircuitBreakers", false, solrConfig.useCircuitBreakers);
+ assertEquals("wrong config for memoryCircuitBreakerThresholdPct", 95, solrConfig.memoryCircuitBreakerThresholdPct);
}
/**
diff --git a/solr/core/src/test/org/apache/solr/core/TestConfigOverlay.java b/solr/core/src/test/org/apache/solr/core/TestConfigOverlay.java
index 1c39fa9..cfdc1e3 100644
--- a/solr/core/src/test/org/apache/solr/core/TestConfigOverlay.java
+++ b/solr/core/src/test/org/apache/solr/core/TestConfigOverlay.java
@@ -46,6 +46,8 @@ public class TestConfigOverlay extends SolrTestCase {
assertTrue(isEditableProp("query.queryResultMaxDocsCached", false, null));
assertTrue(isEditableProp("query.enableLazyFieldLoading", false, null));
assertTrue(isEditableProp("query.boolTofilterOptimizer", false, null));
+ assertTrue(isEditableProp("query.useCircuitBreakers", false, null));
+ assertTrue(isEditableProp("query.memoryCircuitBreakerThresholdPct", false, null));
assertTrue(isEditableProp("jmx.agentId", false, null));
assertTrue(isEditableProp("jmx.serviceUrl", false, null));
assertTrue(isEditableProp("jmx.rootName", false, null));
diff --git a/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java b/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
new file mode 100644
index 0000000..00b8d1a
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.util;
+
+import java.util.HashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.util.ExecutorUtil;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.core.SolrConfig;
+import org.apache.solr.search.QueryParsing;
+import org.apache.solr.util.circuitbreaker.CircuitBreaker;
+import org.apache.solr.util.circuitbreaker.MemoryCircuitBreaker;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestRule;
+
+public class TestCircuitBreaker extends SolrTestCaseJ4 {
+ private final static int NUM_DOCS = 20;
+
+ @Rule
+ public TestRule solrTestRules = RuleChain.outerRule(new SystemPropertiesRestoreRule());
+
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ System.setProperty("filterCache.enabled", "false");
+ System.setProperty("queryResultCache.enabled", "false");
+ System.setProperty("documentCache.enabled", "true");
+
+ initCore("solrconfig-memory-circuitbreaker.xml", "schema.xml");
+ for (int i = 0 ; i < NUM_DOCS ; i ++) {
+ assertU(adoc("name", "john smith", "id", "1"));
+ assertU(adoc("name", "johathon smith", "id", "2"));
+ assertU(adoc("name", "john percival smith", "id", "3"));
+ assertU(adoc("id", "1", "title", "this is a title.", "inStock_b1", "true"));
+ assertU(adoc("id", "2", "title", "this is another title.", "inStock_b1", "true"));
+ assertU(adoc("id", "3", "title", "Mary had a little lamb.", "inStock_b1", "false"));
+
+ //commit inside the loop to get multiple segments to make search as realistic as possible
+ assertU(commit());
+ }
+ }
+
+ @Override
+ public void tearDown() throws Exception {
+ super.tearDown();
+ }
+
+ @After
+ public void after() {
+ h.getCore().getCircuitBreakerManager().deregisterAll();
+ }
+
+ public void testCBAlwaysTrips() {
+ HashMap<String, String> args = new HashMap<String, String>();
+
+ args.put(QueryParsing.DEFTYPE, CircuitBreaker.NAME);
+ args.put(CommonParams.FL, "id");
+
+ CircuitBreaker circuitBreaker = new MockCircuitBreaker(h.getCore().getSolrConfig());
+
+ h.getCore().getCircuitBreakerManager().register(circuitBreaker);
+
+ expectThrows(SolrException.class, () -> {
+ h.query(req("name:\"john smith\""));
+ });
+ }
+
+ public void testCBFakeMemoryPressure() {
+ HashMap<String, String> args = new HashMap<String, String>();
+
+ args.put(QueryParsing.DEFTYPE, CircuitBreaker.NAME);
+ args.put(CommonParams.FL, "id");
+
+ CircuitBreaker circuitBreaker = new FakeMemoryPressureCircuitBreaker(h.getCore().getSolrConfig());
+
+ h.getCore().getCircuitBreakerManager().register(circuitBreaker);
+
+ expectThrows(SolrException.class, () -> {
+ h.query(req("name:\"john smith\""));
+ });
+ }
+
+ public void testBuildingMemoryPressure() {
+ ExecutorService executor = ExecutorUtil.newMDCAwareCachedThreadPool(
+ new SolrNamedThreadFactory("TestCircuitBreaker"));
+ HashMap<String, String> args = new HashMap<String, String>();
+
+ args.put(QueryParsing.DEFTYPE, CircuitBreaker.NAME);
+ args.put(CommonParams.FL, "id");
+
+ AtomicInteger failureCount = new AtomicInteger();
+
+ try {
+ CircuitBreaker circuitBreaker = new BuildingUpMemoryPressureCircuitBreaker(h.getCore().getSolrConfig());
+
+ h.getCore().getCircuitBreakerManager().register(circuitBreaker);
+
+ for (int i = 0; i < 5; i++) {
+ executor.submit(() -> {
+ try {
+ h.query(req("name:\"john smith\""));
+ } catch (SolrException e) {
+ failureCount.incrementAndGet();
+ } catch (Exception e) {
+ throw new RuntimeException(e.getMessage());
+ }
+ });
+ }
+
+ executor.shutdown();
+ try {
+ executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e.getMessage());
+ }
+
+ assertEquals("Number of failed queries is not correct", 1, failureCount.get());
+ } finally {
+ if (!executor.isShutdown()) {
+ executor.shutdown();
+ }
+ }
+ }
+
+ public void testResponseWithCBTiming() {
+ assertQ(req("q", "*:*", CommonParams.DEBUG_QUERY, "true"),
+ "//str[@name='rawquerystring']='*:*'",
+ "//str[@name='querystring']='*:*'",
+ "//str[@name='parsedquery']='MatchAllDocsQuery(*:*)'",
+ "//str[@name='parsedquery_toString']='*:*'",
+ "count(//lst[@name='explain']/*)=3",
+ "//lst[@name='explain']/str[@name='1']",
+ "//lst[@name='explain']/str[@name='2']",
+ "//lst[@name='explain']/str[@name='3']",
+ "//str[@name='QParser']",
+ "count(//lst[@name='timing']/*)=4",
+ "//lst[@name='timing']/double[@name='time']",
+ "count(//lst[@name='circuitbreaker']/*)>0",
+ "//lst[@name='circuitbreaker']/double[@name='time']",
+ "count(//lst[@name='prepare']/*)>0",
+ "//lst[@name='prepare']/double[@name='time']",
+ "count(//lst[@name='process']/*)>0",
+ "//lst[@name='process']/double[@name='time']"
+ );
+ }
+
+ private class MockCircuitBreaker extends CircuitBreaker {
+
+ public MockCircuitBreaker(SolrConfig solrConfig) {
+ super(solrConfig);
+ }
+
+ @Override
+ public boolean isTripped() {
+ // Always return true
+ return true;
+ }
+
+ @Override
+ public String getDebugInfo() {
+ return "MockCircuitBreaker";
+ }
+ }
+
+ private class FakeMemoryPressureCircuitBreaker extends MemoryCircuitBreaker {
+
+ public FakeMemoryPressureCircuitBreaker(SolrConfig solrConfig) {
+ super(solrConfig);
+ }
+
+ @Override
+ protected long calculateLiveMemoryUsage() {
+ // Return a number large enough to trigger a pushback from the circuit breaker
+ return Long.MAX_VALUE;
+ }
+ }
+
+ private class BuildingUpMemoryPressureCircuitBreaker extends MemoryCircuitBreaker {
+ private AtomicInteger count = new AtomicInteger();
+
+ public BuildingUpMemoryPressureCircuitBreaker(SolrConfig solrConfig) {
+ super(solrConfig);
+ }
+
+ @Override
+ protected long calculateLiveMemoryUsage() {
+ if (count.getAndIncrement() >= 4) {
+ return Long.MAX_VALUE;
+ }
+
+ return 5; // Random number guaranteed to not trip the circuit breaker
+ }
+ }
+}
diff --git a/solr/example/files/conf/solrconfig.xml b/solr/example/files/conf/solrconfig.xml
index 964ea81..a6f5a58 100644
--- a/solr/example/files/conf/solrconfig.xml
+++ b/solr/example/files/conf/solrconfig.xml
@@ -560,6 +560,44 @@
</query>
+ <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ Circuit Breaker Section - This section consists of configurations for
+ circuit breakers
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
+ <circuitBreaker>
+ <!-- Enable Circuit Breakers
+
+ Circuit breakers are designed to allow stability and predictable query
+ execution. They prevent operations that can take down the node and cause
+ noisy neighbour issues.
+
+ This flag is the uber control switch which controls the activation/deactivation of all circuit
+ breakers. At the moment, the only circuit breaker (max JVM circuit breaker) does not have its
+ own specific configuration. However, if a circuit breaker wishes to be independently configurable,
+ they are free to add their specific configuration but need to ensure that this flag is always
+ respected - this should have veto over all independent configuration flags.
+ -->
+ <useCircuitBreakers>false</useCircuitBreakers>
+
+ <!-- Memory Circuit Breaker Threshold In Percentage
+
+ Specific configuration for max JVM heap usage circuit breaker. This configuration defines the
+ threshold percentage of maximum heap allocated beyond which queries will be rejected until the
+ current JVM usage goes below the threshold. The valid value range for this parameter is 50 - 95.
+
+ Consider a scenario where the max heap allocated is 4 GB and memoryCircuitBreakerThresholdPct is
+ defined as 75. Threshold JVM usage will be 4 * 0.75 = 3 GB. Its generally a good idea to keep this value between 75 - 80% of maximum heap
+ allocated.
+
+ If, at any point, the current JVM heap usage goes above 3 GB, queries will be rejected until the heap usage goes below 3 GB again.
+ If you see queries getting rejected with 503 error code, check for "Circuit Breakers tripped"
+ in logs and the corresponding error message should tell you what transpired (if the failure
+ was caused by tripped circuit breakers).
+ -->
+ <memoryCircuitBreakerThresholdPct>100</memoryCircuitBreakerThresholdPct>
+
+ </circuitBreaker>
+
<!-- Request Dispatcher
diff --git a/solr/server/solr/configsets/_default/conf/solrconfig.xml b/solr/server/solr/configsets/_default/conf/solrconfig.xml
index ab7e41f..cdd56cf 100644
--- a/solr/server/solr/configsets/_default/conf/solrconfig.xml
+++ b/solr/server/solr/configsets/_default/conf/solrconfig.xml
@@ -529,6 +529,23 @@
-->
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
+ <!-- Use Filter For Sorted Query
+
+ A possible optimization that attempts to use a filter to
+ satisfy a search. If the requested sort does not include
+ score, then the filterCache will be checked for a filter
+ matching the query. If found, the filter will be used as the
+ source of document ids, and then the sort will be applied to
+ that.
+
+ For most situations, this will not be useful unless you
+ frequently get the same search repeatedly with different sort
+ options, and none of them ever use "score"
+-->
+ <!--
+ <useFilterForSortedQuery>true</useFilterForSortedQuery>
+ -->
+
<!-- Query Related Event Listeners
Various IndexSearcher related events can trigger Listeners to
@@ -577,6 +594,48 @@
</query>
+ <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ Circuit Breaker Section - This section consists of configurations for
+ circuit breakers
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
+ <circuitBreaker>
+ <!-- Enable Circuit Breakers
+
+ Circuit breakers are designed to allow stability and predictable query
+ execution. They prevent operations that can take down the node and cause
+ noisy neighbour issues.
+
+ This flag is the uber control switch which controls the activation/deactivation of all circuit
+ breakers. At the moment, the only circuit breaker (max JVM circuit breaker) does not have its
+ own specific configuration. However, if a circuit breaker wishes to be independently configurable,
+ they are free to add their specific configuration but need to ensure that this flag is always
+ respected - this should have veto over all independent configuration flags.
+ -->
+ <!--
+ <useCircuitBreakers>true</useCircuitBreakers>
+ -->
+
+ <!-- Memory Circuit Breaker Threshold In Percentage
+
+ Specific configuration for max JVM heap usage circuit breaker. This configuration defines the
+ threshold percentage of maximum heap allocated beyond which queries will be rejected until the
+ current JVM usage goes below the threshold. The valid value range for this value is 50-95.
+
+ Consider a scenario where the max heap allocated is 4 GB and memoryCircuitBreakerThreshold is
+ defined as 75. Threshold JVM usage will be 4 * 0.75 = 3 GB. Its generally a good idea to keep this value between 75 - 80% of maximum heap
+ allocated.
+
+ If, at any point, the current JVM heap usage goes above 3 GB, queries will be rejected until the heap usage goes below 3 GB again.
+ If you see queries getting rejected with 503 error code, check for "Circuit Breakers tripped"
+ in logs and the corresponding error message should tell you what transpired (if the failure
+ was caused by tripped circuit breakers).
+ -->
+ <!--
+ <memoryCircuitBreakerThresholdPct>100</memoryCircuitBreakerThresholdPct>
+ -->
+
+ </circuitBreaker>
+
<!-- Request Dispatcher
diff --git a/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml b/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
index aca7e5b..027ed61 100644
--- a/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
+++ b/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
@@ -618,6 +618,48 @@
</query>
+ <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ Circuit Breaker Section - This section consists of configurations for
+ circuit breakers
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
+ <circuitBreaker>
+ <!-- Enable Circuit Breakers
+
+ Circuit breakers are designed to allow stability and predictable query
+ execution. They prevent operations that can take down the node and cause
+ noisy neighbour issues.
+
+ This flag is the uber control switch which controls the activation/deactivation of all circuit
+ breakers. At the moment, the only circuit breaker (max JVM circuit breaker) does not have its
+ own specific configuration. However, if a circuit breaker wishes to be independently configurable,
+ they are free to add their specific configuration but need to ensure that this flag is always
+ respected - this should have veto over all independent configuration flags.
+ -->
+ <!--
+ <useCircuitBreakers>true</useCircuitBreakers>
+ -->
+
+ <!-- Memory Circuit Breaker Threshold In Percentage
+
+ Specific configuration for max JVM heap usage circuit breaker. This configuration defines the
+ threshold percentage of maximum heap allocated beyond which queries will be rejected until the
+ current JVM usage goes below the threshold. The valid value for this range is 50-95.
+
+ Consider a scenario where the max heap allocated is 4 GB and memoryCircuitBreakerThresholdPct is
+ defined as 75. Threshold JVM usage will be 4 * 0.75 = 3 GB. Its generally a good idea to keep this value between 75 - 80% of maximum heap
+ allocated.
+
+ If, at any point, the current JVM heap usage goes above 3 GB, queries will be rejected until the heap usage goes below 3 GB again.
+ If you see queries getting rejected with 503 error code, check for "Circuit Breakers tripped"
+ in logs and the corresponding error message should tell you what transpired (if the failure
+ was caused by tripped circuit breakers).
+ -->
+ <!--
+ <memoryCircuitBreakerThresholdPct>100</memoryCircuitBreakerThresholdPct>
+ -->
+
+ </circuitBreaker>
+
<!-- Request Dispatcher
diff --git a/solr/solr-ref-guide/src/circuit-breakers.adoc b/solr/solr-ref-guide/src/circuit-breakers.adoc
new file mode 100644
index 0000000..1629c32
--- /dev/null
+++ b/solr/solr-ref-guide/src/circuit-breakers.adoc
@@ -0,0 +1,68 @@
+= Circuit Breakers
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+Solr's circuit breaker infrastructure allows prevention of actions that can cause a node to go beyond its capacity or to go down. The
+premise of circuit breakers is to ensure a higher quality of service and only accept request loads that are serviceable in the current
+resource configuration.
+
+== When To Use Circuit Breakers
+Circuit breakers should be used when the user wishes to trade request throughput for a higher Solr stability. If circuit breakers
+are enabled, requests may be rejected under the condition of high node duress with an appropriate HTTP error code (typically 503).
+
+It is up to the client to handle this error and potentially build a retrial logic as this should ideally be a transient situation.
+
+== Circuit Breaker Configurations
+The following flag controls the global activation/deactivation of circuit breakers. If this flag is disabled, all circuit breakers
+will be disabled globally. Per circuit breaker configurations are specified in their respective sections later.
+
+[source,xml]
+----
+<useCircuitBreakers>false</useCircuitBreakers>
+----
+
+== Currently Supported Circuit Breakers
+
+=== JVM Heap Usage Based Circuit Breaker
+This circuit breaker tracks JVM heap memory usage and rejects incoming search requests with a 503 error code if the heap usage
+exceeds a configured percentage of maximum heap allocated to the JVM (-Xmx). The main configuration for this circuit breaker is
+controlling the threshold percentage at which the breaker will trip.
+
+It does not logically make sense to have a threshold below 50% and above 95% of the max heap allocated to the JVM. Hence, the range
+of valid values for this parameter is [50, 95], both inclusive.
+
+[source,xml]
+----
+<memoryCircuitBreakerThresholdPct>75</memoryCircuitBreakerThresholdPct>
+----
+
+Consider the following example:
+
+JVM has been allocated a maximum heap of 5GB (-Xmx) and memoryCircuitBreakerThresholdPct is set to 75. In this scenario, the heap usage
+at which the circuit breaker will trip is 3.75GB.
+
+Note that this circuit breaker is checked for each incoming search request and considers the current heap usage of the node, i.e every search
+request will get the live heap usage and compare it against the set memory threshold. The check does not impact performance,
+but any performance regressions that are suspected to be caused by this feature should be reported to the dev list.
+
+
+== Performance Considerations
+It is worth noting that while JVM circuit breaker does not add any noticeable overhead per query, having too many
+circuit breakers checked for a single request can cause a performance overhead.
+
+In addition, it is a good practice to exponentially back off while retrying requests on a busy node.
+
diff --git a/solr/solr-ref-guide/src/config-api.adoc b/solr/solr-ref-guide/src/config-api.adoc
index 21227e4..6e5d30b 100644
--- a/solr/solr-ref-guide/src/config-api.adoc
+++ b/solr/solr-ref-guide/src/config-api.adoc
@@ -202,6 +202,13 @@ _Query Sizing and Warming_
* `query.queryResultWindowSize`
* `query.queryResultMaxDocCached`
+_Query Circuit Breakers_
+
+See <<circuit-breakers.adoc#circuit-breakers,Circuit Breakers in Solr>> for more details
+
+* `query.useCircuitBreakers`
+* `query.memoryCircuitBreakerThresholdPct`
+
*RequestDispatcher Settings*
See <<requestdispatcher-in-solrconfig.adoc#requestdispatcher-in-solrconfig,RequestDispatcher in SolrConfig>> for defaults and acceptable values for these settings.
diff --git a/solr/solr-ref-guide/src/index.adoc b/solr/solr-ref-guide/src/index.adoc
index dd98824..6a750e9 100644
--- a/solr/solr-ref-guide/src/index.adoc
+++ b/solr/solr-ref-guide/src/index.adoc
@@ -10,6 +10,7 @@
streaming-expressions, \
solrcloud, \
legacy-scaling-and-distribution, \
+ circuit-breakers, \
solr-plugins, \
the-well-configured-solr-instance, \
monitoring-solr, \
@@ -121,6 +122,8 @@ The *<<getting-started.adoc#getting-started,Getting Started>>* section guides yo
*<<solrcloud.adoc#solrcloud,SolrCloud>>*: This section describes SolrCloud, which provides comprehensive distributed capabilities.
*<<legacy-scaling-and-distribution.adoc#legacy-scaling-and-distribution,Legacy Scaling and Distribution>>*: This section tells you how to grow a Solr distribution by dividing a large index into sections called shards, which are then distributed across multiple servers, or by replicating a single index across multiple services.
+
+*<<circuit-breakers.adoc#circuit-breakers,Circuit Breakers>>*: This section talks about circuit breakers, a way of allowing a higher stability of Solr nodes and increased service level guarantees of requests that are accepted by Solr.
****
.Advanced Configuration
diff --git a/solr/solr-ref-guide/src/query-settings-in-solrconfig.adoc b/solr/solr-ref-guide/src/query-settings-in-solrconfig.adoc
index fbbb05b..9a336c3 100644
--- a/solr/solr-ref-guide/src/query-settings-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/query-settings-in-solrconfig.adoc
@@ -185,6 +185,26 @@ This parameter sets the maximum number of documents to cache for any entry in th
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
----
+=== useCircuitBreakers
+
+Global control flag for enabling circuit breakers.
+
+[source,xml]
+----
+<useCircuitBreakers>true</useCircuitBreakers>
+----
+
+=== memoryCircuitBreakerThresholdPct
+
+Memory threshold in percentage for JVM heap usage defined in percentage of maximum heap allocated
+to the JVM (-Xmx). Ideally, this value should be in the range of 75-80% of maximum heap allocated
+to the JVM.
+
+[source,xml]
+----
+<memoryCircuitBreakerThresholdPct>75</memoryCircuitBreakerThresholdPct>
+----
+
=== useColdSearcher
This setting controls whether search requests for which there is not a currently registered searcher should wait for a new searcher to warm up (false) or proceed immediately (true). When set to "false", requests will block until the searcher has warmed its caches.
[lucene-solr] 02/04: SOLR-14615: Implement CPU Utilization Based
Circuit Breaker (#1737)
Posted by at...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
atri pushed a commit to branch branch_8x
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit c841f63e4ba8615d648f00741d4ee3628cd60954
Author: Atri Sharma <at...@gmail.com>
AuthorDate: Thu Aug 20 13:21:26 2020 +0530
SOLR-14615: Implement CPU Utilization Based Circuit Breaker (#1737)
This commit introduces CPU based circuit breaker. This circuit breaker
tracks the average CPU load per minute and triggers if the value exceeds
a configurable value.
This commit also adds a specific control flag for Memory Circuit Breaker
to allow enabling/disabling the same.
---
solr/CHANGES.txt | 101 ++++++++++++++++++
.../src/java/org/apache/solr/core/SolrConfig.java | 27 +++--
.../util/circuitbreaker/CPUCircuitBreaker.java | 116 +++++++++++++++++++++
.../solr/util/circuitbreaker/CircuitBreaker.java | 8 ++
.../util/circuitbreaker/CircuitBreakerManager.java | 13 ++-
.../util/circuitbreaker/MemoryCircuitBreaker.java | 20 +++-
.../resources/EditableSolrConfigAttributes.json | 13 ++-
.../conf/solrconfig-memory-circuitbreaker.xml | 11 +-
.../test/org/apache/solr/core/SolrCoreTest.java | 3 +-
.../org/apache/solr/util/TestCircuitBreaker.java | 104 +++++++++++++++++-
.../solr/configsets/_default/conf/solrconfig.xml | 40 ++++---
solr/solr-ref-guide/src/circuit-breakers.adoc | 38 +++++--
12 files changed, 447 insertions(+), 47 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 5f34199..6898b5d 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -29,6 +29,107 @@ Improvements
----------------------
* LUCENE-8984: MoreLikeThis MLT is biased for uncommon fields (Andy Hind via Anshum Gupta)
+* SOLR-14223: PKI Auth can bootstrap from existing key files instead of creating new keys on startup (Mike Drob)
+
+* SOLR-11725: Use corrected sample formula for computing stdDev and variance in JSON aggregations
+ (hossman, Munendra S N, yonik)
+
+* SOLR-14387: SolrClient.getById() will escape comma separater within ids (Markus Schuch via Mike Drob)
+
+* SOLR-10814: Add short-name feature to RuleBasedAuthz plugin (Mike Drob, Hrishikesh Gadre)
+
+* SOLR-7683 Introduce support to identify Solr internal request types (Atri Sharma, Hrishikesh Gadre)
+
+* SOLR-13528 Rate Limiting in Solr (Atri Sharma, Mike Drob)
+
+* SOLR-14615: CPU Utilization Based Circuit Breaker (Atri Sharma)
+
+Other Changes
+----------------------
+* SOLR-14656: Autoscaling framework removed (Ishan Chattopadhyaya, noble, Ilan Ginzburg)
+
+* LUCENE-9391: Upgrade HPPC to 0.8.2. (Haoyu Zhai)
+
+* SOLR-10288: Remove non-minified JavaScript from the webapp. (Erik Hatcher, marcussorealheis)
+
+* SOLR-13655:Upgrade Collections.unModifiableSet to Set.of and Set.copyOf (Atri Sharma via Tomás Fernández Löbbe)
+
+* SOLR-13797: SolrResourceLoader no longer caches bad results when asked for wrong type (Mike Drob)
+
+* LUCENE-9092: Upgrade Carrot2 to 3.16.2 (Dawid Weiss).
+
+* LUCENE-9080: Upgrade ICU4j to 62.2 and make regenerate work (Erick Erickson)
+
+* SOLR-14271: Remove duplicate async id check meant for pre Solr 8 versions (Anshum Gupta)
+
+* SOLR-14272: Remove autoReplicaFailoverBadNodeExpiration and autoReplicaFailoverWorkLoopDelay for 9.0 as it was
+ deprecated in 7.1 (Anshum Gupta)
+
+* SOLR-14258: DocList no longer extends DocSet. (David Smiley)
+
+* SOLR-14256: Remove HashDocSet; add DocSet.getBits() instead. DocSet is now strictly immutable and ascending order.
+ It's now locked-down to external extension; only 2 impls exist. (David Smiley)
+
+* SOLR-14197: SolrResourceLoader: remove deprecated methods and do other improvements. (David Smiley)
+
+* SOLR-14012: Return long value for unique and hll aggregations irrespective of shard count (Munendra S N, hossman)
+
+* SOLR-14322: AbstractFullDistribZkTestBase.waitForRecoveriesToFinish now takes a timeout and time unit instead of
+ assuming that we are passed value in seconds. (Mike Drob)
+
+* SOLR-13893: Remove support to read BlobRepository's max jar size from deprecated `runtme.lib.size` system property
+ (Erick Erickson, Kesharee Nandan Vishwakarma, Munendra S N)
+
+* SOLR-12720: Remove support for `autoReplicaFailoverWaitAfterExpiration`. (marcussorealheis, shalin)
+
+* SOLR-9909: The deprecated SolrjNamedThreadFactory has been removed. Use SolrNamedThreadFactory instead.
+ (Andras Salamon, shalin)
+
+* SOLR-14420: AuthenticationPlugin.authenticate accepts HttpServletRequest instead of ServletRequest. (Mike Drob)
+
+* SOLR-14429: Convert .txt files to properly formatted .md files. (Tomoko Uchida, Uwe Schindler)
+
+* SOLR-14412: Automatically set urlScheme to https when running secure solr with embedded zookeeper. (Mike Drob)
+ Do not erroneously set solr.jetty.https.port system property when running in http mode (Upendra Penegalapati)
+
+* SOLR-14014: Introducing a system property that allows users to disable the Admin UI, which is enabled by default.
+ If you have security concerns or other reasons to disable the Admin UI, you can modify `SOLR_ADMIN_UI_DISABLED`
+ `solr.in.sh`/`solr.in.cmd` at start. (marcussorealheis)
+
+* SOLR-14486: Autoscaling simulation framework no longer creates /clusterstate.json (format 1),
+ instead it creates individual per-collection /state.json files (format 2). (ab)
+
+* SOLR-12823: Remove /clusterstate.json support, including support for collections created with stateFormat=1,
+ as well as support for Collection API MIGRATESTATEFORMAT action and support for the legacyCloud flag (Ilan Ginzburg).
+
+* LUCENE-9411: Fail complation on warnings, 9x gradle-only (Erick Erickson, Dawid Weiss)
+ Deserves mention here as well as Lucene CHANGES.txt since it affects both.
+
+* SOLR-12847: Remove support for maxShardsPerNode. (ab)
+
+* SOLR-14244: Remove ReplicaInfo. (ab)
+
+* SOLR-14654: Remove plugin loading from .system collection (for 9.0) (noble)
+
+* SOLR-14702: All references to "master" and "slave" replaced with "leader" and "follower" (MarcusSorealheis,
+ Erick Erickson, Tomás Fernández Löbbe)
+
+Bug Fixes
+---------------------
+* SOLR-14546: Fix for a relatively hard to hit issue in OverseerTaskProcessor that could lead to out of order execution
+ of Collection API tasks competing for a lock (Ilan Ginzburg).
+
+================== 8.7.0 ==================
+
+Consult the lucene/CHANGES.txt file for additional, low level, changes in this release.
+
+New Features
+---------------------
+
+* SOLR-14151 Make schema components load from packages (noble)
+
+* SOLR-14615: Implement CPU Utilization Based Circuit Breaker (#1737)
+
* SOLR-14681: Introduce ability to delete .jar stored in the Package Store. (MarcusSorealheis , Mike Drob)
* SOLR-14604: Add the ability to uninstall a package from with the Package CLI. (MarcusSorealheis)
diff --git a/solr/core/src/java/org/apache/solr/core/SolrConfig.java b/solr/core/src/java/org/apache/solr/core/SolrConfig.java
index ec81688..707d770 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrConfig.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrConfig.java
@@ -228,10 +228,13 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
queryResultMaxDocsCached = getInt("query/queryResultMaxDocsCached", Integer.MAX_VALUE);
enableLazyFieldLoading = getBool("query/enableLazyFieldLoading", false);
- useCircuitBreakers = getBool("circuitBreaker/useCircuitBreakers", false);
- memoryCircuitBreakerThresholdPct = getInt("circuitBreaker/memoryCircuitBreakerThresholdPct", 95);
+ useCircuitBreakers = getBool("circuitBreakers/@enabled", false);
+ cpuCBEnabled = getBool("circuitBreakers/cpuBreaker/@enabled", false);
+ memCBEnabled = getBool("circuitBreakers/memBreaker/@enabled", false);
+ memCBThreshold = getInt("circuitBreakers/memBreaker/@threshold", 95);
+ cpuCBThreshold = getInt("circuitBreakers/cpuBreaker/@threshold", 95);
- validateMemoryBreakerThreshold();
+ validateCircuitBreakerThresholds();
filterCacheConfig = CacheConfig.getConfig(this, "query/filterCache");
queryResultCacheConfig = CacheConfig.getConfig(this, "query/queryResultCache");
@@ -530,7 +533,10 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
public final boolean enableLazyFieldLoading;
// Circuit Breaker Configuration
public final boolean useCircuitBreakers;
- public final int memoryCircuitBreakerThresholdPct;
+ public final int memCBThreshold;
+ public final boolean memCBEnabled;
+ public final boolean cpuCBEnabled;
+ public final int cpuCBThreshold;
// IndexConfig settings
public final SolrIndexConfig indexConfig;
@@ -811,10 +817,12 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
loader.reloadLuceneSPI();
}
- private void validateMemoryBreakerThreshold() {
+ private void validateCircuitBreakerThresholds() {
if (useCircuitBreakers) {
- if (memoryCircuitBreakerThresholdPct > 95 || memoryCircuitBreakerThresholdPct < 50) {
- throw new IllegalArgumentException("Valid value range of memoryCircuitBreakerThresholdPct is 50 - 95");
+ if (memCBEnabled) {
+ if (memCBThreshold > 95 || memCBThreshold < 50) {
+ throw new IllegalArgumentException("Valid value range of memoryCircuitBreakerThresholdPct is 50 - 95");
+ }
}
}
}
@@ -889,7 +897,10 @@ public class SolrConfig extends XmlConfigFile implements MapSerializable {
m.put("enableLazyFieldLoading", enableLazyFieldLoading);
m.put("maxBooleanClauses", booleanQueryMaxClauseCount);
m.put("useCircuitBreakers", useCircuitBreakers);
- m.put("memoryCircuitBreakerThresholdPct", memoryCircuitBreakerThresholdPct);
+ m.put("cpuCircuitBreakerEnabled", cpuCBEnabled);
+ m.put("memoryCircuitBreakerEnabled", memCBEnabled);
+ m.put("memoryCircuitBreakerThresholdPct", memCBThreshold);
+ m.put("cpuCircuitBreakerThreshold", cpuCBThreshold);
for (SolrPluginInfo plugin : plugins) {
List<PluginInfo> infos = getPluginInfos(plugin.clazz.getName());
if (infos == null || infos.isEmpty()) continue;
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java
new file mode 100644
index 0000000..45dc1e8
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.util.circuitbreaker;
+
+import java.lang.invoke.MethodHandles;
+import java.lang.management.ManagementFactory;
+import java.lang.management.OperatingSystemMXBean;
+
+import org.apache.solr.core.SolrConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * <p>
+ * Tracks current CPU usage and triggers if the specified threshold is breached.
+ *
+ * This circuit breaker gets the average CPU load over the last minute and uses
+ * that data to take a decision. We depend on OperatingSystemMXBean which does
+ * not allow a configurable interval of collection of data.
+ * //TODO: Use Codahale Meter to calculate the value locally.
+ * </p>
+ *
+ * <p>
+ * The configuration to define which mode to use and the trigger threshold are defined in
+ * solrconfig.xml
+ * </p>
+ */
+public class CPUCircuitBreaker extends CircuitBreaker {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final OperatingSystemMXBean operatingSystemMXBean = ManagementFactory.getOperatingSystemMXBean();
+
+ private final boolean enabled;
+ private final double cpuUsageThreshold;
+
+ // Assumption -- the value of these parameters will be set correctly before invoking getDebugInfo()
+ private static final ThreadLocal<Double> seenCPUUsage = ThreadLocal.withInitial(() -> 0.0);
+
+ private static final ThreadLocal<Double> allowedCPUUsage = ThreadLocal.withInitial(() -> 0.0);
+
+ public CPUCircuitBreaker(SolrConfig solrConfig) {
+ super(solrConfig);
+
+ this.enabled = solrConfig.cpuCBEnabled;
+ this.cpuUsageThreshold = solrConfig.cpuCBThreshold;
+ }
+
+ @Override
+ public boolean isTripped() {
+ if (!isEnabled()) {
+ return false;
+ }
+
+ if (!enabled) {
+ return false;
+ }
+
+ double localAllowedCPUUsage = getCpuUsageThreshold();
+ double localSeenCPUUsage = calculateLiveCPUUsage();
+
+ if (localSeenCPUUsage < 0) {
+ if (log.isWarnEnabled()) {
+ String msg = "Unable to get CPU usage";
+
+ log.warn(msg);
+ }
+
+ return false;
+ }
+
+ allowedCPUUsage.set(localAllowedCPUUsage);
+
+ seenCPUUsage.set(localSeenCPUUsage);
+
+ return (localSeenCPUUsage >= localAllowedCPUUsage);
+ }
+
+ @Override
+ public String getDebugInfo() {
+
+ if (seenCPUUsage.get() == 0.0 || seenCPUUsage.get() == 0.0) {
+ log.warn("CPUCircuitBreaker's monitored values (seenCPUUSage, allowedCPUUsage) not set");
+ }
+
+ return "seenCPUUSage=" + seenCPUUsage.get() + " allowedCPUUsage=" + allowedCPUUsage.get();
+ }
+
+ @Override
+ public String getErrorMessage() {
+ return "CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold." +
+ "Seen CPU usage " + seenCPUUsage.get() + " and allocated threshold " +
+ allowedCPUUsage.get();
+ }
+
+ public double getCpuUsageThreshold() {
+ return cpuUsageThreshold;
+ }
+
+ protected double calculateLiveCPUUsage() {
+ return operatingSystemMXBean.getSystemLoadAverage();
+ }
+}
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
index f56f81e..63ad7fd 100644
--- a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreaker.java
@@ -27,6 +27,9 @@ import org.apache.solr.core.SolrConfig;
* 2. Use the circuit breaker in a specific code path(s).
*
* TODO: This class should be grown as the scope of circuit breakers grow.
+ *
+ * The class and its derivatives raise a standard exception when a circuit breaker is triggered.
+ * We should make it into a dedicated exception (https://issues.apache.org/jira/browse/SOLR-14755)
* </p>
*/
public abstract class CircuitBreaker {
@@ -53,4 +56,9 @@ public abstract class CircuitBreaker {
* Get debug useful info.
*/
public abstract String getDebugInfo();
+
+ /**
+ * Get error message when the circuit breaker triggers
+ */
+ public abstract String getErrorMessage();
}
\ No newline at end of file
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
index 584b933..ed7f62d 100644
--- a/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerManager.java
@@ -20,6 +20,7 @@ package org.apache.solr.util.circuitbreaker;
import java.util.ArrayList;
import java.util.List;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.solr.core.SolrConfig;
/**
@@ -107,9 +108,7 @@ public class CircuitBreakerManager {
StringBuilder sb = new StringBuilder();
for (CircuitBreaker circuitBreaker : circuitBreakerList) {
- sb.append(circuitBreaker.getClass().getName());
- sb.append(" ");
- sb.append(circuitBreaker.getDebugInfo());
+ sb.append(circuitBreaker.getErrorMessage());
sb.append("\n");
}
@@ -127,8 +126,16 @@ public class CircuitBreakerManager {
// Install the default circuit breakers
CircuitBreaker memoryCircuitBreaker = new MemoryCircuitBreaker(solrConfig);
+ CircuitBreaker cpuCircuitBreaker = new CPUCircuitBreaker(solrConfig);
+
circuitBreakerManager.register(memoryCircuitBreaker);
+ circuitBreakerManager.register(cpuCircuitBreaker);
return circuitBreakerManager;
}
+
+ @VisibleForTesting
+ public List<CircuitBreaker> getRegisteredCircuitBreakers() {
+ return circuitBreakerList;
+ }
}
diff --git a/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java b/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
index 629d84a..797677d 100644
--- a/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
+++ b/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java
@@ -43,22 +43,25 @@ public class MemoryCircuitBreaker extends CircuitBreaker {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final MemoryMXBean MEMORY_MX_BEAN = ManagementFactory.getMemoryMXBean();
+ private boolean enabled;
private final long heapMemoryThreshold;
// Assumption -- the value of these parameters will be set correctly before invoking getDebugInfo()
- private final ThreadLocal<Long> seenMemory = new ThreadLocal<>();
- private final ThreadLocal<Long> allowedMemory = new ThreadLocal<>();
+ private static final ThreadLocal<Long> seenMemory = ThreadLocal.withInitial(() -> 0L);
+ private static final ThreadLocal<Long> allowedMemory = ThreadLocal.withInitial(() -> 0L);
public MemoryCircuitBreaker(SolrConfig solrConfig) {
super(solrConfig);
+ this.enabled = solrConfig.memCBEnabled;
+
long currentMaxHeap = MEMORY_MX_BEAN.getHeapMemoryUsage().getMax();
if (currentMaxHeap <= 0) {
throw new IllegalArgumentException("Invalid JVM state for the max heap usage");
}
- int thresholdValueInPercentage = solrConfig.memoryCircuitBreakerThresholdPct;
+ int thresholdValueInPercentage = solrConfig.memCBThreshold;
double thresholdInFraction = thresholdValueInPercentage / (double) 100;
heapMemoryThreshold = (long) (currentMaxHeap * thresholdInFraction);
@@ -76,6 +79,10 @@ public class MemoryCircuitBreaker extends CircuitBreaker {
return false;
}
+ if (!enabled) {
+ return false;
+ }
+
long localAllowedMemory = getCurrentMemoryThreshold();
long localSeenMemory = calculateLiveMemoryUsage();
@@ -95,6 +102,13 @@ public class MemoryCircuitBreaker extends CircuitBreaker {
return "seenMemory=" + seenMemory.get() + " allowedMemory=" + allowedMemory.get();
}
+ @Override
+ public String getErrorMessage() {
+ return "Memory Circuit Breaker triggered as JVM heap usage values are greater than allocated threshold." +
+ "Seen JVM heap memory usage " + seenMemory.get() + " and allocated threshold " +
+ allowedMemory.get();
+ }
+
private long getCurrentMemoryThreshold() {
return heapMemoryThreshold;
}
diff --git a/solr/core/src/resources/EditableSolrConfigAttributes.json b/solr/core/src/resources/EditableSolrConfigAttributes.json
index 0ed8333..1f49982 100644
--- a/solr/core/src/resources/EditableSolrConfigAttributes.json
+++ b/solr/core/src/resources/EditableSolrConfigAttributes.json
@@ -55,8 +55,17 @@
"queryResultMaxDocsCached":1,
"enableLazyFieldLoading":1,
"boolTofilterOptimizer":1,
- "useCircuitBreakers":10,
- "memoryCircuitBreakerThresholdPct":20,
+ "circuitBreakers":{
+ "enabled":10,
+ "memBreaker":{
+ "enabled":10,
+ "threshold":20
+ },
+ "cpuBreaker":{
+ "enabled":10,
+ "threshold":20
+ }
+ },
"maxBooleanClauses":1},
"jmx":{
"agentId":0,
diff --git a/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml b/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml
index b6b20ff..d6adf92 100644
--- a/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml
+++ b/solr/core/src/test-files/solr/collection1/conf/solrconfig-memory-circuitbreaker.xml
@@ -78,13 +78,10 @@
</query>
- <circuitBreaker>
-
- <useCircuitBreakers>true</useCircuitBreakers>
-
- <memoryCircuitBreakerThresholdPct>75</memoryCircuitBreakerThresholdPct>
-
- </circuitBreaker>
+ <circuitBreakers enabled="true">
+ <cpuBreaker enabled="true" threshold="75"/>
+ <memBreaker enabled="true" threshold="75"/>
+ </circuitBreakers>
<initParams path="/select">
<lst name="defaults">
diff --git a/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java b/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
index dcf593f..342fa3e 100644
--- a/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
+++ b/solr/core/src/test/org/apache/solr/core/SolrCoreTest.java
@@ -266,7 +266,8 @@ public class SolrCoreTest extends SolrTestCaseJ4 {
assertEquals("wrong config for enableLazyFieldLoading", true, solrConfig.enableLazyFieldLoading);
assertEquals("wrong config for queryResultWindowSize", 10, solrConfig.queryResultWindowSize);
assertEquals("wrong config for useCircuitBreakers", false, solrConfig.useCircuitBreakers);
- assertEquals("wrong config for memoryCircuitBreakerThresholdPct", 95, solrConfig.memoryCircuitBreakerThresholdPct);
+ assertEquals("wrong config for memoryCircuitBreakerThresholdPct", 95, solrConfig.memCBThreshold);
+ assertEquals("wrong config for cpuCircuitBreakerThreshold", 95, solrConfig.cpuCBThreshold);
}
/**
diff --git a/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java b/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
index 00b8d1a..83e8fc5 100644
--- a/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
+++ b/solr/core/src/test/org/apache/solr/util/TestCircuitBreaker.java
@@ -17,8 +17,11 @@
package org.apache.solr.util;
+import java.util.ArrayList;
import java.util.HashMap;
+import java.util.List;
import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
@@ -30,6 +33,7 @@ import org.apache.solr.common.util.ExecutorUtil;
import org.apache.solr.common.util.SolrNamedThreadFactory;
import org.apache.solr.core.SolrConfig;
import org.apache.solr.search.QueryParsing;
+import org.apache.solr.util.circuitbreaker.CPUCircuitBreaker;
import org.apache.solr.util.circuitbreaker.CircuitBreaker;
import org.apache.solr.util.circuitbreaker.MemoryCircuitBreaker;
import org.junit.After;
@@ -38,6 +42,9 @@ import org.junit.Rule;
import org.junit.rules.RuleChain;
import org.junit.rules.TestRule;
+import static org.hamcrest.CoreMatchers.containsString;
+
+@SuppressWarnings({"rawtypes"})
public class TestCircuitBreaker extends SolrTestCaseJ4 {
private final static int NUM_DOCS = 20;
@@ -80,6 +87,8 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
args.put(QueryParsing.DEFTYPE, CircuitBreaker.NAME);
args.put(CommonParams.FL, "id");
+ removeAllExistingCircuitBreakers();
+
CircuitBreaker circuitBreaker = new MockCircuitBreaker(h.getCore().getSolrConfig());
h.getCore().getCircuitBreakerManager().register(circuitBreaker);
@@ -95,6 +104,8 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
args.put(QueryParsing.DEFTYPE, CircuitBreaker.NAME);
args.put(CommonParams.FL, "id");
+ removeAllExistingCircuitBreakers();
+
CircuitBreaker circuitBreaker = new FakeMemoryPressureCircuitBreaker(h.getCore().getSolrConfig());
h.getCore().getCircuitBreakerManager().register(circuitBreaker);
@@ -115,26 +126,43 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
AtomicInteger failureCount = new AtomicInteger();
try {
+ removeAllExistingCircuitBreakers();
+
CircuitBreaker circuitBreaker = new BuildingUpMemoryPressureCircuitBreaker(h.getCore().getSolrConfig());
h.getCore().getCircuitBreakerManager().register(circuitBreaker);
+ List<Future<?>> futures = new ArrayList<>();
+
for (int i = 0; i < 5; i++) {
- executor.submit(() -> {
+ Future<?> future = executor.submit(() -> {
try {
h.query(req("name:\"john smith\""));
} catch (SolrException e) {
+
+ assertThat(e.getMessage(), containsString("Circuit Breakers tripped"));
failureCount.incrementAndGet();
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
});
+
+ futures.add(future);
+ }
+
+ for (Future<?> future : futures) {
+ try {
+ future.get();
+ } catch (Exception e) {
+ throw new RuntimeException(e.getMessage());
+ }
}
executor.shutdown();
try {
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
throw new RuntimeException(e.getMessage());
}
@@ -146,6 +174,59 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
}
}
+ public void testFakeCPUCircuitBreaker() {
+ AtomicInteger failureCount = new AtomicInteger();
+
+ ExecutorService executor = ExecutorUtil.newMDCAwareCachedThreadPool(
+ new SolrNamedThreadFactory("TestCircuitBreaker"));
+ try {
+ removeAllExistingCircuitBreakers();
+
+ CircuitBreaker circuitBreaker = new FakeCPUCircuitBreaker(h.getCore().getSolrConfig());
+
+ h.getCore().getCircuitBreakerManager().register(circuitBreaker);
+
+ List<Future<?>> futures = new ArrayList<>();
+
+ for (int i = 0; i < 5; i++) {
+ Future<?> future = executor.submit(() -> {
+ try {
+ h.query(req("name:\"john smith\""));
+ } catch (SolrException e) {
+ assertThat(e.getMessage(), containsString("Circuit Breakers tripped"));
+ failureCount.incrementAndGet();
+ } catch (Exception e) {
+ throw new RuntimeException(e.getMessage());
+ }
+ });
+
+ futures.add(future);
+ }
+
+ for (Future<?> future : futures) {
+ try {
+ future.get();
+ } catch (Exception e) {
+ throw new RuntimeException(e.getMessage());
+ }
+ }
+
+ executor.shutdown();
+ try {
+ executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e.getMessage());
+ }
+
+ assertEquals("Number of failed queries is not correct",5, failureCount.get());
+ } finally {
+ if (!executor.isShutdown()) {
+ executor.shutdown();
+ }
+ }
+ }
+
public void testResponseWithCBTiming() {
assertQ(req("q", "*:*", CommonParams.DEBUG_QUERY, "true"),
"//str[@name='rawquerystring']='*:*'",
@@ -168,7 +249,13 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
);
}
- private class MockCircuitBreaker extends CircuitBreaker {
+ private void removeAllExistingCircuitBreakers() {
+ List<CircuitBreaker> registeredCircuitBreakers = h.getCore().getCircuitBreakerManager().getRegisteredCircuitBreakers();
+
+ registeredCircuitBreakers.clear();
+ }
+
+ private static class MockCircuitBreaker extends MemoryCircuitBreaker {
public MockCircuitBreaker(SolrConfig solrConfig) {
super(solrConfig);
@@ -186,7 +273,7 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
}
}
- private class FakeMemoryPressureCircuitBreaker extends MemoryCircuitBreaker {
+ private static class FakeMemoryPressureCircuitBreaker extends MemoryCircuitBreaker {
public FakeMemoryPressureCircuitBreaker(SolrConfig solrConfig) {
super(solrConfig);
@@ -215,4 +302,15 @@ public class TestCircuitBreaker extends SolrTestCaseJ4 {
return 5; // Random number guaranteed to not trip the circuit breaker
}
}
+
+ private static class FakeCPUCircuitBreaker extends CPUCircuitBreaker {
+ public FakeCPUCircuitBreaker(SolrConfig solrConfig) {
+ super(solrConfig);
+ }
+
+ @Override
+ protected double calculateLiveCPUUsage() {
+ return 92; // Return a value large enough to trigger the circuit breaker
+ }
+ }
}
diff --git a/solr/server/solr/configsets/_default/conf/solrconfig.xml b/solr/server/solr/configsets/_default/conf/solrconfig.xml
index cdd56cf..85c5671 100644
--- a/solr/server/solr/configsets/_default/conf/solrconfig.xml
+++ b/solr/server/solr/configsets/_default/conf/solrconfig.xml
@@ -598,27 +598,24 @@
Circuit Breaker Section - This section consists of configurations for
circuit breakers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
- <circuitBreaker>
- <!-- Enable Circuit Breakers
+
+ <!-- Circuit Breakers
Circuit breakers are designed to allow stability and predictable query
execution. They prevent operations that can take down the node and cause
noisy neighbour issues.
This flag is the uber control switch which controls the activation/deactivation of all circuit
- breakers. At the moment, the only circuit breaker (max JVM circuit breaker) does not have its
- own specific configuration. However, if a circuit breaker wishes to be independently configurable,
+ breakers. If a circuit breaker wishes to be independently configurable,
they are free to add their specific configuration but need to ensure that this flag is always
respected - this should have veto over all independent configuration flags.
-->
- <!--
- <useCircuitBreakers>true</useCircuitBreakers>
- -->
+ <circuitBreakers enabled="true">
- <!-- Memory Circuit Breaker Threshold In Percentage
+ <!-- Memory Circuit Breaker Configuration
- Specific configuration for max JVM heap usage circuit breaker. This configuration defines the
- threshold percentage of maximum heap allocated beyond which queries will be rejected until the
+ Specific configuration for max JVM heap usage circuit breaker. This configuration defines whether
+ the circuit breaker is enabled and the threshold percentage of maximum heap allocated beyond which queries will be rejected until the
current JVM usage goes below the threshold. The valid value range for this value is 50-95.
Consider a scenario where the max heap allocated is 4 GB and memoryCircuitBreakerThreshold is
@@ -629,12 +626,31 @@
If you see queries getting rejected with 503 error code, check for "Circuit Breakers tripped"
in logs and the corresponding error message should tell you what transpired (if the failure
was caused by tripped circuit breakers).
+
+ If, at any point, the current JVM heap usage goes above 3 GB, queries will be rejected until the heap usage goes below 3 GB again.
+ If you see queries getting rejected with 503 error code, check for "Circuit Breakers tripped"
+ in logs and the corresponding error message should tell you what transpired (if the failure
+ was caused by tripped circuit breakers).
-->
<!--
- <memoryCircuitBreakerThresholdPct>100</memoryCircuitBreakerThresholdPct>
+ <memBreaker enabled="true" threshold="75"/>
-->
- </circuitBreaker>
+ <!-- CPU Circuit Breaker Configuration
+
+ Specific configuration for CPU utilization based circuit breaker. This configuration defines whether the circuit breaker is enabled
+ and the average load over the last minute at which the circuit breaker should start rejecting queries.
+
+ Consider a scenario where the max heap allocated is 4 GB and memoryCircuitBreakerThreshold is
+ defined as 75. Threshold JVM usage will be 4 * 0.75 = 3 GB. Its generally a good idea to keep this value between 75 - 80% of maximum heap
+ allocated.
+ -->
+
+ <!--
+ <cpuBreaker enabled="true" threshold="75"/>
+ -->
+
+ </circuitBreakers>
<!-- Request Dispatcher
diff --git a/solr/solr-ref-guide/src/circuit-breakers.adoc b/solr/solr-ref-guide/src/circuit-breakers.adoc
index 1629c32..56451b0 100644
--- a/solr/solr-ref-guide/src/circuit-breakers.adoc
+++ b/solr/solr-ref-guide/src/circuit-breakers.adoc
@@ -32,9 +32,14 @@ will be disabled globally. Per circuit breaker configurations are specified in t
[source,xml]
----
-<useCircuitBreakers>false</useCircuitBreakers>
+<circuitBreakers enabled="true">
+ <!-- All specific configs in this section -->
+</circuitBreakers>
----
+This flag acts as the highest authority and global controller of circuit breakers. For using specific circuit breakers, each one
+needs to be individually enabled in addition to this flag being enabled.
+
== Currently Supported Circuit Breakers
=== JVM Heap Usage Based Circuit Breaker
@@ -42,26 +47,43 @@ This circuit breaker tracks JVM heap memory usage and rejects incoming search re
exceeds a configured percentage of maximum heap allocated to the JVM (-Xmx). The main configuration for this circuit breaker is
controlling the threshold percentage at which the breaker will trip.
-It does not logically make sense to have a threshold below 50% and above 95% of the max heap allocated to the JVM. Hence, the range
-of valid values for this parameter is [50, 95], both inclusive.
+Configuration for JVM heap usage based circuit breaker:
[source,xml]
----
-<memoryCircuitBreakerThresholdPct>75</memoryCircuitBreakerThresholdPct>
+<memBreaker enabled="true" threshold="75"/>
----
+Note that this configuration will be overridden by the global circuit breaker flag -- if circuit breakers are disabled, this flag
+will not help you. Also, the triggering threshold is defined as a percentage of the max heap allocated to the JVM.
+
+It does not logically make sense to have a threshold below 50% and above 95% of the max heap allocated to the JVM. Hence, the range
+of valid values for this parameter is [50, 95], both inclusive.
+
Consider the following example:
JVM has been allocated a maximum heap of 5GB (-Xmx) and memoryCircuitBreakerThresholdPct is set to 75. In this scenario, the heap usage
at which the circuit breaker will trip is 3.75GB.
-Note that this circuit breaker is checked for each incoming search request and considers the current heap usage of the node, i.e every search
-request will get the live heap usage and compare it against the set memory threshold. The check does not impact performance,
-but any performance regressions that are suspected to be caused by this feature should be reported to the dev list.
+=== CPU Utilization Based Circuit Breaker
+This circuit breaker tracks CPU utilization and triggers if the average CPU utilization over the last one minute
+exceeds a configurable threshold. Note that the value used in computation is over the last one minute -- so a sudden
+spike in traffic that goes down might still cause the circuit breaker to trigger for a short while before it resolves
+and updates the value. For more details of the calculation, please see https://en.wikipedia.org/wiki/Load_(computing)
+
+Configuration for CPU utilization based circuit breaker:
+
+[source,xml]
+----
+<cpuBreaker enabled="true" threshold="20"/>
+----
+
+Note that this configuration will be overridden by the global circuit breaker flag -- if circuit breakers are disabled, this flag
+will not help you. The triggering threshold is defined in units of CPU utilization.
== Performance Considerations
-It is worth noting that while JVM circuit breaker does not add any noticeable overhead per query, having too many
+It is worth noting that while JVM or CPU circuit breakers do not add any noticeable overhead per query, having too many
circuit breakers checked for a single request can cause a performance overhead.
In addition, it is a good practice to exponentially back off while retrying requests on a busy node.