You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by sh...@apache.org on 2019/01/02 06:29:12 UTC
lucene-solr:master: SOLR-13082: A trigger that creates trigger events
more frequently than the cool down period can starve other triggers.
Repository: lucene-solr
Updated Branches:
refs/heads/master 2532a5d31 -> 5016959ce
SOLR-13082: A trigger that creates trigger events more frequently than the cool down period can starve other triggers.
This is mitigated to some extent by randomly choosing the trigger to resume after cool down. It is recommended that scheduled triggers not be used for very frequent operations to avoid this problem.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5016959c
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5016959c
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5016959c
Branch: refs/heads/master
Commit: 5016959ce8c1eed9d354822f01edc4b509e4aa9d
Parents: 2532a5d
Author: Shalin Shekhar Mangar <sh...@apache.org>
Authored: Wed Jan 2 11:59:00 2019 +0530
Committer: Shalin Shekhar Mangar <sh...@apache.org>
Committed: Wed Jan 2 11:59:00 2019 +0530
----------------------------------------------------------------------
solr/CHANGES.txt | 5 +++++
.../cloud/autoscaling/ScheduledTriggers.java | 21 ++++++++++++++++++--
.../src/solrcloud-autoscaling-triggers.adoc | 12 +++++++++++
3 files changed, 36 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5016959c/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 17bf849..0690864 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -193,6 +193,11 @@ Bug Fixes
* SOLR-13080: The "terms" QParser's "automaton" method semi-required that the input terms/IDs be sorted. This
query parser now does this. Unclear if this is a perf issue or actual bug. (Daniel Lowe, David Smiley)
+* SOLR-13082: A trigger that creates trigger events more frequently than the cool down period can starve other triggers.
+ This is mitigated to some extent by randomly choosing the trigger to resume after cool down. It is recommended that
+ scheduled triggers not be used for very frequent operations to avoid this problem.
+ (ab, shalin)
+
Improvements
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5016959c/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
index cad463b..05a8020 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
@@ -28,6 +28,7 @@ import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Random;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
@@ -88,6 +89,18 @@ public class ScheduledTriggers implements Closeable {
DEFAULT_PROPERTIES.put(ACTION_THROTTLE_PERIOD_SECONDS, DEFAULT_ACTION_THROTTLE_PERIOD_SECONDS);
}
+ protected static final Random RANDOM;
+ static {
+ // We try to make things reproducible in the context of our tests by initializing the random instance
+ // based on the current seed
+ String seed = System.getProperty("tests.seed");
+ if (seed == null) {
+ RANDOM = new Random();
+ } else {
+ RANDOM = new Random(seed.hashCode());
+ }
+ }
+
private final Map<String, TriggerWrapper> scheduledTriggerWrappers = new ConcurrentHashMap<>();
/**
@@ -381,9 +394,13 @@ public class ScheduledTriggers implements Closeable {
* @lucene.internal
*/
public synchronized void resumeTriggers(long afterDelayMillis) {
- scheduledTriggerWrappers.forEach((s, triggerWrapper) -> {
+ List<Map.Entry<String, TriggerWrapper>> entries = new ArrayList<>(scheduledTriggerWrappers.entrySet());
+ Collections.shuffle(entries, RANDOM);
+ entries.forEach(e -> {
+ String key = e.getKey();
+ TriggerWrapper triggerWrapper = e.getValue();
if (triggerWrapper.scheduledFuture.isCancelled()) {
- log.debug("Resuming trigger: {} after {}ms", s, afterDelayMillis);
+ log.debug("Resuming trigger: {} after {}ms", key, afterDelayMillis);
triggerWrapper.scheduledFuture = scheduledThreadPoolExecutor.scheduleWithFixedDelay(triggerWrapper, afterDelayMillis,
cloudManager.getTimeSource().convertDelay(TimeUnit.SECONDS, triggerDelay.get(), TimeUnit.MILLISECONDS), TimeUnit.MILLISECONDS);
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5016959c/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
index d091537..cd89d75 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
@@ -527,3 +527,15 @@ This trigger applies the `every` date math expression on the `startTime` or the
Apart from the common event properties described in the <<Event Types>> section, the trigger adds an additional `actualEventTime` event property which has the actual event time as opposed to the scheduled time.
For example, if the scheduled time was `2018-01-31T15:30:00Z` and grace time was `+15MINUTES` then an event may be fired at `2018-01-31T15:45:00Z`. Such an event will have `eventTime` as `2018-01-31T15:30:00Z`, the scheduled time, but the `actualEventTime` property will have a value of `2018-01-31T15:45:00Z`, the actual time.
+
+.Frequently scheduled events and trigger starvation
+[CAUTION]
+====
+Be cautious with scheduled triggers that are set to run as or more frequently than the trigger cooldown period (defaults to 5 seconds).
+
+Solr pauses all triggers for a cooldown period after a trigger fires so that the system has some time to stabilize. An aggressive scheduled trigger can starve all other triggers from
+ever executing if a new scheduled event is ready as soon as the cooldown period is over. The same starvation scenario can happen to the scheduled trigger as well.
+
+Solr randomizes the order in which the triggers are resumed after the cooldown period to mitigate this problem. However, it is recommended that scheduled triggers
+are not used with low `every` values and an external scheduling process such as cron be used for such cases instead.
+====
\ No newline at end of file