You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by dp...@apache.org on 2026/07/08 14:12:17 UTC

(stormcrawler) branch feat/1106-adaptive-host-backoff created (now c3f64948)

This is an automated email from the ASF dual-hosted git repository.

dpol1 pushed a change to branch feat/1106-adaptive-host-backoff
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git


      at c3f64948 Merge branch 'main' into feat/1106-adaptive-host-backoff

This branch includes the following new commits:

     new ca436fc9 refactor: rename HostBlockBolt to QueueRegulatorBolt
     new 38e30e74 feat: adaptive host back-off for rate-limited hosts via URLFrontier
     new c3f64948 Merge branch 'main' into feat/1106-adaptive-host-backoff

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



(stormcrawler) 02/03: feat: adaptive host back-off for rate-limited hosts via URLFrontier

Posted by dp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

dpol1 pushed a commit to branch feat/1106-adaptive-host-backoff
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git

commit 38e30e742c3ebcc8a9bb60ddddef0bd9bc3b5085
Author: Davide Polato <dp...@apache.org>
AuthorDate: Tue Jul 7 15:51:23 2026 +0200

    feat: adaptive host back-off for rate-limited hosts via URLFrontier
    
    Headerless 429/503 responses (and optionally fetch exceptions) now block
    the host's frontier queue for a growing duration: base 60s doubled per
    incident up to 24h, forgiven after 1800s of quiet counted from the end
    of the block. Explicit Retry-After keeps being honoured and feeds the
    same per-host state: a block is only ever extended, suppressed signals
    re-assert it so a lost fire-and-forget call converges, and a jitter
    fraction avoids synchronized expiries at the cap. The trigger status
    codes are configurable via urlfrontier.backoff.status.codes.
    
    Closes #1106
---
 .../apache/stormcrawler/urlfrontier/Constants.java |  61 ++++
 .../stormcrawler/urlfrontier/HostBackoff.java      | 329 ++++++++++++++++++++
 .../urlfrontier/QueueRegulatorBolt.java            | 145 ++++-----
 .../stormcrawler/urlfrontier/HostBackoffTest.java  | 330 +++++++++++++++++++++
 .../QueueRegulatorBoltDecisionTest.java            | 141 ++-------
 .../urlfrontier/QueueRegulatorBoltTest.java        |  42 +++
 6 files changed, 836 insertions(+), 212 deletions(-)

diff --git a/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/Constants.java b/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/Constants.java
index 67f5cdf4..667617bd 100644
--- a/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/Constants.java
+++ b/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/Constants.java
@@ -51,4 +51,65 @@ public final class Constants {
     public static final String URLFRONTIER_MAX_RETRY_AFTER_KEY = "urlfrontier.max.retry.after";
 
     public static final int URLFRONTIER_MAX_RETRY_AFTER_DEFAULT = 86400;
+
+    // QueueRegulatorBolt adaptive back-off (#867, #1106)
+
+    /**
+     * First block duration in seconds applied when a host rate-limits without a usable Retry-After
+     * header. Defaults to 60; 0 disables the adaptive back-off while keeping the explicit
+     * Retry-After handling.
+     */
+    public static final String URLFRONTIER_BACKOFF_BASE_KEY = "urlfrontier.backoff.base.secs";
+
+    public static final int URLFRONTIER_BACKOFF_BASE_DEFAULT = 60;
+
+    /**
+     * Multiplicative increase applied to the block duration on each new pressure event for the same
+     * host. Defaults to 2.
+     */
+    public static final String URLFRONTIER_BACKOFF_FACTOR_KEY = "urlfrontier.backoff.factor";
+
+    public static final float URLFRONTIER_BACKOFF_FACTOR_DEFAULT = 2.0f;
+
+    /**
+     * Upper bound in seconds for the escalated block duration. Defaults to 86400 (24h), the same
+     * spirit as {@link #URLFRONTIER_MAX_RETRY_AFTER_KEY} for the explicit-header case.
+     */
+    public static final String URLFRONTIER_BACKOFF_MAX_KEY = "urlfrontier.backoff.max.secs";
+
+    public static final int URLFRONTIER_BACKOFF_MAX_DEFAULT = 86400;
+
+    /**
+     * Quiet window in seconds, counted from the end of the host's block, after which its escalation
+     * is forgotten and the next pressure event starts again from the base duration. Defaults to
+     * 1800, matching Nutch's {@code fetcher.exceptions.per.queue.clear.after}.
+     */
+    public static final String URLFRONTIER_BACKOFF_DECAY_KEY = "urlfrontier.backoff.decay.secs";
+
+    public static final int URLFRONTIER_BACKOFF_DECAY_DEFAULT = 1800;
+
+    /**
+     * Whether fetch exceptions (timeouts, connection failures) count as pressure events and
+     * escalate the host's block like a headerless rate-limit response. Defaults to false.
+     */
+    public static final String URLFRONTIER_BACKOFF_ON_EXCEPTIONS_KEY =
+            "urlfrontier.backoff.on.exceptions";
+
+    public static final boolean URLFRONTIER_BACKOFF_ON_EXCEPTIONS_DEFAULT = false;
+
+    /**
+     * Random extra fraction added to every computed block duration, so that hosts blocked on the
+     * same schedule do not all expire together. Defaults to 0.1; 0 disables jitter.
+     */
+    public static final String URLFRONTIER_BACKOFF_JITTER_KEY = "urlfrontier.backoff.jitter";
+
+    public static final float URLFRONTIER_BACKOFF_JITTER_DEFAULT = 0.1f;
+
+    /**
+     * HTTP status codes treated as a rate-limit signal by the {@link QueueRegulatorBolt}: with a usable
+     * Retry-After header the requested delay is honoured, without one the adaptive back-off
+     * escalates. Defaults to 429 and 503.
+     */
+    public static final String URLFRONTIER_BACKOFF_STATUS_CODES_KEY =
+            "urlfrontier.backoff.status.codes";
 }
diff --git a/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/HostBackoff.java b/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/HostBackoff.java
new file mode 100644
index 00000000..c19c8b15
--- /dev/null
+++ b/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/HostBackoff.java
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.stormcrawler.urlfrontier;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Expiry;
+import com.github.benmanes.caffeine.cache.Ticker;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.stormcrawler.Metadata;
+import org.apache.stormcrawler.protocol.ProtocolResponse;
+import org.apache.stormcrawler.util.ConfUtils;
+import org.apache.stormcrawler.util.RetryAfterParser;
+
+/**
+ * Per-host back-off state and decision logic for the {@link QueueRegulatorBolt}: given a queue-stream
+ * tuple, decides whether and until when the host's frontier queue should be blocked.
+ *
+ * <p>Two kinds of signal are recognised, both derived from the tuple's metadata:
+ *
+ * <ul>
+ *   <li><b>explicit</b> — a rate-limit status code with a usable Retry-After header: the requested
+ *       delay is honoured, capped by {@code urlfrontier.max.retry.after};
+ *   <li><b>pressure</b> — a rate-limit status code without a usable header (or, when {@code
+ *       urlfrontier.backoff.on.exceptions} is set, a fetch exception): each pressure event blocks
+ *       the host for a growing duration, {@code base × factor^n} capped at {@code max}, the same
+ *       curve Nutch applies since 1.19 (NUTCH-2946). A host that stays quiet for {@code decay}
+ *       seconds <b>after its block has expired</b> is forgiven and restarts from {@code base} — a
+ *       blocked host is silent because it is blocked, so the quiet clock only starts once it may be
+ *       fetched again.
+ * </ul>
+ *
+ * <p>Two guards keep the escalation honest:
+ *
+ * <ul>
+ *   <li><b>one event per window</b> — pressure signals arriving while the host's block is still
+ *       active never escalate: URLs of that host already in flight when the block was raised fail
+ *       one after the other, and counting each failure would let a single incident saturate the cap
+ *       (multiple losses inside one RTT are one congestion event, RFC 5681);
+ *   <li><b>only-extend</b> — {@code blockQueueUntil} overwrites, and signals race: a block is only
+ *       ever replaced by a later one, never shrunk.
+ * </ul>
+ *
+ * <p>Signals suppressed by either guard still return the <b>stored</b> block end, so the caller
+ * re-asserts the same block at the frontier: the RPC is fire-and-forget, and a lost call would
+ * otherwise leave the frontier unblocked while this state believes the host is covered. The
+ * repeated call is idempotent and self-limiting — once the frontier applies the block, the host
+ * stops being served and the signals stop.
+ *
+ * <p>A random fraction ({@code urlfrontier.backoff.jitter}) is added to every computed duration so
+ * that hosts blocked on the same schedule do not all expire together.
+ *
+ * <p>All per-host state lives in a single cache entry that expires {@code decay} seconds after the
+ * block end, which bounds the memory to the hosts blocked recently. State is soft: it restarts
+ * empty, escalation resumes from {@code base} on the next signal. Not thread-safe; a bolt task
+ * calls {@link #blockUntilFor} from a single thread.
+ */
+final class HostBackoff {
+
+    /** Metadata key set by the FetcherBolt with the HTTP status code of the fetch. */
+    static final String STATUS_CODE_KEY = "fetch.statusCode";
+
+    /**
+     * Metadata key set by the FetcherBolt with the reason of a fetch failure. The FetcherBolt
+     * removes it before every fetch, so its presence means the last cycle failed — unlike {@link
+     * #STATUS_CODE_KEY}, which lingers from the previous response and is stale on such tuples.
+     */
+    static final String EXCEPTION_KEY = "fetch.exception";
+
+    /** Name of the Retry-After HTTP header, lower-cased as stored by the protocol layer. */
+    static final String RETRY_AFTER_HEADER = "retry-after";
+
+    /**
+     * Queue key used by the status updaters when no partition key can be derived from a URL. The
+     * queue is shared by unrelated URLs, so it is never blocked.
+     */
+    static final String DEFAULT_QUEUE_KEY = "_DEFAULT_";
+
+    private static final Set<String> DEFAULT_STATUS_CODES = Set.of("429", "503");
+
+    /**
+     * Back-off state of one host: the escalated duration reached so far, the end of the current (or
+     * last) block, and the entry's lifetime computed when it was written — block remainder plus the
+     * decay window, so the escalation is forgotten {@code decay} seconds after the block ends, and
+     * the block guard can never be evicted while the block runs.
+     */
+    private record HostState(long backoffSecs, long blockUntilSecs, long ttlSecs) {}
+
+    private final Cache<String, HostState> hosts;
+
+    /** Metadata key holding the Retry-After header, including the protocol prefix. */
+    private final String retryAfterKey;
+
+    /** Upper bound in ms for the honoured Retry-After delay; -1 means no cap. */
+    private final long maxRetryAfterMs;
+
+    private final Set<String> statusCodes;
+    private final long baseSecs;
+    private final float factor;
+    private final long maxSecs;
+    private final long decaySecs;
+    private final boolean onExceptions;
+    private final float jitter;
+    private final Random jitterSource;
+
+    HostBackoff(Map<String, Object> conf) {
+        this(conf, Ticker.systemTicker(), new Random());
+    }
+
+    /** Visible for tests: a controllable ticker drives the expiry, a seeded source the jitter. */
+    HostBackoff(Map<String, Object> conf, Ticker ticker, Random jitterSource) {
+        // the protocol layer stores response headers in the metadata with this
+        // prefix; the FetcherBolt merges them into the status metadata
+        this.retryAfterKey =
+                ConfUtils.getString(conf, ProtocolResponse.PROTOCOL_MD_PREFIX_PARAM, "")
+                        + RETRY_AFTER_HEADER;
+        long maxRetryAfterSecs =
+                ConfUtils.getLong(
+                        conf,
+                        Constants.URLFRONTIER_MAX_RETRY_AFTER_KEY,
+                        Constants.URLFRONTIER_MAX_RETRY_AFTER_DEFAULT);
+        this.maxRetryAfterMs = maxRetryAfterSecs < 0 ? -1L : maxRetryAfterSecs * 1000L;
+        this.statusCodes = statusCodesFromConf(conf);
+        this.baseSecs =
+                ConfUtils.getLong(
+                        conf,
+                        Constants.URLFRONTIER_BACKOFF_BASE_KEY,
+                        Constants.URLFRONTIER_BACKOFF_BASE_DEFAULT);
+        // a factor below 1 would de-escalate, negative jitter would shrink
+        // blocks, a negative decay would evict a running block: clamp all
+        // three rather than let a typo break the invariants silently
+        this.factor =
+                Math.max(
+                        1f,
+                        ConfUtils.getFloat(
+                                conf,
+                                Constants.URLFRONTIER_BACKOFF_FACTOR_KEY,
+                                Constants.URLFRONTIER_BACKOFF_FACTOR_DEFAULT));
+        this.maxSecs =
+                ConfUtils.getLong(
+                        conf,
+                        Constants.URLFRONTIER_BACKOFF_MAX_KEY,
+                        Constants.URLFRONTIER_BACKOFF_MAX_DEFAULT);
+        this.decaySecs =
+                Math.max(
+                        0L,
+                        ConfUtils.getLong(
+                                conf,
+                                Constants.URLFRONTIER_BACKOFF_DECAY_KEY,
+                                Constants.URLFRONTIER_BACKOFF_DECAY_DEFAULT));
+        this.onExceptions =
+                ConfUtils.getBoolean(
+                        conf,
+                        Constants.URLFRONTIER_BACKOFF_ON_EXCEPTIONS_KEY,
+                        Constants.URLFRONTIER_BACKOFF_ON_EXCEPTIONS_DEFAULT);
+        this.jitter =
+                Math.max(
+                        0f,
+                        ConfUtils.getFloat(
+                                conf,
+                                Constants.URLFRONTIER_BACKOFF_JITTER_KEY,
+                                Constants.URLFRONTIER_BACKOFF_JITTER_DEFAULT));
+        this.jitterSource = jitterSource;
+        this.hosts =
+                Caffeine.newBuilder()
+                        .expireAfter(
+                                new Expiry<String, HostState>() {
+                                    @Override
+                                    public long expireAfterCreate(
+                                            String key, HostState state, long currentTime) {
+                                        return TimeUnit.SECONDS.toNanos(state.ttlSecs());
+                                    }
+
+                                    @Override
+                                    public long expireAfterUpdate(
+                                            String key,
+                                            HostState state,
+                                            long currentTime,
+                                            long currentDuration) {
+                                        return TimeUnit.SECONDS.toNanos(state.ttlSecs());
+                                    }
+
+                                    @Override
+                                    public long expireAfterRead(
+                                            String key,
+                                            HostState state,
+                                            long currentTime,
+                                            long currentDuration) {
+                                        return currentDuration;
+                                    }
+                                })
+                        .ticker(ticker)
+                        .build();
+    }
+
+    /**
+     * Decides whether a queue-stream tuple calls for blocking the host's frontier queue, updating
+     * the per-host state accordingly.
+     *
+     * @return the absolute time to block the queue until, in epoch seconds — either a new (or
+     *     extended) block, or the unchanged stored one re-asserted so a previously lost call
+     *     converges — or {@code -1} if the tuple carries no back-off signal
+     */
+    long blockUntilFor(String key, Metadata metadata, long nowMs) {
+        if (metadata == null || DEFAULT_QUEUE_KEY.equals(key)) {
+            return -1L;
+        }
+        long explicitDelayMs = -1L;
+        if (metadata.getFirstValue(EXCEPTION_KEY) != null) {
+            // the last fetch failed with an exception: any status code (and
+            // Retry-After header) in the metadata is left over from a
+            // previous response, so only the pressure path may apply
+            if (!onExceptions) {
+                return -1L;
+            }
+        } else {
+            final String statusCode = metadata.getFirstValue(STATUS_CODE_KEY);
+            if (statusCode == null || !statusCodes.contains(statusCode)) {
+                return -1L;
+            }
+            explicitDelayMs = RetryAfterParser.parseDelay(metadata.getFirstValue(retryAfterKey));
+            if (explicitDelayMs > 0 && maxRetryAfterMs >= 0 && explicitDelayMs > maxRetryAfterMs) {
+                explicitDelayMs = maxRetryAfterMs;
+            }
+        }
+
+        final long nowSecs = nowMs / 1000L;
+        final HostState state = hosts.getIfPresent(key);
+        final long currentBlockEnd = state == null ? -1L : state.blockUntilSecs();
+        final boolean blockActive = currentBlockEnd > nowSecs;
+
+        final long delayMs;
+        long backoffSecs = state == null ? 0L : state.backoffSecs();
+        if (explicitDelayMs > 0) {
+            // explicit server signal: always considered, only-extend decides
+            delayMs = explicitDelayMs;
+        } else {
+            // pressure: while the block is active this is the same congestion
+            // event — do not escalate, re-assert the stored block instead
+            if (blockActive) {
+                return currentBlockEnd;
+            }
+            backoffSecs =
+                    backoffSecs == 0L
+                            ? baseSecs
+                            : (long) Math.min(backoffSecs * factor, (double) maxSecs);
+            if (backoffSecs <= 0) {
+                return -1L;
+            }
+            // clamp: an absurdly large configured duration must not wrap
+            // negative and silently skip the block
+            delayMs = backoffSecs > Long.MAX_VALUE / 1000L ? Long.MAX_VALUE : backoffSecs * 1000L;
+        }
+
+        final long candidate = jitteredBlockUntilSecs(delayMs, nowMs);
+        // only-extend: never shrink an active block, whatever kind set it —
+        // but re-assert it, in case the call that raised it was lost
+        if (candidate <= currentBlockEnd) {
+            return blockActive ? currentBlockEnd : -1L;
+        }
+        hosts.put(key, new HostState(backoffSecs, candidate, entryTtlSecs(candidate, nowSecs)));
+        return candidate;
+    }
+
+    /** Applies the jitter fraction to the delay and turns it into an epoch-seconds block end. */
+    private long jitteredBlockUntilSecs(long delayMs, long nowMs) {
+        final double jittered = delayMs * (1.0d + jitter * jitterSource.nextDouble());
+        final long delay = jittered >= (double) Long.MAX_VALUE ? Long.MAX_VALUE : (long) jittered;
+        long blockUntilMs;
+        try {
+            blockUntilMs = Math.addExact(nowMs, delay);
+        } catch (ArithmeticException e) {
+            blockUntilMs = Long.MAX_VALUE;
+        }
+        return blockUntilMs / 1000L;
+    }
+
+    /** Entry lifetime: the remainder of the block plus the decay window, saturated. */
+    private long entryTtlSecs(long blockUntilSecs, long nowSecs) {
+        final long remainder = blockUntilSecs - nowSecs;
+        final long ttl = remainder + decaySecs;
+        return ttl < remainder ? Long.MAX_VALUE : ttl;
+    }
+
+    private static Set<String> statusCodesFromConf(Map<String, Object> conf) {
+        final Object value = conf.get(Constants.URLFRONTIER_BACKOFF_STATUS_CODES_KEY);
+        if (value == null) {
+            return DEFAULT_STATUS_CODES;
+        }
+        final Set<String> codes = new HashSet<>();
+        if (value instanceof Collection<?> values) {
+            // the codes are usually parsed from YAML as numbers
+            for (Object code : values) {
+                codes.add(String.valueOf(code));
+            }
+        } else {
+            codes.add(String.valueOf(value));
+        }
+        return Set.copyOf(codes);
+    }
+
+    String retryAfterKey() {
+        return retryAfterKey;
+    }
+
+    boolean onExceptions() {
+        return onExceptions;
+    }
+}
diff --git a/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBolt.java b/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBolt.java
index 5dfed9dd..d7260a2f 100644
--- a/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBolt.java
+++ b/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBolt.java
@@ -36,25 +36,42 @@ import org.apache.storm.topology.OutputFieldsDeclarer;
 import org.apache.storm.topology.base.BaseRichBolt;
 import org.apache.storm.tuple.Tuple;
 import org.apache.stormcrawler.Metadata;
-import org.apache.stormcrawler.protocol.ProtocolResponse;
 import org.apache.stormcrawler.util.ConfUtils;
 import org.apache.stormcrawler.util.MetadataTransfer;
-import org.apache.stormcrawler.util.RetryAfterParser;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * Consumes the {@code queue} stream emitted by the status updater and blocks a queue in URLFrontier
- * via {@code blockQueueUntil} whenever the tuple's metadata reports a rate-limit response (HTTP 429
- * or 503) carrying a <a
- * href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After">Retry-After</a>
- * header. See issues #867 and #784.
+ * Regulates a URLFrontier queue from the {@code queue} stream emitted by the status updater:
+ * blocks it via {@code blockQueueUntil} whenever the tuple's metadata reports a rate-limit
+ * response. See issues #867, #784 and #1106.
+ *
+ * <p>Two cases are handled, both gated on the status codes configured with {@code
+ * urlfrontier.backoff.status.codes} (default 429 and 503):
+ *
+ * <ul>
+ *   <li>the response carries a usable <a
+ *       href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After">Retry-After</a>
+ *       header: the requested delay is honoured, capped by {@code urlfrontier.max.retry.after} (in
+ *       seconds, default 86400, -1 to disable the cap);
+ *   <li>the response carries no usable header — the common case in the wild: the host is blocked
+ *       for a growing duration, starting at {@code urlfrontier.backoff.base.secs} (default 60) and
+ *       multiplied by {@code urlfrontier.backoff.factor} (default 2) on each new incident, capped
+ *       at {@code urlfrontier.backoff.max.secs} (default 86400). A host that stays quiet for {@code
+ *       urlfrontier.backoff.decay.secs} (default 1800) after its block has expired is forgiven and
+ *       restarts from the base. Failures of URLs already in flight when a block was raised count as
+ *       the same incident, not as new ones. With {@code urlfrontier.backoff.on.exceptions} (default
+ *       false) fetch exceptions also count as incidents.
+ * </ul>
+ *
+ * <p>A random fraction ({@code urlfrontier.backoff.jitter}, default 0.1) is added to every computed
+ * block so that hosts blocked on the same schedule do not all retry at once, and a block is only
+ * ever extended, never shrunk by a later, shorter signal.
  *
  * <p>Wire it with a fields grouping on {@code "key"} from the status updater's {@code queue}
- * stream. The {@code "key"} is the frontier queue key derived from {@code partition.url.mode} (the
- * same setting the frontier uses to assign queues), so the block targets the matching queue. The
- * honoured delay is capped by {@code urlfrontier.max.retry.after} (in seconds, default 86400, -1 to
- * disable the cap).
+ * stream: the per-host state lives in the bolt task, so all signals for a host must reach the same
+ * task. The {@code "key"} is the frontier queue key derived from {@code partition.url.mode} (the
+ * same setting the frontier uses to assign queues), so the block targets the matching queue.
  *
  * <p><b>The metadata on the queue stream is filtered by {@code metadata.persist}</b>: with the
  * default configuration neither {@code fetch.statusCode} nor the Retry-After header survive the
@@ -66,33 +83,27 @@ import org.slf4j.LoggerFactory;
  *    - protocol.retry-after
  * </pre>
  *
- * (the second entry depends on {@code protocol.md.prefix}). A warning is logged at startup when the
- * configuration would strip them.
+ * (the second entry depends on {@code protocol.md.prefix}), and {@code fetch.exception} as well
+ * when the back-off on exceptions is enabled. A warning is logged at startup when the configuration
+ * would strip them.
+ *
+ * <p>Blocking stops new hand-outs; it does not recall URLs of the host already prefetched by the
+ * spout into the topology, which still fail against the rate-limited server and reschedule via the
+ * status stream. Keep {@code urlfrontier.max.urls.per.bucket} and {@code
+ * topology.max.spout.pending} small for a block to bite quickly.
  *
  * <p>Connects via {@code urlfrontier.address}, falling back to {@code urlfrontier.host} / {@code
  * urlfrontier.port}. The block is issued with {@code local=false} and propagates to the whole
  * cluster, so with several frontier nodes a single connection to any one of them is enough.
  *
  * <p>The block is fire-and-forget with a short deadline: a failed or expired call is logged but the
- * tuple is acked anyway. A missed block means the host is fetched once more and the next 429
- * re-emits the signal.
+ * tuple is acked anyway. A missed block heals itself: every further rate-limit signal for the host
+ * re-asserts the stored block until the frontier stops serving it.
  */
 public class QueueRegulatorBolt extends BaseRichBolt {
 
     private static final Logger LOG = LoggerFactory.getLogger(QueueRegulatorBolt.class);
 
-    /** Metadata key set by the FetcherBolt with the HTTP status code of the fetch. */
-    private static final String STATUS_CODE_KEY = "fetch.statusCode";
-
-    /** Name of the Retry-After HTTP header, lower-cased as stored by the protocol layer. */
-    private static final String RETRY_AFTER_HEADER = "retry-after";
-
-    /**
-     * Queue key used by the status updaters when no partition key can be derived from a URL. The
-     * queue is shared by unrelated URLs, so it is never blocked.
-     */
-    private static final String DEFAULT_QUEUE_KEY = "_DEFAULT_";
-
     /**
      * Deadline for the fire-and-forget blockQueueUntil call, so a frontier outage cannot pile up
      * pending RPCs and flush stale blocks on reconnect.
@@ -118,32 +129,20 @@ public class QueueRegulatorBolt extends BaseRichBolt {
     private URLFrontierStub frontier;
     private String globalCrawlID;
 
-    /** Metadata key holding the Retry-After header, including the protocol prefix. */
-    private String retryAfterKey;
-
-    /** Upper bound in ms for the honoured Retry-After delay; -1 means no cap. */
-    private long maxRetryAfterMs;
+    /** Per-host back-off state and decision logic. */
+    private HostBackoff backoff;
 
     @Override
     public void prepare(Map<String, Object> conf, TopologyContext context, OutputCollector c) {
         this.collector = c;
         this.globalCrawlID =
                 ConfUtils.getString(conf, Constants.URLFRONTIER_CRAWL_ID_KEY, CrawlID.DEFAULT);
-        // the protocol layer stores response headers in the metadata with this
-        // prefix; the FetcherBolt merges them into the status metadata
-        this.retryAfterKey =
-                ConfUtils.getString(conf, ProtocolResponse.PROTOCOL_MD_PREFIX_PARAM, "")
-                        + RETRY_AFTER_HEADER;
-        long maxRetryAfterSecs =
-                ConfUtils.getLong(
-                        conf,
-                        Constants.URLFRONTIER_MAX_RETRY_AFTER_KEY,
-                        Constants.URLFRONTIER_MAX_RETRY_AFTER_DEFAULT);
-        this.maxRetryAfterMs = maxRetryAfterSecs < 0 ? -1L : maxRetryAfterSecs * 1000L;
+        this.backoff = new HostBackoff(conf);
         // the status updater emits the queue stream after the metadata.persist
         // filter: warn upfront when the keys this bolt relies on would be
         // stripped, as the bolt would otherwise silently never block anything
-        Set<String> missing = missingQueueStreamKeys(conf, retryAfterKey);
+        Set<String> missing =
+                missingQueueStreamKeys(conf, backoff.retryAfterKey(), backoff.onExceptions());
         if (!missing.isEmpty()) {
             LOG.warn(
                     "{} do(es) not survive the metadata.persist filter: the queue stream will"
@@ -179,9 +178,7 @@ public class QueueRegulatorBolt extends BaseRichBolt {
     public void execute(Tuple t) {
         final String key = t.getStringByField("key");
         final Metadata metadata = (Metadata) t.getValueByField("metadata");
-        final long blockUntil =
-                blockUntilFor(
-                        key, metadata, retryAfterKey, maxRetryAfterMs, System.currentTimeMillis());
+        final long blockUntil = backoff.blockUntilFor(key, metadata, System.currentTimeMillis());
         if (blockUntil > 0) {
             LOG.debug("Blocking queue {} until {}", key, blockUntil);
             BlockQueueParams params =
@@ -197,60 +194,28 @@ public class QueueRegulatorBolt extends BaseRichBolt {
         collector.ack(t);
     }
 
-    /**
-     * Decides whether a queue-stream tuple carries a server-requested back-off worth enforcing.
-     * Only a rate-limit (429) or unavailable (503) response with a valid Retry-After header
-     * qualifies; the requested delay is capped by {@code maxRetryAfterMs} unless negative. The
-     * shared {@code _DEFAULT_} queue is never blocked.
-     *
-     * @return the absolute time to block the queue until, in epoch seconds, or {@code -1} if the
-     *     tuple does not call for a block
-     */
-    static long blockUntilFor(
-            String key, Metadata metadata, String retryAfterKey, long maxRetryAfterMs, long nowMs) {
-        if (metadata == null || DEFAULT_QUEUE_KEY.equals(key)) {
-            return -1L;
-        }
-        final String statusCode = metadata.getFirstValue(STATUS_CODE_KEY);
-        // only on a rate-limit (429) or unavailable (503) response does
-        // Retry-After signal a host back-off worth acting on
-        if (!"429".equals(statusCode) && !"503".equals(statusCode)) {
-            return -1L;
-        }
-        long retryAfterMs = RetryAfterParser.parseDelay(metadata.getFirstValue(retryAfterKey));
-        if (retryAfterMs <= 0) {
-            return -1L;
-        }
-        if (maxRetryAfterMs >= 0 && retryAfterMs > maxRetryAfterMs) {
-            retryAfterMs = maxRetryAfterMs;
-        }
-        long blockUntilMs;
-        try {
-            blockUntilMs = Math.addExact(nowMs, retryAfterMs);
-        } catch (ArithmeticException e) {
-            // uncapped delay pushed the block time past Long.MAX_VALUE:
-            // clamp instead of wrapping negative and skipping the block
-            blockUntilMs = Long.MAX_VALUE;
-        }
-        return blockUntilMs / 1000L;
-    }
-
     /**
      * Returns the queue-stream keys this bolt relies on that would not survive the {@code
-     * metadata.persist} filter applied by the status updater before emitting.
+     * metadata.persist} filter applied by the status updater before emitting. The {@code
+     * fetch.exception} key only matters when the back-off on exceptions is enabled.
      */
-    static Set<String> missingQueueStreamKeys(Map<String, Object> conf, String retryAfterKey) {
+    static Set<String> missingQueueStreamKeys(
+            Map<String, Object> conf, String retryAfterKey, boolean onExceptions) {
         Metadata probe = new Metadata();
-        probe.setValue(STATUS_CODE_KEY, "429");
+        probe.setValue(HostBackoff.STATUS_CODE_KEY, "429");
         probe.setValue(retryAfterKey, "1");
+        probe.setValue(HostBackoff.EXCEPTION_KEY, "probe");
         Metadata filtered = MetadataTransfer.getInstance(conf).filter(probe);
         Set<String> missing = new LinkedHashSet<>();
-        if (filtered.getFirstValue(STATUS_CODE_KEY) == null) {
-            missing.add(STATUS_CODE_KEY);
+        if (filtered.getFirstValue(HostBackoff.STATUS_CODE_KEY) == null) {
+            missing.add(HostBackoff.STATUS_CODE_KEY);
         }
         if (filtered.getFirstValue(retryAfterKey) == null) {
             missing.add(retryAfterKey);
         }
+        if (onExceptions && filtered.getFirstValue(HostBackoff.EXCEPTION_KEY) == null) {
+            missing.add(HostBackoff.EXCEPTION_KEY);
+        }
         return missing;
     }
 
diff --git a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBackoffTest.java b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBackoffTest.java
new file mode 100644
index 00000000..0ac63c9f
--- /dev/null
+++ b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBackoffTest.java
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.stormcrawler.urlfrontier;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.github.benmanes.caffeine.cache.Ticker;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import org.apache.stormcrawler.Metadata;
+import org.apache.stormcrawler.protocol.ProtocolResponse;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Unit coverage for {@link HostBackoff} - the per-host decision whether a queue-stream tuple calls
+ * for blocking the host's frontier queue, and until when. Time is driven by a controllable ticker
+ * (escalation decay) and an explicit clock (block ends); jitter is pinned to 0 except where it is
+ * the behaviour under test.
+ */
+class HostBackoffTest {
+
+    private static final String KEY = "example.com";
+    private static final String RETRY_AFTER_KEY = "protocol.retry-after";
+    private static final long START_MS = 1_700_000_000_000L;
+    private static final long BASE_SECS = Constants.URLFRONTIER_BACKOFF_BASE_DEFAULT;
+
+    /** Both the decay ticker and the block clock, advanced together like wall time. */
+    private static final class FakeClock implements Ticker {
+        private long nowMs = START_MS;
+
+        @Override
+        public long read() {
+            return TimeUnit.MILLISECONDS.toNanos(nowMs);
+        }
+
+        void advanceSecs(long secs) {
+            nowMs += secs * 1000L;
+        }
+    }
+
+    private static final class FixedRandom extends Random {
+        private final double value;
+
+        FixedRandom(double value) {
+            this.value = value;
+        }
+
+        @Override
+        public double nextDouble() {
+            return value;
+        }
+    }
+
+    private final FakeClock clock = new FakeClock();
+
+    private static Map<String, Object> conf(Object... keysAndValues) {
+        Map<String, Object> conf = new HashMap<>();
+        conf.put(ProtocolResponse.PROTOCOL_MD_PREFIX_PARAM, "protocol.");
+        conf.put(Constants.URLFRONTIER_BACKOFF_JITTER_KEY, 0.0);
+        for (int i = 0; i < keysAndValues.length; i += 2) {
+            conf.put((String) keysAndValues[i], keysAndValues[i + 1]);
+        }
+        return conf;
+    }
+
+    private HostBackoff backoff(Map<String, Object> conf) {
+        return new HostBackoff(conf, clock, new FixedRandom(0.0d));
+    }
+
+    private static Metadata md(String statusCode, String retryAfter) {
+        Metadata md = new Metadata();
+        if (statusCode != null) {
+            md.setValue(HostBackoff.STATUS_CODE_KEY, statusCode);
+        }
+        if (retryAfter != null) {
+            md.setValue(RETRY_AFTER_KEY, retryAfter);
+        }
+        return md;
+    }
+
+    private long expectedAt(long delaySecs) {
+        return (clock.nowMs + delaySecs * 1000L) / 1000L;
+    }
+
+    private long signal(HostBackoff backoff, Metadata metadata) {
+        return backoff.blockUntilFor(KEY, metadata, clock.nowMs);
+    }
+
+    // --- explicit Retry-After path ---
+
+    @Test
+    void explicitRetryAfterHonoured() {
+        HostBackoff backoff = backoff(conf());
+        assertEquals(expectedAt(120), signal(backoff, md("429", "120")));
+    }
+
+    @Test
+    void explicitRetryAfterCappedAtConfiguredMaximum() {
+        HostBackoff backoff = backoff(conf(Constants.URLFRONTIER_MAX_RETRY_AFTER_KEY, 3600));
+        assertEquals(expectedAt(3600), signal(backoff, md("429", "86400")));
+    }
+
+    @Test
+    void negativeCapMeansUncapped() {
+        HostBackoff backoff = backoff(conf(Constants.URLFRONTIER_MAX_RETRY_AFTER_KEY, -1));
+        assertEquals(expectedAt(86400), signal(backoff, md("503", "86400")));
+    }
+
+    @Test
+    void hugeUncappedDelayClampsInsteadOfOverflowing() {
+        // 9223372036854775 seconds survive the parser's multiplyExact (just
+        // below Long.MAX_VALUE in ms); the addition of now must clamp, not
+        // wrap negative and silently skip the block (regression)
+        HostBackoff backoff = backoff(conf(Constants.URLFRONTIER_MAX_RETRY_AFTER_KEY, -1));
+        long blockUntil = signal(backoff, md("429", "9223372036854775"));
+        assertEquals(Long.MAX_VALUE / 1000L, blockUntil);
+    }
+
+    // --- pressure path: headerless rate limits ---
+
+    @Test
+    void headerlessRateLimitBlocksForBase() {
+        HostBackoff backoff = backoff(conf());
+        assertEquals(expectedAt(BASE_SECS), signal(backoff, md("429", null)));
+    }
+
+    @Test
+    void malformedHeaderEscalatesLikeHeaderless() {
+        // Phase 1 ignored a 429 with an unparseable header; the server still
+        // rate-limited, so it now feeds the adaptive back-off instead
+        HostBackoff backoff = backoff(conf());
+        assertEquals(expectedAt(BASE_SECS), signal(backoff, md("429", "not-a-date")));
+        clock.advanceSecs(BASE_SECS + 1);
+        // a zero delay is no reason to skip the back-off either
+        assertEquals(expectedAt(BASE_SECS * 2), signal(backoff, md("429", "0")));
+    }
+
+    @Test
+    void escalationMultipliesAndCaps() {
+        HostBackoff backoff = backoff(conf(Constants.URLFRONTIER_BACKOFF_MAX_KEY, 200));
+        assertEquals(expectedAt(60), signal(backoff, md("503", null)));
+        clock.advanceSecs(61);
+        assertEquals(expectedAt(120), signal(backoff, md("503", null)));
+        clock.advanceSecs(121);
+        assertEquals(expectedAt(200), signal(backoff, md("503", null)));
+        clock.advanceSecs(201);
+        assertEquals(expectedAt(200), signal(backoff, md("503", null)));
+    }
+
+    @Test
+    void pressureDuringActiveBlockIsOneCongestionEvent() {
+        HostBackoff backoff = backoff(conf());
+        long block = signal(backoff, md("429", null));
+        assertEquals(expectedAt(60), block);
+        // the fetcher drains its internal backlog for the host: more 429s
+        // arrive while the block is active and must not escalate — they
+        // re-assert the unchanged block, in case the original call was lost
+        clock.advanceSecs(10);
+        assertEquals(block, signal(backoff, md("429", null)));
+        clock.advanceSecs(10);
+        assertEquals(block, signal(backoff, md("429", null)));
+        // after the block expires the next incident escalates exactly one
+        // step: the burst counted as a single event
+        clock.advanceSecs(41);
+        assertEquals(expectedAt(120), signal(backoff, md("429", null)));
+    }
+
+    @Test
+    void capIsReachableUnderSustainedPressure() {
+        // the quiet window counts from the END of the block: a host blocked
+        // for longer than the decay window is silent because it is blocked,
+        // not because it calmed down, so the escalation must survive
+        // (regression: an expire-after-write decay forgot the escalation
+        // during any block longer than the decay and capped the real-world
+        // curve at the first duration above it)
+        HostBackoff backoff = backoff(conf());
+        long expected = 60;
+        for (int i = 0; i < 6; i++) {
+            assertEquals(expectedAt(expected), signal(backoff, md("429", null)));
+            clock.advanceSecs(expected + 1);
+            expected = Math.min(expected * 2, 86400);
+        }
+        // the previous block (1920s) outlived the 1800s decay window: the
+        // escalation must still hold and keep climbing towards the cap
+        assertEquals(expectedAt(3840), signal(backoff, md("429", null)));
+    }
+
+    @Test
+    void quietHostIsForgivenAfterDecay() {
+        HostBackoff backoff = backoff(conf());
+        signal(backoff, md("429", null));
+        clock.advanceSecs(61);
+        assertEquals(expectedAt(120), signal(backoff, md("429", null)));
+        // quiet for longer than the decay window: escalation restarts at base
+        clock.advanceSecs(120 + Constants.URLFRONTIER_BACKOFF_DECAY_DEFAULT + 1);
+        assertEquals(expectedAt(BASE_SECS), signal(backoff, md("429", null)));
+    }
+
+    // --- only-extend: a block is never shrunk ---
+
+    @Test
+    void shorterExplicitSignalNeverShrinksActiveBlock() {
+        HostBackoff backoff = backoff(conf());
+        long first = signal(backoff, md("429", "3600"));
+        assertEquals(expectedAt(3600), first);
+        clock.advanceSecs(10);
+        // a racing shorter Retry-After must not shrink the active block: the
+        // stored end is re-asserted unchanged
+        assertEquals(first, signal(backoff, md("429", "60")));
+        assertEquals(first, signal(backoff, md("429", "3500")));
+    }
+
+    @Test
+    void longerExplicitSignalExtendsActiveBlock() {
+        HostBackoff backoff = backoff(conf());
+        assertEquals(expectedAt(BASE_SECS), signal(backoff, md("429", null)));
+        // the server now sends an explicit, longer time: honour it even while
+        // the pressure block is active
+        long extended = signal(backoff, md("429", "3600"));
+        assertEquals(expectedAt(3600), extended);
+        // and the pressure path stays silenced by the extended block, which
+        // is re-asserted as is
+        clock.advanceSecs(BASE_SECS + 1);
+        assertEquals(extended, signal(backoff, md("429", null)));
+    }
+
+    // --- signal gating ---
+
+    @Test
+    void statusCodesOutsideConfiguredSetAreIgnored() {
+        HostBackoff backoff = backoff(conf());
+        assertEquals(-1L, signal(backoff, md("200", "120")));
+        assertEquals(-1L, signal(backoff, md("301", "120")));
+        assertEquals(-1L, signal(backoff, md("403", "120")));
+        assertEquals(-1L, signal(backoff, md(null, "120")));
+    }
+
+    @Test
+    void configuredStatusCodesReplaceTheDefaults() {
+        // YAML parses the codes as numbers
+        HostBackoff backoff =
+                backoff(conf(Constants.URLFRONTIER_BACKOFF_STATUS_CODES_KEY, List.of(403)));
+        assertEquals(expectedAt(120), signal(backoff, md("403", "120")));
+        clock.advanceSecs(121);
+        assertEquals(expectedAt(BASE_SECS), signal(backoff, md("403", null)));
+        assertEquals(
+                -1L, backoff.blockUntilFor("other.example.com", md("429", "120"), clock.nowMs));
+    }
+
+    @Test
+    void defaultQueueSentinelIsNeverBlocked() {
+        // a 429 on a URL whose partition key fell back to the shared catch-all
+        // queue must not stall every unrelated URL in it
+        HostBackoff backoff = backoff(conf());
+        assertEquals(-1L, backoff.blockUntilFor("_DEFAULT_", md("429", "120"), clock.nowMs));
+    }
+
+    @Test
+    void nullMetadataIsIgnored() {
+        HostBackoff backoff = backoff(conf());
+        assertEquals(-1L, backoff.blockUntilFor(KEY, null, clock.nowMs));
+    }
+
+    // --- fetch exceptions as pressure ---
+
+    @Test
+    void exceptionsAreIgnoredByDefault() {
+        HostBackoff backoff = backoff(conf());
+        Metadata md = new Metadata();
+        md.setValue(HostBackoff.EXCEPTION_KEY, "Socket timeout fetching");
+        assertEquals(-1L, signal(backoff, md));
+    }
+
+    @Test
+    void exceptionsEscalateWhenEnabledAndStaleResponseKeysAreIgnored() {
+        HostBackoff backoff = backoff(conf(Constants.URLFRONTIER_BACKOFF_ON_EXCEPTIONS_KEY, true));
+        // the FetcherBolt clears fetch.exception before every fetch, so on an
+        // exception tuple the status code and Retry-After header are leftovers
+        // of a previous response and must not drive the decision
+        Metadata md = md("200", "3600");
+        md.setValue(HostBackoff.EXCEPTION_KEY, "Socket timeout fetching");
+        assertEquals(expectedAt(BASE_SECS), signal(backoff, md));
+    }
+
+    // --- jitter ---
+
+    @Test
+    void jitterStretchesTheBlockByAtMostTheConfiguredFraction() {
+        Map<String, Object> conf = conf(Constants.URLFRONTIER_BACKOFF_JITTER_KEY, 0.1);
+        // lower bound of the jitter range: the block is untouched
+        HostBackoff min = new HostBackoff(conf, clock, new FixedRandom(0.0d));
+        assertEquals(expectedAt(1000), min.blockUntilFor(KEY, md("429", "1000"), clock.nowMs));
+        // upper bound: the block is stretched by the full fraction
+        HostBackoff max = new HostBackoff(conf, clock, new FixedRandom(1.0d));
+        assertEquals(expectedAt(1100), max.blockUntilFor(KEY, md("429", "1000"), clock.nowMs));
+        // and the escalated path is jittered the same way
+        assertEquals(
+                expectedAt(BASE_SECS + BASE_SECS / 10),
+                max.blockUntilFor("other.example.com", md("429", null), clock.nowMs));
+    }
+
+    @Test
+    void defaultJitterStaysWithinItsBound() {
+        Map<String, Object> conf = new HashMap<>();
+        conf.put(ProtocolResponse.PROTOCOL_MD_PREFIX_PARAM, "protocol.");
+        HostBackoff backoff = new HostBackoff(conf, clock, new Random(42));
+        long blockUntil = backoff.blockUntilFor(KEY, md("429", "1000"), clock.nowMs);
+        assertTrue(blockUntil >= expectedAt(1000), "jitter must never shorten a block");
+        assertTrue(blockUntil <= expectedAt(1100), "jitter must stay within the fraction");
+    }
+}
diff --git a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltDecisionTest.java b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltDecisionTest.java
index 0e60c568..84581dfe 100644
--- a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltDecisionTest.java
+++ b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltDecisionTest.java
@@ -24,138 +24,23 @@ import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import org.apache.stormcrawler.Metadata;
 import org.junit.jupiter.api.Test;
 
 /**
- * Unit coverage for {@link QueueRegulatorBolt#blockUntilFor(String, Metadata, String, long, long)} - the
- * decision whether a queue-stream tuple carries a server-requested back-off worth enforcing - and
- * for {@link QueueRegulatorBolt#missingQueueStreamKeys(Map, String)} - the startup check on {@code
- * metadata.persist}.
+ * Unit coverage for {@link QueueRegulatorBolt#missingQueueStreamKeys(Map, String, boolean)} - the
+ * startup check that the metadata keys the bolt relies on survive the {@code metadata.persist}
+ * filter applied by the status updater. The block decision itself is covered by {@link
+ * HostBackoffTest}.
  */
 class QueueRegulatorBoltDecisionTest {
 
-    private static final String KEY = "example.com";
     private static final String RETRY_AFTER_KEY = "protocol.retry-after";
-    private static final long NOW_MS = 1_700_000_000_000L;
-    private static final long NO_CAP = -1L;
-
-    private static Metadata md(String statusCode, String retryAfter) {
-        Metadata md = new Metadata();
-        if (statusCode != null) {
-            md.setValue("fetch.statusCode", statusCode);
-        }
-        if (retryAfter != null) {
-            md.setValue(RETRY_AFTER_KEY, retryAfter);
-        }
-        return md;
-    }
-
-    @Test
-    void blocksOn429WithRetryAfterSeconds() {
-        long blockUntil =
-                QueueRegulatorBolt.blockUntilFor(KEY, md("429", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
-        assertEquals((NOW_MS + 120_000L) / 1000L, blockUntil);
-    }
-
-    @Test
-    void blocksOn503WithRetryAfterSeconds() {
-        long blockUntil =
-                QueueRegulatorBolt.blockUntilFor(KEY, md("503", "60"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
-        assertEquals((NOW_MS + 60_000L) / 1000L, blockUntil);
-    }
-
-    @Test
-    void ignoresOtherStatusCodesEvenWithHeader() {
-        assertEquals(
-                -1L,
-                QueueRegulatorBolt.blockUntilFor(
-                        KEY, md("200", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-        assertEquals(
-                -1L,
-                QueueRegulatorBolt.blockUntilFor(
-                        KEY, md("301", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-        assertEquals(
-                -1L,
-                QueueRegulatorBolt.blockUntilFor(
-                        KEY, md("403", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-    }
-
-    @Test
-    void ignores429WithoutHeader() {
-        assertEquals(
-                -1L,
-                QueueRegulatorBolt.blockUntilFor(KEY, md("429", null), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-    }
-
-    @Test
-    void ignoresMissingStatusCode() {
-        assertEquals(
-                -1L,
-                QueueRegulatorBolt.blockUntilFor(KEY, md(null, "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-    }
-
-    @Test
-    void ignoresMalformedHeaderAndZeroDelay() {
-        assertEquals(
-                -1L,
-                QueueRegulatorBolt.blockUntilFor(
-                        KEY, md("429", "not-a-date"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-        // a zero delay is not worth a frontier round-trip
-        assertEquals(
-                -1L,
-                QueueRegulatorBolt.blockUntilFor(KEY, md("429", "0"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-    }
-
-    @Test
-    void capsDelayAtConfiguredMaximum() {
-        long capMs = 3_600_000L; // one hour
-        long blockUntil =
-                QueueRegulatorBolt.blockUntilFor(
-                        KEY, md("429", "86400"), RETRY_AFTER_KEY, capMs, NOW_MS);
-        assertEquals((NOW_MS + capMs) / 1000L, blockUntil);
-    }
-
-    @Test
-    void negativeCapMeansUncapped() {
-        long blockUntil =
-                QueueRegulatorBolt.blockUntilFor(
-                        KEY, md("429", "86400"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
-        assertEquals((NOW_MS + 86_400_000L) / 1000L, blockUntil);
-    }
-
-    @Test
-    void hugeUncappedDelayClampsInsteadOfOverflowing() {
-        // 9223372036854775 seconds survive the parser's multiplyExact (just
-        // below Long.MAX_VALUE in ms); the addition of nowMs must clamp, not
-        // wrap negative and silently skip the block (regression)
-        long blockUntil =
-                QueueRegulatorBolt.blockUntilFor(
-                        KEY, md("429", "9223372036854775"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
-        assertEquals(Long.MAX_VALUE / 1000L, blockUntil);
-        assertTrue(blockUntil > NOW_MS / 1000L);
-    }
-
-    @Test
-    void defaultQueueSentinelIsNeverBlocked() {
-        // a 429 on a URL whose partition key fell back to the shared catch-all
-        // queue must not stall every unrelated URL in it
-        assertEquals(
-                -1L,
-                QueueRegulatorBolt.blockUntilFor(
-                        "_DEFAULT_", md("429", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-    }
-
-    @Test
-    void nullMetadataIsIgnored() {
-        assertEquals(-1L, QueueRegulatorBolt.blockUntilFor(KEY, null, RETRY_AFTER_KEY, NO_CAP, NOW_MS));
-    }
 
     @Test
     void missingQueueStreamKeysReportsBothWithDefaultConfig() {
         // with the default metadata.persist neither key survives the filter
         Set<String> missing =
-                QueueRegulatorBolt.missingQueueStreamKeys(new HashMap<>(), RETRY_AFTER_KEY);
+                QueueRegulatorBolt.missingQueueStreamKeys(new HashMap<>(), RETRY_AFTER_KEY, false);
         assertEquals(Set.of("fetch.statusCode", RETRY_AFTER_KEY), missing);
     }
 
@@ -163,13 +48,25 @@ class QueueRegulatorBoltDecisionTest {
     void missingQueueStreamKeysEmptyWhenBothPersisted() {
         Map<String, Object> conf = new HashMap<>();
         conf.put("metadata.persist", List.of("fetch.statusCode", RETRY_AFTER_KEY));
-        assertTrue(QueueRegulatorBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY).isEmpty());
+        assertTrue(QueueRegulatorBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY, false).isEmpty());
     }
 
     @Test
     void missingQueueStreamKeysHonoursWildcards() {
         Map<String, Object> conf = new HashMap<>();
         conf.put("metadata.persist", List.of("fetch.*", "protocol.*"));
-        assertTrue(QueueRegulatorBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY).isEmpty());
+        assertTrue(QueueRegulatorBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY, false).isEmpty());
+    }
+
+    @Test
+    void exceptionKeyOnlyRequiredWhenBackoffOnExceptionsIsEnabled() {
+        Map<String, Object> conf = new HashMap<>();
+        conf.put("metadata.persist", List.of("fetch.statusCode", RETRY_AFTER_KEY));
+        // the same configuration is complete without the exceptions gate and
+        // incomplete with it
+        assertTrue(QueueRegulatorBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY, false).isEmpty());
+        assertEquals(
+                Set.of("fetch.exception"),
+                QueueRegulatorBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY, true));
     }
 }
diff --git a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltTest.java b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltTest.java
index 4d5a219f..87a88dcc 100644
--- a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltTest.java
+++ b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltTest.java
@@ -182,4 +182,46 @@ class QueueRegulatorBoltTest {
 
         bolt.cleanup();
     }
+
+    @Test
+    @Timeout(value = 2, unit = TimeUnit.MINUTES)
+    void blocksHostQueueOnHeaderless429() {
+        seedOneUrl();
+
+        // a 429 without a Retry-After header: the adaptive back-off must
+        // block the host for the configured base duration. The bolt is
+        // configured through urlfrontier.host/port to cover that resolution
+        // path as well
+        QueueRegulatorBolt bolt = new QueueRegulatorBolt();
+        Map<String, Object> conf = frontierConfig();
+        conf.put(ProtocolResponse.PROTOCOL_MD_PREFIX_PARAM, "protocol.");
+        conf.put(Constants.URLFRONTIER_BACKOFF_BASE_KEY, 3600);
+        bolt.prepare(conf, TestUtil.getMockedTopologyContext(), mock(OutputCollector.class));
+        Tuple t = mock(Tuple.class);
+        when(t.getStringByField("key")).thenReturn(HOST);
+        Metadata md = new Metadata();
+        md.setValue("fetch.statusCode", "429");
+        when(t.getValueByField("metadata")).thenReturn(md);
+        bolt.execute(t);
+
+        // same blind delay rationale as blocksHostQueueUntilTime: getURLs
+        // must never hand the URL out, and polling too early would put it
+        // in-flight and make the assertion pass even without a block
+        await().pollDelay(Duration.ofSeconds(2)).atMost(3, TimeUnit.SECONDS).until(() -> true);
+        assertTrue(
+                keysFromGetURLs().isEmpty(),
+                "the URL was handed out although the queue should be blocked");
+
+        // unblock (a past time releases the queue) and the host becomes
+        // available again
+        blocking.blockQueueUntil(
+                BlockQueueParams.newBuilder()
+                        .setKey(HOST)
+                        .setCrawlID(CrawlID.DEFAULT)
+                        .setTime(System.currentTimeMillis() / 1000L - 1)
+                        .build());
+        await().atMost(15, TimeUnit.SECONDS).until(() -> keysFromGetURLs().contains(HOST));
+
+        bolt.cleanup();
+    }
 }


(stormcrawler) 01/03: refactor: rename HostBlockBolt to QueueRegulatorBolt

Posted by dp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

dpol1 pushed a commit to branch feat/1106-adaptive-host-backoff
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git

commit ca436fc9e55b73c6dbd3e0fec99fc45a95dc92c2
Author: Davide Polato <dp...@apache.org>
AuthorDate: Tue Jul 7 14:48:06 2026 +0200

    refactor: rename HostBlockBolt to QueueRegulatorBolt
    
    The bolt is about to regulate queues beyond blocking them: adaptive
    back-off for headerless rate limits first, robots crawl-delay pacing
    next (#867). Give it a name that covers the role. No behaviour change.
---
 ...{HostBlockBolt.java => QueueRegulatorBolt.java} |  4 +--
 ...st.java => QueueRegulatorBoltDecisionTest.java} | 40 +++++++++++-----------
 ...ckBoltTest.java => QueueRegulatorBoltTest.java} |  4 +--
 3 files changed, 24 insertions(+), 24 deletions(-)

diff --git a/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/HostBlockBolt.java b/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBolt.java
similarity index 98%
rename from external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/HostBlockBolt.java
rename to external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBolt.java
index 2ec791f6..5dfed9dd 100644
--- a/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/HostBlockBolt.java
+++ b/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBolt.java
@@ -77,9 +77,9 @@ import org.slf4j.LoggerFactory;
  * tuple is acked anyway. A missed block means the host is fetched once more and the next 429
  * re-emits the signal.
  */
-public class HostBlockBolt extends BaseRichBolt {
+public class QueueRegulatorBolt extends BaseRichBolt {
 
-    private static final Logger LOG = LoggerFactory.getLogger(HostBlockBolt.class);
+    private static final Logger LOG = LoggerFactory.getLogger(QueueRegulatorBolt.class);
 
     /** Metadata key set by the FetcherBolt with the HTTP status code of the fetch. */
     private static final String STATUS_CODE_KEY = "fetch.statusCode";
diff --git a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBlockBoltDecisionTest.java b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltDecisionTest.java
similarity index 76%
rename from external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBlockBoltDecisionTest.java
rename to external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltDecisionTest.java
index c3d40c07..0e60c568 100644
--- a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBlockBoltDecisionTest.java
+++ b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltDecisionTest.java
@@ -28,12 +28,12 @@ import org.apache.stormcrawler.Metadata;
 import org.junit.jupiter.api.Test;
 
 /**
- * Unit coverage for {@link HostBlockBolt#blockUntilFor(String, Metadata, String, long, long)} - the
+ * Unit coverage for {@link QueueRegulatorBolt#blockUntilFor(String, Metadata, String, long, long)} - the
  * decision whether a queue-stream tuple carries a server-requested back-off worth enforcing - and
- * for {@link HostBlockBolt#missingQueueStreamKeys(Map, String)} - the startup check on {@code
+ * for {@link QueueRegulatorBolt#missingQueueStreamKeys(Map, String)} - the startup check on {@code
  * metadata.persist}.
  */
-class HostBlockBoltDecisionTest {
+class QueueRegulatorBoltDecisionTest {
 
     private static final String KEY = "example.com";
     private static final String RETRY_AFTER_KEY = "protocol.retry-after";
@@ -54,14 +54,14 @@ class HostBlockBoltDecisionTest {
     @Test
     void blocksOn429WithRetryAfterSeconds() {
         long blockUntil =
-                HostBlockBolt.blockUntilFor(KEY, md("429", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
+                QueueRegulatorBolt.blockUntilFor(KEY, md("429", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
         assertEquals((NOW_MS + 120_000L) / 1000L, blockUntil);
     }
 
     @Test
     void blocksOn503WithRetryAfterSeconds() {
         long blockUntil =
-                HostBlockBolt.blockUntilFor(KEY, md("503", "60"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
+                QueueRegulatorBolt.blockUntilFor(KEY, md("503", "60"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
         assertEquals((NOW_MS + 60_000L) / 1000L, blockUntil);
     }
 
@@ -69,15 +69,15 @@ class HostBlockBoltDecisionTest {
     void ignoresOtherStatusCodesEvenWithHeader() {
         assertEquals(
                 -1L,
-                HostBlockBolt.blockUntilFor(
+                QueueRegulatorBolt.blockUntilFor(
                         KEY, md("200", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
         assertEquals(
                 -1L,
-                HostBlockBolt.blockUntilFor(
+                QueueRegulatorBolt.blockUntilFor(
                         KEY, md("301", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
         assertEquals(
                 -1L,
-                HostBlockBolt.blockUntilFor(
+                QueueRegulatorBolt.blockUntilFor(
                         KEY, md("403", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
     }
 
@@ -85,33 +85,33 @@ class HostBlockBoltDecisionTest {
     void ignores429WithoutHeader() {
         assertEquals(
                 -1L,
-                HostBlockBolt.blockUntilFor(KEY, md("429", null), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
+                QueueRegulatorBolt.blockUntilFor(KEY, md("429", null), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
     }
 
     @Test
     void ignoresMissingStatusCode() {
         assertEquals(
                 -1L,
-                HostBlockBolt.blockUntilFor(KEY, md(null, "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
+                QueueRegulatorBolt.blockUntilFor(KEY, md(null, "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
     }
 
     @Test
     void ignoresMalformedHeaderAndZeroDelay() {
         assertEquals(
                 -1L,
-                HostBlockBolt.blockUntilFor(
+                QueueRegulatorBolt.blockUntilFor(
                         KEY, md("429", "not-a-date"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
         // a zero delay is not worth a frontier round-trip
         assertEquals(
                 -1L,
-                HostBlockBolt.blockUntilFor(KEY, md("429", "0"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
+                QueueRegulatorBolt.blockUntilFor(KEY, md("429", "0"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
     }
 
     @Test
     void capsDelayAtConfiguredMaximum() {
         long capMs = 3_600_000L; // one hour
         long blockUntil =
-                HostBlockBolt.blockUntilFor(
+                QueueRegulatorBolt.blockUntilFor(
                         KEY, md("429", "86400"), RETRY_AFTER_KEY, capMs, NOW_MS);
         assertEquals((NOW_MS + capMs) / 1000L, blockUntil);
     }
@@ -119,7 +119,7 @@ class HostBlockBoltDecisionTest {
     @Test
     void negativeCapMeansUncapped() {
         long blockUntil =
-                HostBlockBolt.blockUntilFor(
+                QueueRegulatorBolt.blockUntilFor(
                         KEY, md("429", "86400"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
         assertEquals((NOW_MS + 86_400_000L) / 1000L, blockUntil);
     }
@@ -130,7 +130,7 @@ class HostBlockBoltDecisionTest {
         // below Long.MAX_VALUE in ms); the addition of nowMs must clamp, not
         // wrap negative and silently skip the block (regression)
         long blockUntil =
-                HostBlockBolt.blockUntilFor(
+                QueueRegulatorBolt.blockUntilFor(
                         KEY, md("429", "9223372036854775"), RETRY_AFTER_KEY, NO_CAP, NOW_MS);
         assertEquals(Long.MAX_VALUE / 1000L, blockUntil);
         assertTrue(blockUntil > NOW_MS / 1000L);
@@ -142,20 +142,20 @@ class HostBlockBoltDecisionTest {
         // queue must not stall every unrelated URL in it
         assertEquals(
                 -1L,
-                HostBlockBolt.blockUntilFor(
+                QueueRegulatorBolt.blockUntilFor(
                         "_DEFAULT_", md("429", "120"), RETRY_AFTER_KEY, NO_CAP, NOW_MS));
     }
 
     @Test
     void nullMetadataIsIgnored() {
-        assertEquals(-1L, HostBlockBolt.blockUntilFor(KEY, null, RETRY_AFTER_KEY, NO_CAP, NOW_MS));
+        assertEquals(-1L, QueueRegulatorBolt.blockUntilFor(KEY, null, RETRY_AFTER_KEY, NO_CAP, NOW_MS));
     }
 
     @Test
     void missingQueueStreamKeysReportsBothWithDefaultConfig() {
         // with the default metadata.persist neither key survives the filter
         Set<String> missing =
-                HostBlockBolt.missingQueueStreamKeys(new HashMap<>(), RETRY_AFTER_KEY);
+                QueueRegulatorBolt.missingQueueStreamKeys(new HashMap<>(), RETRY_AFTER_KEY);
         assertEquals(Set.of("fetch.statusCode", RETRY_AFTER_KEY), missing);
     }
 
@@ -163,13 +163,13 @@ class HostBlockBoltDecisionTest {
     void missingQueueStreamKeysEmptyWhenBothPersisted() {
         Map<String, Object> conf = new HashMap<>();
         conf.put("metadata.persist", List.of("fetch.statusCode", RETRY_AFTER_KEY));
-        assertTrue(HostBlockBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY).isEmpty());
+        assertTrue(QueueRegulatorBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY).isEmpty());
     }
 
     @Test
     void missingQueueStreamKeysHonoursWildcards() {
         Map<String, Object> conf = new HashMap<>();
         conf.put("metadata.persist", List.of("fetch.*", "protocol.*"));
-        assertTrue(HostBlockBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY).isEmpty());
+        assertTrue(QueueRegulatorBolt.missingQueueStreamKeys(conf, RETRY_AFTER_KEY).isEmpty());
     }
 }
diff --git a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBlockBoltTest.java b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltTest.java
similarity index 98%
rename from external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBlockBoltTest.java
rename to external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltTest.java
index 8f8afb6f..4d5a219f 100644
--- a/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/HostBlockBoltTest.java
+++ b/external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/QueueRegulatorBoltTest.java
@@ -50,7 +50,7 @@ import org.junit.jupiter.api.Timeout;
 import org.testcontainers.junit.jupiter.Testcontainers;
 
 @Testcontainers(disabledWithoutDocker = true)
-class HostBlockBoltTest {
+class QueueRegulatorBoltTest {
 
     private static final String HOST = "example.com";
 
@@ -143,7 +143,7 @@ class HostBlockBoltTest {
         // tuple whose metadata reports a 429 with a Retry-After of one hour.
         // The bolt is configured through urlfrontier.address to cover the
         // address resolution as well
-        HostBlockBolt bolt = new HostBlockBolt();
+        QueueRegulatorBolt bolt = new QueueRegulatorBolt();
         var connection = container.getFrontierConnection();
         Map<String, Object> conf = new HashMap<>();
         conf.put(


(stormcrawler) 03/03: Merge branch 'main' into feat/1106-adaptive-host-backoff

Posted by dp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

dpol1 pushed a commit to branch feat/1106-adaptive-host-backoff
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git

commit c3f6494851af69a7ade687c0c05907ebc474aeee
Merge: 38e30e74 b60589b2
Author: Davide Polato <dp...@apache.org>
AuthorDate: Wed Jul 8 14:14:21 2026 +0200

    Merge branch 'main' into feat/1106-adaptive-host-backoff

 THIRD-PARTY.txt                                    | 129 +++++++-------
 external/ai/pom.xml                                |   2 +-
 external/aws/pom.xml                               |   2 +-
 .../opensearch/persistence/QueueBolt.java          | 191 +++++++++++++++++++++
 .../opensearch/persistence/StatusUpdaterBolt.java  |   7 +
 .../{StatusBoltTest.java => QueueBoltTest.java}    | 116 +++++++++----
 .../opensearch/bolt/StatusBoltTest.java            |  22 +++
 .../src/test/resources/queues.mapping              |  33 ++++
 external/opensearch/pom.xml                        |   2 +-
 external/playwright/pom.xml                        |   2 +-
 10 files changed, 404 insertions(+), 102 deletions(-)