You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by dp...@apache.org on 2026/07/15 12:43:08 UTC
(stormcrawler) 08/08: docs: document URLFrontier robots crawl-delay pacing
This is an automated email from the ASF dual-hosted git repository.
dpol1 pushed a commit to branch feat/1979-robots-delay
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
commit 31598b63019f5ba5d7941dad1210e437c982fa2e
Author: Davide Polato <dp...@apache.org>
AuthorDate: Wed Jul 15 13:57:43 2026 +0200
docs: document URLFrontier robots crawl-delay pacing
Describe the opt-in wiring and the host partitioning, single-URL hand-out, persist-only metadata, and delay-cap requirements for #1979.
---
core/src/main/resources/crawler-default.yaml | 7 +++--
docs/src/main/asciidoc/configuration.adoc | 18 ++++++++++--
docs/src/main/asciidoc/extending.adoc | 2 +-
external/urlfrontier/README.md | 42 ++++++++++++++++++++++++++++
4 files changed, 63 insertions(+), 6 deletions(-)
diff --git a/core/src/main/resources/crawler-default.yaml b/core/src/main/resources/crawler-default.yaml
index 9961320b..9b176084 100644
--- a/core/src/main/resources/crawler-default.yaml
+++ b/core/src/main/resources/crawler-default.yaml
@@ -40,8 +40,11 @@ config:
# skip URLs from this queue to avoid that any overlong
# crawl-delay throttles the crawler
# (if true)
- # set the delay to fetcher.max.crawl.delay,
- # making fetcher more aggressive than requested
+ # cap the local delay at fetcher.max.crawl.delay and emit the
+ # requested value in whole seconds as robots.crawl.delay;
+ # URLFrontier's QueueRegulatorBolt can forward a capped value when
+ # its robots crawl-delay integration is explicitly enabled. Without
+ # that consumer the fetcher remains more aggressive than requested.
fetcher.max.crawl.delay.force: false
# behavior of fetcher when the crawl-delay in the robots.txt
# is smaller (ev. less than one second) than the default delay:
diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc
index fd9e52b6..f938b14f 100644
--- a/docs/src/main/asciidoc/configuration.adoc
+++ b/docs/src/main/asciidoc/configuration.adoc
@@ -177,7 +177,7 @@ features you are using in your Apache StormCrawler topology. Some external modul
| fetcher.max.crawl.delay | 30 | Maximum accepted Crawl-delay from robots.txt (seconds). If the crawl-delay exceeds this value the behavior depends on the value of fetcher.max.crawl.delay.force.
| fetcher.max.crawl.delay.force | false | Configures the behavior of fetcher if the robots.txt crawl-delay exceeds fetcher.max.crawl.delay. If false: the tuple is emitted to the StatusStream
-as an ERROR. If true: the queue delay is set to fetcher.max.crawl.delay.
+as an ERROR. If true: the fetcher caps its local delay and emits the requested delay in whole seconds as `robots.crawl.delay`; an explicitly enabled URLFrontier `QueueRegulatorBolt` can forward a capped value to the host queue.
| fetcher.max.queue.size | -1 | The maximum length of the queue used to store items to be fetched by the FetcherBolt. A setting of -1 sets the length to Integer.MAX_VALUE.
| fetcher.max.throttle.sleep | -1 | The maximum amount of time to wait between fetches; if exceeded, the item is sent to the back of the queue. Used in SimpleFetcherBolt. -1 disables it.
| fetcher.max.urls.in.queues | -1 | Limits the number of URLs that can be stored in a fetch queue. -1 disables the limit.
@@ -472,6 +472,17 @@ See the link:https://github.com/apache/stormcrawler/tree/main/external/sql[sql m
Integration with link:https://github.com/crawler-commons/url-frontier[URLFrontier], a language-agnostic API for URL management and scheduling.
See the link:https://github.com/apache/stormcrawler/tree/main/external/urlfrontier[urlfrontier module] for full details.
+`QueueRegulatorBolt` consumes the `queue` stream from the URLFrontier status updater and applies
+rate-limit blocks or robots.txt pacing. Robots pacing is enabled when both
+`urlfrontier.robots.crawl.delay.enabled` and `fetcher.max.crawl.delay.force` are `true`. It requires `partition.url.mode: byHost`,
+`urlfrontier.max.urls.per.bucket: 1`, and `robots.crawl.delay` in `metadata.persist`. The same key
+must not match `metadata.transfer`, including wildcards such as `robots.*`, because it is a
+per-host control signal. The bolt probes these requirements at startup. A batch size of one
+bounds each frontier hand-out but does not recall URLs already emitted or eliminate concurrent
+requests while the new delay is reaching the frontier. A custom `metadata.transfer.class` is
+responsible for preserving the same persist-only contract for all URLs and values; the startup
+probe exercises only representative metadata.
+
[cols="1,1,3", options="header"]
|===
| key | default value | description
@@ -480,7 +491,8 @@ See the link:https://github.com/apache/stormcrawler/tree/main/external/urlfronti
| urlfrontier.port | 7071 | URLFrontier service port.
| urlfrontier.address | - | URLFrontier service address (can be used instead of host and port).
| urlfrontier.max.buckets | 10 | Number of buckets to request from the frontier.
-| urlfrontier.max.urls.per.bucket | 10 | Maximum URLs per bucket.
+| urlfrontier.max.urls.per.bucket | 10 | Maximum URLs per bucket. Must be 1 when robots crawl-delay pacing is enabled so one request cannot hand out several URLs from the same host.
+| urlfrontier.robots.crawl.delay.enabled | false | Enables forwarding long robots.txt Crawl-delays from the queue stream to URLFrontier. Also requires `fetcher.max.crawl.delay.force: true` and the validated configuration described above.
| urlfrontier.delay.requestable | - | Time in seconds for which the URLs are locked in URLFrontier.
| urlfrontier.cache.expireafter.sec | - | Time in seconds after which the cache expires.
| urlfrontier.max.messages.in.flight | - | Maximum number of messages in flight for the StatusUpdater.
@@ -490,7 +502,7 @@ See the link:https://github.com/apache/stormcrawler/tree/main/external/urlfronti
| urlfrontier.max.retry.after | 86400 | Maximum delay in seconds honoured when a server requests a back-off via the Retry-After HTTP response header. -1 disables the cap.
| urlfrontier.backoff.base.secs | 60 | First block duration in seconds applied when a host rate-limits without a usable Retry-After header. 0 disables the adaptive back-off.
| urlfrontier.backoff.factor | 2.0 | Multiplicative increase applied to the block duration on each new pressure event for the same host.
-| urlfrontier.backoff.max.secs | 86400 | Upper bound in seconds for the escalated block duration.
+| urlfrontier.backoff.max.secs | 86400 | Upper bound in seconds for the escalated block duration and for a robots crawl-delay forwarded to the frontier.
| urlfrontier.backoff.decay.secs | 1800 | Quiet window in seconds after which its escalation is forgotten and the next pressure event starts again from the base duration.
| urlfrontier.backoff.on.exceptions | false | Whether fetch exceptions count as pressure events and escalate the host's block.
| urlfrontier.backoff.jitter | 0.1 | Random extra fraction added to every computed block duration.
diff --git a/docs/src/main/asciidoc/extending.adoc b/docs/src/main/asciidoc/extending.adoc
index 7277f88b..5d1d7479 100644
--- a/docs/src/main/asciidoc/extending.adoc
+++ b/docs/src/main/asciidoc/extending.adoc
@@ -273,7 +273,7 @@ When `robots.txt` specifies a `Crawl-delay`:
* If the delay is *smaller* than `fetcher.server.delay` and `fetcher.server.delay.force` is `true`, the larger default delay is used instead
* If the delay is *larger* than `fetcher.max.crawl.delay`:
** `fetcher.max.crawl.delay.force: false` (default) — the queue is skipped and the URL gets an ERROR status
-** `fetcher.max.crawl.delay.force: true` — the delay is capped at `fetcher.max.crawl.delay`
+** `fetcher.max.crawl.delay.force: true` — the fetcher caps its local delay at `fetcher.max.crawl.delay` and reports the requested delay in whole seconds as `robots.crawl.delay`; when `QueueRegulatorBolt` is wired and `urlfrontier.robots.crawl.delay.enabled: true`, the frontier forwards a value capped by `urlfrontier.backoff.max.secs` to the host queue
==== Robots.txt Compliance
diff --git a/external/urlfrontier/README.md b/external/urlfrontier/README.md
index 8647fc04..869ed043 100644
--- a/external/urlfrontier/README.md
+++ b/external/urlfrontier/README.md
@@ -24,6 +24,48 @@ urlfrontier.max.buckets: 10
urlfrontier.max.urls.per.bucket:10
```
+## Robots crawl-delay pacing
+
+`QueueRegulatorBolt` can pace host queues when a robots.txt Crawl-delay exceeds the fetcher's local
+limit. Wire it to the `queue` stream emitted by `StatusUpdaterBolt`:
+
+```yaml
+bolts:
+ - id: "queue-regulator"
+ className: "org.apache.stormcrawler.urlfrontier.QueueRegulatorBolt"
+ parallelism: 1
+
+streams:
+ - from: "status"
+ to: "queue-regulator"
+ grouping:
+ type: FIELDS
+ args: ["key"]
+ streamId: "queue"
+```
+
+Robots pacing is opt-in. It requires host partitioning, one URL per frontier hand-out, a positive
+delay cap, and a persist-only control signal:
+
+```yaml
+partition.url.mode: byHost
+fetcher.max.crawl.delay.force: true
+urlfrontier.robots.crawl.delay.enabled: true
+urlfrontier.max.urls.per.bucket: 1
+urlfrontier.backoff.max.secs: 86400
+
+metadata.persist:
+ - robots.crawl.delay
+```
+
+Do not include `robots.crawl.delay` in `metadata.transfer`, directly or through a wildcard such as
+`robots.*`: an outlink must not inherit its parent's host delay. The bolt rejects an unsafe robots
+configuration at startup. A custom `metadata.transfer.class` must preserve this contract for every
+URL and value; the startup probe can only exercise representative metadata. `setDelay(key, 0)` is
+not sent if a site later removes its Crawl-delay, so that host can remain slower than necessary but
+is not made less polite. A batch size of one bounds each hand-out; it does not recall URLs already
+emitted or eliminate concurrent requests while the new delay is reaching the frontier.
+
Your StormCrawler topology requires the following dependency in its pom.xml (just like with any other module)
```