You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by rz...@apache.org on 2026/03/27 09:13:39 UTC

(stormcrawler) branch main updated: Improve documentation (missing configs, etc.) (#1843)

This is an automated email from the ASF dual-hosted git repository.

rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git


The following commit(s) were added to refs/heads/main by this push:
     new fbefb88c Improve documentation (missing configs, etc.) (#1843)
fbefb88c is described below

commit fbefb88cb951336bd41957d7470616fc81153686
Author: Richard Zowalla <13...@users.noreply.github.com>
AuthorDate: Fri Mar 27 10:13:35 2026 +0100

    Improve documentation (missing configs, etc.) (#1843)
    
    * fix(docs): correct configuration defaults and class references in adoc files
    
    Fix inaccurate default values, wrong config key names, and a
    nonexistent class reference in the documentation:
    - protocols: add missing "file" to default
    - redirections.allowed → http.allow.redirects (default false, not true)
    - fetchInterval.error: -1, not 44640
    - cacheConfigParamName → robots.cache.spec
    - errorcacheConfigParamName → robots.error.cache.spec
    - http.accept and http.accept.language: add actual defaults
    - protocol.md.prefix: default is "protocol.", not empty
    - FrontierSpout → Spout (class doesn't exist)
    
    * fix(docs): correct errors, broken links, and outdated references across documentation
    
    - Fix wrong package names: org.apache.storm.crawler → org.apache.stormcrawler
    - Fix outdated import: backtype.storm.Config → org.apache.storm.Config
    - Fix non-existent class reference: StatusStreamBolt → DummyIndexer
    - Fix method signature to match actual code (MetadataTransfer.getMetaForOutlink)
    - Fix broken internal links (StatusStream, Configuration, HTTPProtocol anchors)
    - Fix Selenium protocol links pointing to core/ instead of external/selenium/
    - Fix incorrect GitHub line number references for crawler-default.yaml
    - Fix incomplete references ("See blog post", "See example", "See default")
    - Fix Storm UI URL protocol: https → http
    - Fix TikaParser → ParserBolt (actual class name)
    - Remove undocumented/unimplemented http.store.responsetime config entry
    - Replace unexplained %HEAP-MEM% placeholder with concrete 2g value
    - Fix typos: anonynmous, asomething, extra commas, grammar
    - Fix AsciiDoc link syntax in powered-by.adoc
    
    * docs: add 27 missing config keys and 9 undocumented core classes
    
    Configuration tables in configuration.adoc now cover all keys from
    crawler-default.yaml, including robots, protocol, OkHttp, parsing,
    sitemap, scheduling, and indexing options.
    
    Internals documentation now covers:
    - FeedParserBolt, URLPartitionerBolt, StatusEmitterBolt
    - FileSpout, MemorySpout, AbstractQueryingSpout
    - AdaptiveScheduler (with config example)
    - DelegatorProtocol (with config example)
    - SelfURLFilter
    
    Also fixes missing default values for indexer.text.fieldname and
    indexer.url.fieldname.
    
    * docs: add external module configuration reference for all 11 modules
    
    Add configuration tables for OpenSearch, Solr, SQL, URLFrontier, Tika,
    AWS, AI/LLM, Playwright, Language ID, WARC, and common spout options.
    
    Each module section includes all key config options with defaults and
    descriptions, plus links to the module source for full setup details.
    The Language ID module (which had no README) now has its parse filter
    configuration documented for the first time.
    
    * docs: add extending and operating guide with 12 how-to topics
    
    New extending.adoc covers:
    - Writing custom URL filters, parse filters, protocols, bolts/spouts
      with complete code examples and registration instructions
    - Politeness and rate limiting: queue modes, crawl-delay handling,
      robots.txt compliance
    - Error handling and retry logic: status lifecycle, retry mechanism,
      custom fetch intervals
    - Monitoring and metrics: fetcher metrics, MetricsConsumer setup
    - Scaling and tuning: thread config, queue sizing, connection pooling
    - Security: SSL/TLS, basic auth, proxy auth
    
    * Address review comments
---
 core/src/main/resources/crawler-default.yaml |   2 +-
 docs/src/main/asciidoc/architecture.adoc     |   6 +-
 docs/src/main/asciidoc/configuration.adoc    | 293 ++++++++++++++++++---
 docs/src/main/asciidoc/extending.adoc        | 370 +++++++++++++++++++++++++++
 docs/src/main/asciidoc/index.adoc            |   2 +
 docs/src/main/asciidoc/internals.adoc        |  97 +++++--
 docs/src/main/asciidoc/powered-by.adoc       |   2 +-
 docs/src/main/asciidoc/quick-start.adoc      |  10 +-
 8 files changed, 726 insertions(+), 56 deletions(-)

diff --git a/core/src/main/resources/crawler-default.yaml b/core/src/main/resources/crawler-default.yaml
index 2c194a2a..47540871 100644
--- a/core/src/main/resources/crawler-default.yaml
+++ b/core/src/main/resources/crawler-default.yaml
@@ -82,7 +82,7 @@ config:
   metadata.track.path: true
   metadata.track.depth: true
 
-  # Agent name info - given here as an example. Do not be an anonynmous coward, use your real information!
+  # Agent name info - given here as an example. Do not be an anonymous coward, use your real information!
   # The full user agent value sent as part of the HTTP requests
   # is built from the elements below. Only the agent.name is mandatory,
   # it is also used to parse the robots.txt directives.
diff --git a/docs/src/main/asciidoc/architecture.adoc b/docs/src/main/asciidoc/architecture.adoc
index bbacfae6..5b2a22e5 100644
--- a/docs/src/main/asciidoc/architecture.adoc
+++ b/docs/src/main/asciidoc/architecture.adoc
@@ -29,13 +29,13 @@ Black arrows show the *main data flow*, gray arrows represent URLs *taken from t
 
 The crawl begins with the *Frontier*, which is responsible for scheduling,
 prioritization, politeness, and retry logic. URLs are emitted by a
-`FrontierSpout` and partitioned by the `URLPartitioner`, typically using the
+`Spout` (e.g., `AggregationSpout`, `SQLSpout`) and partitioned by the `URLPartitioner`, typically using the
 host as a key to enforce politeness constraints.
 
 The `Fetcher` retrieves web resources and emits both the fetched content and
 associated metadata such as HTTP status codes, headers, and MIME types. Based
 on the content type, documents are routed to specialized parsers, including
-`SiteMapParser`, `JSoupParser` for HTML content, and `TikaParser` for binary
+`SiteMapParser`, `JSoupParser` for HTML content, and `ParserBolt` (Tika module) for binary
 formats via Apache Tika.
 
 Parsed content is then sent to the `Indexer` and persisted by the `Storage`
@@ -74,7 +74,7 @@ They allow fine-grained control over how extracted data and outgoing links are
 processed.
 
 Parse Filters are applied within the parsing bolts, following parsing by
-`SiteMapParser`, `JSoupParser`, or `TikaParser`. They can modify extracted text,
+`SiteMapParser`, `JSoupParser`, or `ParserBolt` (Tika). They can modify extracted text,
 metadata, and links before the content is indexed or new URLs are emitted.
 
 Common Parse Filters include:
diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc
index b5cfbaca..7526546c 100644
--- a/docs/src/main/asciidoc/configuration.adoc
+++ b/docs/src/main/asciidoc/configuration.adoc
@@ -96,9 +96,9 @@ The **SCProxy** class contains all of the information associated with proxy conn
 
 ==== Registering Metadata for Kryo Serialization
 
-If your Apache StormCrawler topology doesn't extend `org.apache.storm.crawler.ConfigurableTopology`, you will need to manually register StormCrawler's `Metadata` class for serialization in Storm. For more information on Kryo serialization in Apache Storm, you can refer to the link:https://storm.apache.org/documentation/Serialization.html[documentation].
+If your Apache StormCrawler topology doesn't extend `org.apache.stormcrawler.ConfigurableTopology`, you will need to manually register StormCrawler's `Metadata` class for serialization in Storm. For more information on Kryo serialization in Apache Storm, you can refer to the link:https://storm.apache.org/documentation/Serialization.html[documentation].
 
-To register `Metadata` for serialization, you'll need to import `backtype.storm.Config` and `org.apache.storm.crawler.Metadata`. Then, in your topology class, you'll register the class with:
+To register `Metadata` for serialization, you'll need to import `org.apache.storm.Config` and `org.apache.stormcrawler.Metadata`. Then, in your topology class, you'll register the class with:
 
 [source,java]
 ----
@@ -112,7 +112,7 @@ Alternatively, you can specify in the configuration file:
 [source,yaml]
 ----
 topology.kryo.register:
-  - org.apache.storm.crawler.Metadata
+  - org.apache.stormcrawler.Metadata
 ----
 
 ==== MetadataTransfer
@@ -164,8 +164,7 @@ features you are using in your Apache StormCrawler topology. Some external modul
 |===
 | key | default value | description
 
-| fetcher.max.crawl.delay | 30 | The maximum number in seconds that will be accepted by Crawl-delay
-directives in robots.txt files. If the crawl-delay exceeds this value the behavior depends on the value of fetcher.max.crawl.delay.force.
+| fetcher.max.crawl.delay | 30 | Maximum accepted Crawl-delay from robots.txt (seconds). If the crawl-delay exceeds this value the behavior depends on the value of fetcher.max.crawl.delay.force.
 | fetcher.max.crawl.delay.force | false | Configures the behavior of fetcher if the robots.txt crawl-delay exceeds fetcher.max.crawl.delay. If false: the tuple is emitted to the StatusStream
 as an ERROR. If true: the queue delay is set to fetcher.max.crawl.delay.
 | fetcher.max.queue.size | -1 | The maximum length of the queue used to store items to be fetched by the FetcherBolt. A setting of -1 sets the length to Integer.MAX_VALUE.
@@ -180,6 +179,7 @@ is defined.
 | fetcher.server.min.delay | 0 | Delay between crawls for queues with >1 thread. Ignores robots.txt.
 | fetcher.threads.number | 10 | Total concurrent threads fetching pages. Adjust carefully based on system capacity.
 | fetcher.threads.per.queue | 1 | Default number of threads per queue. Can be overridden.
+| fetcher.threads.start.delay | 10 | Delay (milliseconds) between starting next fetcher thread. Avoids overloading DNS or network resources during fetcher startup when all threads simultaneously start requesting pages.
 | fetcher.timeout.queue | -1 | Maximum wait time (seconds) for items in the queue. -1 disables timeout.
 | fetcherbolt.queue.debug.filepath | "" | Path to a debug log (e.g. /tmp/fetcher-dump-{port}).
 | http.agent.description | - | Description for the User-Agent header.
@@ -196,20 +196,27 @@ implementation.
 | http.proxy.pass | - | Proxy password.
 | http.proxy.port | 8080 | Proxy port.
 | http.proxy.user | - | Proxy username.
-| http.robots.403.allow | true | Defines behavior when robots.txt returns HTTP 403.
+| http.retry.on.connection.failure | true | (OkHttp only) Retry fetching on connection failure. See link:https://square.github.io/okhttp/5.x/okhttp/okhttp3/-ok-http-client/-builder/retry-on-connection-failure.html[OkHttp docs].
+| http.robots.403.allow | true | Allow crawling when robots.txt returns HTTP 403.
+| http.robots.5xx.allow | false | Allow crawling when robots.txt returns a server error (5xx).
 | http.robots.agents | '' | Additional user-agent strings for interpreting robots.txt.
-| http.robots.file.skip | false | Ignore robots.txt rules (1.17+).
+| http.robots.content.limit | -1 | Maximum bytes to fetch for robots.txt. -1 uses http.content.limit.
+| http.robots.file.skip | false | Ignore robots.txt rules entirely.
+| http.robots.headers.skip | false | Ignore robots directives from HTTP headers.
+| http.robots.meta.skip | false | Ignore robots directives from HTML meta tags.
 | http.skip.robots | false | Deprecated (replaced by http.robots.file.skip).
+| robots.noFollow.strict | true | If true, remove all outlinks from pages marked as noFollow.
 | http.store.headers | false | Whether to store response headers.
-| http.store.responsetime | true | Not yet implemented — store response time in Metadata.
 | http.timeout | 10000 | Connection timeout (ms).
 | http.use.cookies | false | Use cookies in subsequent requests.
 | https.protocol.implementation | org.apache.stormcrawler.protocol.httpclient.HttpProtocol | HTTPS Protocol
 implementation.
 | partition.url.mode | byHost | Defines how URLs are partitioned: byHost, byDomain, or byIP.
-| protocols | http,https | Supported protocols.
-| redirections.allowed | true | Allow URL redirects.
+| protocols | http,https,file | Supported protocols.
+| redirections.allowed | true | If true, emit redirect target URLs as "outlinks" to the status stream. If false, do not follow redirects. See also `http.allow.redirects`.
+| http.allow.redirects | false | (OkHttp only) Follow HTTP redirects immediately in the HTTP protocol client. Note: if followed immediately, redirect target URLs are not emitted to the status stream, are not filtered, not deduplicated, and not checked against robots.txt.
 | sitemap.discovery | false | Enable automatic sitemap discovery.
+| urlbuffer.class | org.apache.stormcrawler.persistence.urlbuffer.SimpleURLBuffer | URL buffer implementation used by spouts.
 |===
 
 ==== Protocol
@@ -218,18 +225,22 @@ implementation.
 |===
 | key | default value | description
 
-| cacheConfigParamName | maximumSize=10000,expireAfterWrite=6h | CacheBuilder configuration for robots cache.
-| errorcacheConfigParamName | maximumSize=10000,expireAfterWrite=1h | CacheBuilder configuration for error cache.
+| file.protocol.implementation | org.apache.stormcrawler.protocol.file.FileProtocol | Protocol implementation for file:// URLs.
 | file.encoding | UTF-8 | Encoding for FileProtocol.
+| protocol.instances.num | 1 | Number of instances per protocol implementation.
+| http.protocol.versions | - | HTTP protocol versions in order of preference (h2, http/1.1, http/1.0, h2c). If empty, uses implementation defaults.
+| robots.cache.spec | maximumSize=10000,expireAfterWrite=6h | CacheBuilder configuration for robots cache.
+| robots.error.cache.spec | maximumSize=10000,expireAfterWrite=1h | CacheBuilder configuration for error cache.
+| okhttp.protocol.connection.pool.max.idle.connections | 5 | OkHttp maximum number of idle connections.
+| okhttp.protocol.connection.pool.connection.keep.alive | 300 | OkHttp connection keep-alive time (seconds).
 | http.custom.headers | - | Custom HTTP headers.
-| http.accept | - | HTTP Accept
+| http.accept | text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 | HTTP Accept
 header.
-| http.accept.language | - | HTTP Accept-Language
+| http.accept.language | en-us,en-gb,en;q=0.7,*;q=0.3 | HTTP Accept-Language
 header.
 | http.content.partial.as.trimmed | false | Accepts partially fetched content in OKHTTP.
 | http.trust.everything | true | If true, trust all SSL/TLS connections.
-| navigationfilters.config.file | - | JSON config for NavigationFilter. See blog post
-.
+| navigationfilters.config.file | - | JSON config for NavigationFilter (used by the Selenium protocol module).
 | selenium.addresses | - | WebDriver server addresses.
 | selenium.capabilities | - | Desired WebDriver capabilities
 .
@@ -249,27 +260,31 @@ The values below are used by sub-classes of `AbstractIndexerBolt`.
 |===
 | key | default value | description
 
+| indexer.canonical.name | canonical | Metadata key for the canonical URL. Used to replace the URL with its canonical form before indexing.
+| indexer.ignore.empty.fields | false | If true, skip fields with empty values when indexing.
 | indexer.md.filter | - | YAML list of key=value filters for metadata-based indexing.
 | indexer.md.mapping | - | YAML mapping from metadata fields to persistence layer fields.
-| indexer.text.fieldname | - | Field name for indexed HTML body text.
-| indexer.url.fieldname | - | Field name for indexed URL.
+| indexer.text.fieldname | content | Field name for indexed HTML body text.
+| indexer.text.maxlength | -1 | Maximum length of text to index. -1 means no limit.
+| indexer.url.fieldname | url | Field name for indexed URL.
 |===
 
 ==== Status Persistence
 
-This refers to persisting the status of a URL (e.g. ERROR, DISCOVERED etc.) along with asomething like a `nextFetchDate`
+This refers to persisting the status of a URL (e.g. ERROR, DISCOVERED etc.) along with something like a `nextFetchDate`
 that is being calculated by a link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/DefaultScheduler.java[Scheduler].
 
 [cols="1,1,3", options="header"]
 |===
 | key | default value | description
 
-| fetchInterval.default | 1440 | Default revisit interval (minutes). Used by DefaultScheduler
-.
-| fetchInterval.error | 44640 | Revisit interval for error pages (minutes).
+| fetchInterval.default | 1440 | Default revisit interval (minutes). Used by DefaultScheduler.
+| fetchInterval.error | -1 | Revisit interval for error pages (minutes). -1 means never refetch.
 | fetchInterval.fetch.error | 120 | Revisit interval for fetch errors (minutes).
-| status.updater.cache.spec | maximumSize=10000, expireAfterAccess=1h | Cache specification
-.
+| max.fetch.errors | 3 | Maximum number of successive fetch errors before changing status to ERROR.
+| scheduler.class | org.apache.stormcrawler.persistence.DefaultScheduler | Scheduler implementation for computing next fetch dates. Use AdaptiveScheduler for change-rate-based intervals.
+| status.updater.cache.spec | maximumSize=10000, expireAfterAccess=1h | Cache specification for the status updater.
+| status.updater.unit.round.date | SECOND | Unit for rounding the next fetch date. Can also be MINUTE or HOUR.
 | status.updater.use.cache | true | Whether to use cache to avoid re-persisting URLs.
 |===
 
@@ -281,21 +296,25 @@ Configures parsing of fetched text and the handling of discovered URIs
 |===
 | key | default value | description
 
-| collections.file | collections.json | Config for CollectionTagger
-.
+| collections.file | collections.json | Config for CollectionTagger.
 | collections.key | collections | Key under which tags are stored in metadata.
+| detect.charset.maxlength | 10000 | Maximum number of bytes used for charset detection.
+| detect.mimetype | true | Enable MIME type detection during parsing.
 | feed.filter.hours.since.published | -1 | Discard feeds older than value hours.
 | feed.sniffContent | false | Try to detect feeds automatically.
-| parsefilters.config.file | parsefilters.json | Path to JSON config defining ParseFilters. See example
-.
+| jsoup.treat.non.html.as.error | true | If true, non-HTML content is treated as an error by JSoupParserBolt.
+| parsefilters.config.file | parsefilters.json | Path to JSON config defining ParseFilters. See link:https://github.com/apache/stormcrawler/blob/main/core/src/main/resources/parsefilters.json[default parsefilters.json].
 | parser.emitOutlinks | true | Emit discovered links as DISCOVERED tuples.
 | parser.emitOutlinks.max.per.page | -1 | Limit number of emitted links per page.
+| sitemap.filter.hours.since.modified | -1 | Filter URLs in sitemaps based on their modification date. -1 disables filtering.
+| sitemap.schedule.delay | -1 | Staggered scheduling delay for sitemaps (minutes). -1 disables staggering.
+| sitemap.extensions | - | Sitemap extensions to parse (IMAGE, LINKS, MOBILE, NEWS, VIDEO).
 | textextractor.exclude.tags | "" | HTML tags ignored by TextExtractor.
 | textextractor.include.pattern | "" | Regex patterns to include for TextExtractor.
 | textextractor.no.text | false | Disable text extraction entirely.
+| textextractor.skip.after | -1 | Stop text extraction after this many characters. -1 means no limit.
 | track.anchors | true | Add anchor text to outlink metadata.
-| urlfilters.config.file | urlfilters.json | JSON file defining URL filters. See default
-.
+| urlfilters.config.file | urlfilters.json | JSON file defining URL filters. See link:https://github.com/apache/stormcrawler/blob/main/archetype/src/main/resources/archetype-resources/src/main/resources/urlfilters.json[default urlfilters.json].
 |===
 
 ====  Metadata
@@ -312,5 +331,217 @@ minimising metadata clashes
 | metadata.track.path | true | Track URL path history in metadata.
 | metadata.transfer | - | Metadata to transfer to outlinks.
 | metadata.transfer.class | org.apache.stormcrawler.util.MetadataTransfer | Class handling metadata transfer.
-| protocol.md.prefix | - | Prefix for remote metadata keys to avoid collisions.
+| protocol.md.prefix | protocol. | Prefix for remote metadata keys to avoid collisions.
+|===
+
+=== External Module Configuration
+
+The following sections document configuration options for the external integration modules.
+Each module has its own Maven dependency and may require additional infrastructure (e.g., OpenSearch, Solr, a SQL database).
+For full setup instructions, see the README in each module's directory.
+
+==== OpenSearch
+
+Integration with link:https://opensearch.org/[OpenSearch] for indexing, URL status persistence, and metrics.
+See the link:https://github.com/apache/stormcrawler/tree/main/external/opensearch[opensearch module] for full details.
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| opensearch.addresses | - | OpenSearch server address(es).
+| opensearch.user | - | Username for authentication (optional).
+| opensearch.password | - | Password for authentication (optional).
+| opensearch.concurrentRequests | 2 | Number of concurrent bulk requests.
+| opensearch.indexer.index.name | content | Index name for crawled documents.
+| opensearch.indexer.create | false | Auto-create index if it does not exist.
+| opensearch.indexer.bulkActions | 100 | Number of documents to buffer before flushing.
+| opensearch.indexer.flushInterval | 2s | Maximum time before flushing the bulk buffer.
+| opensearch.indexer.pipeline | - | Ingest pipeline name (optional).
+| opensearch.status.index.name | status | Index name for URL status.
+| opensearch.status.bulkActions | 500 | Batch size for status updates.
+| opensearch.status.flushInterval | 5s | Flush interval for status updates.
+| opensearch.status.routing | true | Enable index routing for status documents.
+| opensearch.status.routing.fieldname | key | Field used for routing.
+| opensearch.status.max.buckets | 50 | Number of buckets for AggregationSpout.
+| opensearch.status.max.urls.per.bucket | 2 | URLs returned per bucket.
+| opensearch.status.bucket.field | key | Field used for bucketing.
+| opensearch.status.sample | false | Use random sampling in AggregationSpout.
+| opensearch.metrics.index.name | metrics | Index name for Storm metrics.
+|===
+
+==== Solr
+
+Integration with link:https://solr.apache.org/[Apache Solr] for indexing, URL status persistence, and metrics.
+See the link:https://github.com/apache/stormcrawler/tree/main/external/solr[solr module] for full details. Supports both standalone Solr and SolrCloud (via ZooKeeper).
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| solr.indexer.url | - | Solr collection URL for indexing.
+| solr.indexer.zkhost | - | ZooKeeper host for SolrCloud indexing (alternative to URL).
+| solr.indexer.collection | - | SolrCloud collection name for indexing.
+| solr.status.url | - | Solr collection URL for status storage.
+| solr.status.zkhost | - | ZooKeeper host for SolrCloud status (alternative to URL).
+| solr.status.collection | - | SolrCloud collection name for status.
+| solr.status.bucket.field | host | Field used for bucketing in SolrSpout.
+| solr.status.bucket.maxsize | 5 | Maximum URLs per bucket.
+| solr.status.max.results | 10 | Maximum results per spout query.
+| solr.status.metadata.prefix | metadata | Prefix for metadata fields in Solr.
+| solr.metrics.url | - | Solr collection URL for metrics.
+|===
+
+==== SQL
+
+Integration with relational databases via JDBC for URL status persistence, indexing, and metrics.
+See the link:https://github.com/apache/stormcrawler/tree/main/external/sql[sql module] for full details and table creation scripts.
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| sql.connection.url | - | JDBC connection URL.
+| sql.connection.user | - | Database username.
+| sql.connection.password | - | Database password.
+| sql.status.table | urls | Table name for URL status storage.
+| sql.max.urls.per.bucket | 5 | Maximum URLs per bucket in SQLSpout.
+| sql.spout.max.results | 100 | Maximum results per spout query.
+| sql.metrics.table | metrics | Table name for metrics storage.
+| sql.index.table | content | Table name for indexed content.
+|===
+
+==== URLFrontier
+
+Integration with link:https://github.com/crawler-commons/url-frontier[URLFrontier], a language-agnostic API for URL management and scheduling.
+See the link:https://github.com/apache/stormcrawler/tree/main/external/urlfrontier[urlfrontier module] for full details.
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| urlfrontier.host | localhost | URLFrontier service hostname.
+| urlfrontier.port | 7071 | URLFrontier service port.
+| urlfrontier.max.buckets | 10 | Number of buckets to request from the frontier.
+| urlfrontier.max.urls.per.bucket | 10 | Maximum URLs per bucket.
+|===
+
+==== Tika
+
+Apache Tika integration for parsing non-HTML documents (PDF, Word, etc.).
+See the link:https://github.com/apache/stormcrawler/tree/main/external/tika[tika module] for full details.
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| parser.mimetype.whitelist | - | Regex list of allowed MIME types. If set, only matching types are parsed by the Tika ParserBolt.
+| parser.tika.config.file | tika-config.xml | Path to the Tika configuration XML file.
+|===
+
+NOTE: When using the Tika `ParserBolt` alongside `JSoupParserBolt`, set `jsoup.treat.non.html.as.error` to `false` so that non-HTML content is passed through to the Tika parser rather than being treated as an error.
+
+==== AWS
+
+Integration with AWS services: CloudSearch for indexing and S3 for content caching.
+See the link:https://github.com/apache/stormcrawler/tree/main/external/aws[aws module] for full details.
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| cloudsearch.endpoint | - | AWS CloudSearch document endpoint URL.
+| cloudsearch.region | - | AWS region (e.g., "eu-west-1").
+| cloudsearch.batch.maxSize | -1 | Documents to buffer before sending a batch. -1 disables batching.
+| cloudsearch.batch.max.time.buffered | 10 | Maximum time (seconds) before flushing the buffer.
+| cloudsearch.batch.dump | false | Dump batch JSON to a temp directory for debugging.
+| s3.region | - | AWS region for S3 operations.
+| s3.bucket | - | S3 bucket name for content caching.
+| s3.endpoint | - | Custom S3 endpoint (optional, for S3-compatible services).
+|===
+
+==== AI (LLM Text Extraction)
+
+LLM-based text extraction using OpenAI-compatible APIs (including local models via Ollama).
+See the link:https://github.com/apache/stormcrawler/tree/main/external/ai[ai module] for full details.
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| textextractor.class | - | Set to `org.apache.stormcrawler.ai.OpenAITextExtractor` to enable LLM extraction.
+| textextractor.llm.api_key | - | API key for the LLM service.
+| textextractor.llm.url | - | LLM API endpoint URL (e.g., OpenAI or Ollama endpoint).
+| textextractor.llm.model | - | Model name to use (e.g., "gpt-4", "llama2").
+| textextractor.system.prompt | - | System prompt for the LLM (optional).
+| textextractor.llm.prompt | - | User prompt template. Use `\{HTML}` and `\{REQUEST}` as placeholders (optional).
+| textextractor.llm.user_request | - | Extra user request passed to the prompt template (optional).
+| textextractor.llm.listener.clazz | - | Listener class for tracking LLM response metrics (optional).
+|===
+
+==== Playwright
+
+Browser-based fetching using link:https://playwright.dev/[Playwright] for JavaScript-rendered pages.
+See the link:https://github.com/apache/stormcrawler/tree/main/external/playwright[playwright module] for full details.
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| playwright.cdp.url | - | Chrome DevTools Protocol URL for connecting to an existing browser instance.
+| playwright.remote.ws | - | Remote WebSocket URL for Playwright (alternative to CDP).
+| playwright.skip.download | true | Skip automatic browser download. Set to false to let Playwright manage its own browser.
+| playwright.load.event | - | Page load event to wait for (e.g., "domcontentloaded", "networkidle").
+|===
+
+==== Language ID
+
+Language identification for crawled documents using the lang-detect library.
+See the link:https://github.com/apache/stormcrawler/tree/main/external/langid[langid module] source code for details.
+
+The `LanguageID` parse filter is configured in `parsefilters.json`:
+
+[source,json]
+----
+{
+  "class": "org.apache.stormcrawler.parse.filter.LanguageID",
+  "name": "LanguageID",
+  "params": {
+    "key": "lang",
+    "minProb": 0.99,
+    "extracted": "parse.lang"
+  }
+}
+----
+
+* `key` — metadata key to store the detected language code (default: "lang")
+* `minProb` — minimum probability threshold for detection (default: 0.999)
+* `extracted` — metadata key to check for a pre-extracted language value (default: "parse.lang")
+
+==== WARC
+
+Reading and writing link:https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/[WARC] (Web ARChive) files for archival and replay.
+See the link:https://github.com/apache/stormcrawler/tree/main/external/warc[warc module] for full details.
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| warc.metadata.keys | - | Metadata keys to include as WARC metadata records (optional list).
+|===
+
+NOTE: For complete WARC records, set `http.store.headers` to `true`. The OkHttp protocol (`org.apache.stormcrawler.protocol.okhttp.HttpProtocol`) is recommended for WARC generation as it provides verbatim HTTP headers.
+
+==== Common Spout Configuration
+
+The following options are shared across storage-backed spouts (OpenSearch, Solr, SQL) that extend `AbstractQueryingSpout`:
+
+[cols="1,1,3", options="header"]
+|===
+| key | default value | description
+
+| spout.ttl.purgatory | 30 | Time (seconds) a URL remains in purgatory after ack/fail before it can be re-emitted.
+| spout.min.delay.queries | 2000 | Minimum delay (ms) between backend queries.
+| spout.max.delay.queries | 20000 | Maximum delay (ms) between backend queries (OpenSearch only).
+| spout.reset.fetchdate.after | 120 | Reset a URL's fetch date if it has not been acked within this many seconds.
 |===
diff --git a/docs/src/main/asciidoc/extending.adoc b/docs/src/main/asciidoc/extending.adoc
new file mode 100644
index 00000000..f6db54ab
--- /dev/null
+++ b/docs/src/main/asciidoc/extending.adoc
@@ -0,0 +1,370 @@
+////
+Licensed under the Apache License, Version 2.0 (the "License");
+You may not use this file except in compliance with the License.
+You may obtain a copy of the License at:
+https://www.apache.org/licenses/LICENSE-2.0
+////
+== Extending and Operating StormCrawler
+
+This section provides practical guides for extending StormCrawler with custom components, tuning crawl performance, and operating crawlers in production.
+
+=== Writing Custom Components
+
+==== Custom URL Filter
+
+URL filters accept, reject, or modify URLs during parsing and redirection handling. To create a custom filter, extend the abstract `URLFilter` class and implement the `filter` method.
+
+[source,java]
+----
+import org.apache.stormcrawler.filtering.URLFilter;
+import org.apache.stormcrawler.Metadata;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.net.URL;
+import java.util.Map;
+
+public class MyURLFilter extends URLFilter {
+
+    private int maxLength;
+
+    @Override
+    public void configure(Map<String, Object> stormConf, JsonNode paramNode) {
+        maxLength = paramNode.path("maxLength").asInt(1024);
+    }
+
+    @Override
+    public String filter(URL sourceUrl, Metadata sourceMetadata, String urlToFilter) {
+        if (urlToFilter.length() > maxLength) {
+            return null; // reject
+        }
+        return urlToFilter; // accept (or return a normalized version)
+    }
+}
+----
+
+Register the filter in `urlfilters.json`:
+
+[source,json]
+----
+{
+  "org.apache.stormcrawler.filtering.URLFilters": [
+    {
+      "class": "com.example.MyURLFilter",
+      "name": "MyURLFilter",
+      "params": {
+        "maxLength": 2048
+      }
+    }
+  ]
+}
+----
+
+Filters are executed in the order they appear in the JSON file. Return `null` to reject a URL, or return the (possibly modified) URL string to accept it.
+
+==== Custom Parse Filter
+
+Parse filters extract data from fetched documents and can modify metadata and outlinks. Extend `ParseFilter` and implement the `filter` method.
+
+[source,java]
+----
+import org.apache.stormcrawler.parse.ParseFilter;
+import org.apache.stormcrawler.parse.ParseResult;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.w3c.dom.DocumentFragment;
+import java.util.Map;
+
+public class MyParseFilter extends ParseFilter {
+
+    private String metadataKey;
+
+    @Override
+    public void configure(Map<String, Object> stormConf, JsonNode paramNode) {
+        metadataKey = paramNode.path("key").asText("custom-field");
+    }
+
+    @Override
+    public boolean needsDOM() {
+        return false; // set to true if you need the DOM tree
+    }
+
+    @Override
+    public void filter(String url, byte[] content, DocumentFragment doc,
+                       ParseResult parse) {
+        // Extract data and store in metadata
+        parse.get(url).getMetadata().setValue(metadataKey, "extracted-value");
+    }
+}
+----
+
+Register it in `parsefilters.json`:
+
+[source,json]
+----
+{
+  "org.apache.stormcrawler.parse.ParseFilters": [
+    {
+      "class": "com.example.MyParseFilter",
+      "name": "MyParseFilter",
+      "params": {
+        "key": "custom-field"
+      }
+    }
+  ]
+}
+----
+
+If `needsDOM()` returns `false` for all configured parse filters, the parsing bolt skips DOM generation entirely, improving performance.
+
+==== Custom Protocol
+
+Protocol implementations define how content is fetched. Implement the `Protocol` interface:
+
+[source,java]
+----
+import org.apache.stormcrawler.protocol.Protocol;
+import org.apache.stormcrawler.protocol.ProtocolResponse;
+import org.apache.stormcrawler.Metadata;
+import crawlercommons.robots.BaseRobotRules;
+import org.apache.storm.Config;
+
+public class MyProtocol implements Protocol {
+
+    @Override
+    public void configure(Config conf) {
+        // Initialize from Storm configuration
+    }
+
+    @Override
+    public ProtocolResponse getProtocolOutput(String url, Metadata metadata)
+            throws Exception {
+        byte[] content = /* fetch content */;
+        int statusCode = 200;
+        Metadata responseMeta = new Metadata();
+        return new ProtocolResponse(content, statusCode, responseMeta);
+    }
+
+    @Override
+    public BaseRobotRules getRobotRules(String url) {
+        return RobotRulesParser.EMPTY_RULES;
+    }
+
+    @Override
+    public void cleanup() {}
+}
+----
+
+Register it in your configuration YAML:
+
+[source,yaml]
+----
+http.protocol.implementation: "com.example.MyProtocol"
+https.protocol.implementation: "com.example.MyProtocol"
+----
+
+Use the link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/DelegatorProtocol.java[DelegatorProtocol] when you need to route URLs to different protocol implementations based on metadata or URL patterns.
+
+==== Custom Bolt or Spout
+
+For a bolt that emits on the status stream (like fetchers and parsers), extend link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/StatusEmitterBolt.java[StatusEmitterBolt]:
+
+[source,java]
+----
+import org.apache.stormcrawler.bolt.StatusEmitterBolt;
+import org.apache.storm.tuple.Tuple;
+
+public class MyBolt extends StatusEmitterBolt {
+
+    @Override
+    public void execute(Tuple tuple) {
+        String url = tuple.getStringByField("url");
+        byte[] content = tuple.getBinaryByField("content");
+        Metadata metadata = (Metadata) tuple.getValueByField("metadata");
+
+        // Process content...
+
+        collector.emit(tuple, new Values(url, content, metadata));
+        collector.ack(tuple);
+    }
+
+    @Override
+    public void declareOutputFields(OutputFieldsDeclarer declarer) {
+        declarer.declare(new Fields("url", "content", "metadata"));
+        declarer.declareStream(
+            org.apache.stormcrawler.Constants.StatusStreamName,
+            new Fields("url", "metadata", "status"));
+    }
+}
+----
+
+For a custom spout, extend `BaseRichSpout` and implement `open()`, `nextTuple()`, `ack()`, `fail()`, and `declareOutputFields()`. See link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/spout/FileSpout.java[FileSpout] for a simple example. For spouts backed by a storage layer, extend link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/AbstractQueryingSpout.java[AbstractQueryingSpout] instead.
+
+=== Politeness and Rate Limiting
+
+StormCrawler enforces politeness at multiple levels to avoid overloading target servers.
+
+==== Queue-Based Politeness
+
+The **FetcherBolt** groups incoming URLs into queues based on the `fetcher.queue.mode` setting:
+
+* `byHost` (default) — one queue per hostname
+* `byDomain` — one queue per registered domain (e.g., `sub.example.com` and `example.com` share a queue)
+* `byIP` — one queue per resolved IP address
+
+Each queue enforces a minimum delay between consecutive requests. The delay is determined by:
+
+1. The `Crawl-delay` directive in the site's `robots.txt` (if present and within limits)
+2. The `fetcher.server.delay` default (1 second)
+3. For multi-threaded queues, `fetcher.server.min.delay` is used instead
+
+==== Crawl-Delay Handling
+
+When `robots.txt` specifies a `Crawl-delay`:
+
+* If the delay is *smaller* than `fetcher.server.delay` and `fetcher.server.delay.force` is `true`, the larger default delay is used instead
+* If the delay is *larger* than `fetcher.max.crawl.delay`:
+** `fetcher.max.crawl.delay.force: false` (default) — the queue is skipped and the URL gets an ERROR status
+** `fetcher.max.crawl.delay.force: true` — the delay is capped at `fetcher.max.crawl.delay`
+
+==== Robots.txt Compliance
+
+StormCrawler fetches and caches `robots.txt` rules per host. The cache is configured via:
+
+* `robots.cache.spec` — Guava cache spec for valid rules (default: 6 hour expiry, 10000 entries)
+* `robots.error.cache.spec` — cache spec for error responses (default: 1 hour expiry)
+
+Individual aspects of robot compliance can be toggled:
+
+* `http.robots.file.skip` — ignore `robots.txt` entirely
+* `http.robots.headers.skip` — ignore `X-Robots-Tag` HTTP headers
+* `http.robots.meta.skip` — ignore `<meta name="robots">` HTML tags
+* `robots.noFollow.strict` — if `true`, remove all outlinks from pages marked as `noFollow`
+
+=== Error Handling and Retry Logic
+
+StormCrawler uses a status-based system to handle errors and schedule retries.
+
+==== Status Lifecycle
+
+Each URL has a `Status` value that drives scheduling:
+
+* **DISCOVERED** — new URL, not yet fetched
+* **FETCHED** — successfully retrieved and processed
+* **FETCH_ERROR** — transient error (network timeout, HTTP 5xx). Will be retried.
+* **ERROR** — terminal error (too many retries, HTTP 4xx). Not retried by default.
+* **REDIRECTION** — HTTP 3xx redirect encountered
+
+==== Retry Mechanism
+
+When a URL receives `FETCH_ERROR` status, the `AbstractStatusUpdaterBolt` increments a `fetch.error.count` counter in the URL's metadata. Once this counter reaches `max.fetch.errors` (default: 3), the status is escalated to `ERROR`.
+
+The `DefaultScheduler` computes the next fetch date based on status:
+
+* **FETCHED** — `fetchInterval.default` (default: 1440 minutes = 1 day)
+* **FETCH_ERROR** — `fetchInterval.fetch.error` (default: 120 minutes = 2 hours)
+* **ERROR** — `fetchInterval.error` (default: -1 = never refetch)
+
+On a successful fetch, the error counter and error metadata (`error.cause`, `error.message`, `error.source`) are cleared automatically.
+
+==== Custom Fetch Intervals
+
+You can set fetch intervals based on metadata key-value pairs:
+
+[source,yaml]
+----
+# All feeds refetch every 10 minutes
+fetchInterval.isFeed=true: 10
+
+# Feeds with fetch errors retry after 60 minutes
+fetchInterval.FETCH_ERROR.isFeed=true: 60
+----
+
+=== Monitoring and Metrics
+
+StormCrawler exposes metrics through Apache Storm's built-in metrics system. Bolts register counters, gauges, and averages which are reported at configurable intervals.
+
+==== Key Fetcher Metrics
+
+The `FetcherBolt` tracks:
+
+* `fetcher_counter` — event counts (fetched, error, redirection, robots_rules_fetches)
+* `fetcher_average` — averaged values (download_time, queue_wait_time)
+* `fetcher_average_persec` — throughput rates (fetched_per_sec, error_per_sec)
+* `activethreads` — number of currently active fetching threads
+
+The reporting interval is controlled by `fetcher.metrics.time.bucket.secs` (default: 10 seconds).
+
+==== Metrics Consumers
+
+To persist metrics, register a `MetricsConsumer` in your topology configuration:
+
+[source,yaml]
+----
+topology.metrics.consumer.register:
+  - class: "org.apache.stormcrawler.opensearch.metrics.MetricsConsumer"
+    parallelism.hint: 1
+----
+
+StormCrawler provides MetricsConsumer implementations for link:https://github.com/apache/stormcrawler/tree/main/external/opensearch[OpenSearch], link:https://github.com/apache/stormcrawler/tree/main/external/solr[Solr], and link:https://github.com/apache/stormcrawler/tree/main/external/sql[SQL].
+
+The **Storm UI** (default port 8080) provides real-time visibility into topology health, including execute/process latency, emit counts, and error rates per bolt and spout.
+
+=== Scaling and Tuning
+
+==== Thread and Parallelism Tuning
+
+* `fetcher.threads.number` — total fetching threads across all queues (default: 10). Increase for broader crawls.
+* `fetcher.threads.per.queue` — threads per host/domain/IP queue (default: 1). Increase only for sites that allow aggressive crawling.
+* `fetcher.maxThreads.host/domain/ip` — override threads for a specific queue.
+* `protocol.instances.num` — number of protocol instances (default: 1). Increase for OkHttp connection pool efficiency.
+
+In Flux or topology code, set `parallelism` for each bolt to control Storm executor allocation. Increase spout parallelism when the frontier is the bottleneck.
+
+==== Queue and Buffer Tuning
+
+* `fetcher.max.urls.in.queues` — caps total queued URLs to prevent memory exhaustion. -1 for unlimited.
+* `fetcher.max.queue.size` — caps individual queue length.
+* `urlbuffer.class` — URL buffer implementation used by spouts. The default `SimpleURLBuffer` is suitable for most use cases.
+
+==== Memory and Connection Pooling (OkHttp)
+
+For crawls with many concurrent hosts, tune OkHttp's connection pool:
+
+[source,yaml]
+----
+okhttp.protocol.connection.pool:
+  max.idle.connections: 5
+  connection.keep.alive: 300
+----
+
+Keep the pool size (idle + active) under 1000. For large crawls, increase `protocol.instances.num` to distribute connections across multiple pools.
+
+=== Security Considerations
+
+==== SSL/TLS
+
+By default, StormCrawler's OkHttp protocol trusts all certificates (`http.trust.everything: true`). For production crawls over untrusted networks, consider setting this to `false` and configuring a proper trust store.
+
+==== Authentication
+
+**Basic Authentication** is supported via:
+
+[source,yaml]
+----
+http.basicauth.user: "username"
+http.basicauth.password: "password"
+----
+
+These credentials are sent as an `Authorization` header with every request. For per-site authentication, use metadata-driven headers instead (see <<Metadata-dependent Behavior For HTTP Protocols>> in the Internals section).
+
+==== Proxy Authentication
+
+Proxy credentials can be configured via:
+
+[source,yaml]
+----
+http.proxy.host: "proxy.example.com"
+http.proxy.port: 8080
+http.proxy.user: "proxyuser"
+http.proxy.pass: "proxypass"
+----
+
+For rotating proxies, use the `MultiProxyManager` with a proxy list file. See the xref:configuration.adoc[Proxy configuration section] for details.
diff --git a/docs/src/main/asciidoc/index.adoc b/docs/src/main/asciidoc/index.adoc
index 8c99dbb5..8a71db35 100644
--- a/docs/src/main/asciidoc/index.adoc
+++ b/docs/src/main/asciidoc/index.adoc
@@ -26,6 +26,8 @@ include::internals.adoc[]
 
 include::configuration.adoc[]
 
+include::extending.adoc[]
+
 include::debugging.adoc[]
 
 include::powered-by.adoc[]
diff --git a/docs/src/main/asciidoc/internals.adoc b/docs/src/main/asciidoc/internals.adoc
index 6f517adc..4896e18d 100644
--- a/docs/src/main/asciidoc/internals.adoc
+++ b/docs/src/main/asciidoc/internals.adoc
@@ -21,7 +21,7 @@ A bolt which sends tuples on the _status_ stream declares its output in the foll
 [source,java]
 ----
 declarer.declareStream(
-    org.apache.storm.crawler.Constants.StatusStreamName,
+    org.apache.stormcrawler.Constants.StatusStreamName,
     new Fields("url", "metadata", "status"));
 ----
 
@@ -33,7 +33,7 @@ The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org
 * REDIRECTION:: set by the fetcher bolts.
 * FETCH_ERROR:: set by the fetcher bolts.
 * ERROR:: used by either the fetcher, parser, or indexer bolts.
-* FETCHED:: set by the StatusStreamBolt bolt (see below).
+* FETCHED:: set by the DummyIndexer bolt (see below).
 
 The difference between FETCH_ERROR and ERROR is that the former is possibly transient whereas the latter is terminal. The bolt which is in charge of updating the status (see below) can then decide when and whether to schedule a new fetch for a URL based on the status value.
 
@@ -43,6 +43,23 @@ The class link:https://github.com/apache/stormcrawler/blob/main/core/src/main/ja
 
 In most cases, the extending classes will just need to implement the method `store(String URL, Status status, Metadata metadata, Date nextFetch)` and handle their own initialisation in `prepare()`. You can find an example of a class which extends it in the link:https://github.com/apache/stormcrawler/blob/main/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/persistence/StatusUpdaterBolt.java[StatusUpdaterBolt] for OpenSearch.
 
+==== AdaptiveScheduler
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/AdaptiveScheduler.java[AdaptiveScheduler] extends the `DefaultScheduler` and adjusts the fetch interval based on whether a page has changed since the last fetch (detected via MD5 signature comparison). If the page changed, the interval shrinks toward a minimum; if unchanged, it grows toward a maximum. This avoids unnecessary re-fetches of static pages while keeping dynamic pag [...]
+
+Configuration example:
+
+[source,yaml]
+----
+scheduler.class: "org.apache.stormcrawler.persistence.AdaptiveScheduler"
+scheduler.adaptive.setLastModified: true
+scheduler.adaptive.fetchInterval.min: 60
+scheduler.adaptive.fetchInterval.max: 20160
+scheduler.adaptive.fetchInterval.rate.incr: .5
+scheduler.adaptive.fetchInterval.rate.decr: .5
+----
+
+The `AdaptiveScheduler` requires the `MD5SignatureParseFilter` to be configured so that the `signature` metadata field is populated.
+
 
 === Bolts
 
@@ -59,11 +76,11 @@ Both declare the same output:
 ----
 declarer.declare(new Fields("url", "content", "metadata"));
 declarer.declareStream(
-        org.apache.storm.crawler.Constants.StatusStreamName,
+        org.apache.stormcrawler.Constants.StatusStreamName,
         new Fields("url", "metadata", "status"));
 ----
 
-with the `StatusStream` being used for handling redirections, restrictions by robots directives, or fetch errors, whereas the default stream gets the binary content returned by the server as well as the metadata to the following components (typically a parsing bolt).
+with the status stream being used for handling redirections, restrictions by robots directives, or fetch errors, whereas the default stream gets the binary content returned by the server as well as the metadata to the following components (typically a parsing bolt).
 
 Both use the same xref:protocols[Protocols] implementations and xref:urlfilters[URLFilters] to control the redirections.
 
@@ -108,7 +125,7 @@ The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org
 
 This parser calls the xref:urlfilters[URLFilters] and xref:parsefilters[ParseFilters] defined in the configuration. Please note that it calls xref:metadatatransfer[MetadataTransfer] prior to calling the xref:parsefilters[ParseFilters]. If you create new Outlinks in your [[ParseFilters]], you'll need to make sure that you use MetadataTransfer there to inherit the Metadata from the parent document.
 
-The **JSoupParserBolt** automatically identifies the charset of the documents. It uses the link:StatusStream[status stream] to report parsing errors but also for the outlinks it extracts from a page. These would typically be used by an extension of link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/AbstractStatusUpdaterBolt.java[AbstractStatusUpdaterBolt] and persisted in some form of storage.
+The **JSoupParserBolt** automatically identifies the charset of the documents. It uses the status stream to report parsing errors but also for the outlinks it extracts from a page. These would typically be used by an extension of link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/AbstractStatusUpdaterBolt.java[AbstractStatusUpdaterBolt] and persisted in some form of storage.
 
 ==== SiteMapParserBolt
 StormCrawler can handle sitemap files thanks to the **SiteMapParserBolt**. This bolt should be placed before the standard **ParserBolt** in the topology, as illustrated in link:https://github.com/apache/stormcrawler/blob/main/archetype/src/main/resources/archetype-resources/src/main/java/CrawlTopology.java[CrawlTopology].
@@ -117,6 +134,28 @@ The reason for this is that the **SiteMapParserBolt** acts as a filter: it passe
 
 The **SiteMapParserBolt** applies any configured xref:parsefilters[ParseFilters] to the documents it parses and, just like its equivalent for HTML pages, it uses xref:metadatatransfer[MetadataTransfer] to populate the Metadata objects for the Outlinks it finds.
 
+==== FeedParserBolt
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/FeedParserBolt.java[FeedParserBolt] handles RSS and Atom feeds. Like the **SiteMapParserBolt**, it acts as a filter in the topology: it passes non-feed tuples through on the default stream, and only parses tuples whose metadata contains `isFeed=true`. Discovered feed entries are emitted on the status stream. The bolt is configured with `feed.sniffContent` (to auto-detect feeds) and ` [...]
+
+==== URLPartitionerBolt
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/URLPartitionerBolt.java[URLPartitionerBolt] partitions incoming URLs by host, domain, or IP address (controlled by `partition.url.mode`). It is typically placed before a fetcher bolt to ensure that URLs destined for the same server are routed to the same bolt instance, enabling effective politeness enforcement. It uses the link:https://github.com/apache/stormcrawler/blob/main/core/sr [...]
+
+==== StatusEmitterBolt
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/bolt/StatusEmitterBolt.java[StatusEmitterBolt] is an abstract base class for bolts that emit tuples on both the default stream and the status stream. It provides utility methods for emitting status updates (e.g., errors, redirections, discovered URLs). The fetcher and parser bolts (`FetcherBolt`, `SimpleFetcherBolt`, `JSoupParserBolt`, `SiteMapParserBolt`, `FeedParserBolt`) all extend thi [...]
+
+=== Spouts
+
+StormCrawler provides several spout implementations for feeding URLs into the topology:
+
+==== FileSpout
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/spout/FileSpout.java[FileSpout] reads seed URLs from UTF-8 text files and loads them into memory. Each line is parsed using `StringTabScheme` into a URL and optional metadata (tab-separated). This is useful for one-off crawls with a known list of URLs.
+
+==== MemorySpout
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/spout/MemorySpout.java[MemorySpout] stores URLs in an in-memory priority queue. It is intended for testing and debugging in local mode or with a single worker. It can be used with `MemoryStatusUpdater` to emulate a recursive crawl without an external storage backend.
+
+==== AbstractQueryingSpout
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/persistence/AbstractQueryingSpout.java[AbstractQueryingSpout] is the abstract base class for spouts that query a storage backend for URLs to fetch. External module spouts such as `AggregationSpout` (OpenSearch), `SolrSpout`, `SQLSpout`, and the URLFrontier `Spout` all extend this class. It handles throttling, acking, and coordination with the URL buffer (`urlbuffer.class`).
+
 === Filters
 
 [[parsefilters]]
@@ -137,7 +176,7 @@ public boolean needsDOM();
 
 * The `filter` method is where the extraction occurs. ParseResult objects contain the outlinks extracted from the document as well as a Map of String to ParseData, where the String is the URL of a subdocument or the main document itself. ParseData objects contain Metadata, binary content, and text for the subdocuments, which is useful for indexing subdocuments independently of the main document.
 * The `needsDOM` method indicates whether the ParseFilter instance requires the DOM structure. If no ParseFilters need it, the parsing bolt will skip generating the DOM, slightly improving performance.
-* The `configure` method takes a JSON object loaded by the wrapper class ParseFilters. The Storm configuration map can also be used to configure the filters, as described in link:Configuration[Configuration].
+* The `configure` method takes a JSON object loaded by the wrapper class ParseFilters. The Storm configuration map can also be used to configure the filters, as described in xref:configuration.adoc[Configuration].
 
 Here is the default link:https://github.com/apache/stormcrawler/blob/main/core/src/main/resources/parsefilters.json[JSON configuration file] for ParseFilters. The configuration allows multiple instances of the same filter class with different parameters and supports complex parameter objects. ParseFilters are executed in the order they appear in the JSON file.
 
@@ -285,6 +324,9 @@ The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org
 ====== RobotsFilter
 The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/filtering/robots/RobotsFilter.java[RobotsFilter] discards URLs based on the robots.txt directives. This is meant for small, limited crawls where the number of hosts is finite. Using this on a larger or open crawl would impact performance as the filter tries to retrieve the robots.txt files for any host found.
 
+====== SelfURLFilter
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/filtering/basic/SelfURLFilter.java[SelfURLFilter] removes self-referential URLs, i.e., outlinks that point back to the same page they were discovered on (after normalization).
+
 ====== SitemapFilter
 The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/filtering/sitemap/SitemapFilter.java[SitemapFilter] discards the outlinks of URLs which are not sitemaps when sitemaps have been found.
 
@@ -297,11 +339,11 @@ The class https://github.com/apache/stormcrawler/blob/main/core/src/main/java/or
 * Parsing
 * Updating bolts
 
-An instance (or extension) of MetadataTransfer gets created and configured with the method `public static MetadataTransfer getInstance(Map++<++String, Object++>++ conf)` which takes as parameter with the standard Apache Storm configuration.
+An instance (or extension) of MetadataTransfer gets created and configured with the method `public static MetadataTransfer getInstance(Map++<++String, Object++>++ conf)` which takes the standard Apache Storm configuration as a parameter.
 
 A *MetadataTransfer* instance has mainly two methods, both returning Metadata objects :
 
-* `getMetaForOutlink(String targetURL, String sourceURL,           Metadata parentMD)`
+* `getMetaForOutlink(String targetUrl, String sourceUrl, Metadata parentMetadata)`
 * `filter(Metadata metadata)`
 
 The former is used when creating https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/parse/Outlink.java[Outlinks] i.e. in the parsing bolts but also for handling redirections in the FetcherBolt(s)
@@ -309,12 +351,12 @@ The latter is used by extensions of the https://github.com/apache/stormcrawler/b
 
 The behavior of the default MetadataTransfer class is driven by configuration only. It has the following options.
 
-* `metadata.transfer` list of metadata key values to filter or transfer to the outlinks. See https://github.com/apache/stormcrawler/blob/main/core/src/main/resources/crawler-default.yaml#L23[crawler-default.yaml]
-* `metadata.persist` list of metadata key values to persist in the status storage. See https://github.com/apache/stormcrawler/blob/main/core/src/main/resources/crawler-default.yaml#L28[crawler-default.yaml]
+* `metadata.transfer` list of metadata key values to filter or transfer to the outlinks. See https://github.com/apache/stormcrawler/blob/main/core/src/main/resources/crawler-default.yaml#L70[crawler-default.yaml]
+* `metadata.persist` list of metadata key values to persist in the status storage. See https://github.com/apache/stormcrawler/blob/main/core/src/main/resources/crawler-default.yaml#L75[crawler-default.yaml]
 * `metadata.track.path` whether to track the URL path or not. Boolean value, true by default.
 * `metadata.track.depth` whether to track the depth from seed. Boolean value, true by default.
 
-Note that the method `getMetaForOutlink` calls `filter` to determine what to key values to keep.
+Note that the method `getMetaForOutlink` calls `filter` to determine what key values to keep.
 
 [[protocols]]
 === Protocols
@@ -327,6 +369,16 @@ While HTTP and HTTPS are the most commonly used, other protocols like `file:` ar
 
 Use these configurations to fine-tune fetching performance, authentication, connection handling, and protocol-level optimizations across your crawler topology.
 
+==== Following Redirects
+
+When handling HTTP redirects, StormCrawler offers three modes:
+
+1. **Default behavior** (`redirections.allowed: true`, `http.allow.redirects: false`): The redirect target URL is emitted to the status stream. URL filter and normalization rules are applied to the target URL, the crawler verifies that the target URL is allowed per `robots.txt`, and it is ensured that the redirect target is not fetched multiple times (URLs are deduplicated in the status index).
+
+2. **Redirects disabled** (`redirections.allowed: false`): Redirect target URLs are not sent to the status stream. Redirects are effectively ignored.
+
+3. **Immediate follow** (`http.allow.redirects: true`): Redirects are followed immediately in the HTTP client and the target URLs are not emitted to the status stream. This is the default behavior for browser-based protocols (Selenium and Playwright), but it is also supported by the OkHttp protocol. Note that with immediate follows, redirect targets bypass URL filtering, deduplication, and `robots.txt` checks.
+
 ==== Network Protocols
 
 The following network protocols are implemented in StormCrawler:
@@ -334,11 +386,26 @@ The following network protocols are implemented in StormCrawler:
 ===== File
 * link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/file/FileProtocol.java[FileProtocol]
 
+===== DelegatorProtocol
+The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/DelegatorProtocol.java[DelegatorProtocol] enables selection from a collection of sub-protocols using filters based on metadata and URL patterns. The last protocol in the list acts as the default. This is useful when certain URLs need to be fetched with a browser-based protocol (e.g., Selenium) while the majority use a standard HTTP client.
+
+[source,yaml]
+----
+http.protocol.implementation: "org.apache.stormcrawler.protocol.DelegatorProtocol"
+protocol.delegator.config:
+  - className: "org.apache.stormcrawler.protocol.httpclient.HttpProtocol"
+    filters:
+      domain: "example.com"
+  - className: "org.apache.stormcrawler.protocol.okhttp.HttpProtocol"
+----
+
+The protocols are tried in order; the first matching filter wins. Filters can match on metadata keys, domain names, or URL regex patterns.
+
 ===== HTTP/S
 
-See [[HTTPProtocol]] for the effect of metadata content on protocol behaviour.
+See <<Metadata-dependent Behavior For HTTP Protocols>> for the effect of metadata content on protocol behaviour.
 
-To change the implementation, add the following lines to your _crawler-conf.yaml_:
+To change the implementation, add the following lines to your custom configuration YAML file:
 
 [source,yaml]
 ----
@@ -347,7 +414,7 @@ https.protocol.implementation: "org.apache.stormcrawler.protocol.okhttp.HttpProt
 ----
 
 * link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/httpclient/HttpProtocol.java[HttpClient]
-* link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java[Selenium]
+* link:https://github.com/apache/stormcrawler/blob/main/external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java[Selenium]
 * link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java[OKHttp]
 
 ==== Feature grid
@@ -376,7 +443,7 @@ https.protocol.implementation: "org.apache.stormcrawler.protocol.okhttp.HttpProt
 
 * The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java[OKHttp] protocol supports link:https://en.wikipedia.org/wiki/HTTP/2[HTTP/2] if the JDK includes link:https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation[ALPN] (Java 9 and upwards or Java 8 builds starting early/mid 2020).
 * link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/httpclient/HttpProtocol.java[HttpClient] does not yet support HTTP/2.
-* link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java[Selenium]: whether HTTP/2 is used or not depends on the used driver.
+* link:https://github.com/apache/stormcrawler/blob/main/external/selenium/src/main/java/org/apache/stormcrawler/protocol/selenium/SeleniumProtocol.java[Selenium]: whether HTTP/2 is used or not depends on the used driver.
 
 Since link:https://github.com/apache/stormcrawler/pull/829[#829], the HTTP protocol version used is configurable via `http.protocol.versions` (see also comments in link:https://github.com/apache/stormcrawler/blob/main/core/src/main/resources/crawler-default.yaml[crawler-default.yaml]).
 
diff --git a/docs/src/main/asciidoc/powered-by.adoc b/docs/src/main/asciidoc/powered-by.adoc
index fa12f0c7..e333ce82 100644
--- a/docs/src/main/asciidoc/powered-by.adoc
+++ b/docs/src/main/asciidoc/powered-by.adoc
@@ -34,7 +34,7 @@ If your organization is also making use of Apache StormCrawler, we’d love to h
 * link:https://www.clarin.eu/[Clarin ERIC]
 * link:https://openwebsearch.eu/owler/[OpenWebSearch]
 * link:https://shc-info.zml.hs-heilbronn.de/[Heilbronn University]
-* link:https://www.contexity.com[Contexity]
+* link:https://www.contexity.com/[Contexity]
 * link:https://www.kodis.iao.fraunhofer.de/de/projekte/SPIDERWISE.html[Fraunhofer IAO - KODIS]
 
 Drop us a line at mailto:dev@stormcrawler.apache.org[dev@stormcrawler.apache.org] if you want to be added to this page.
diff --git a/docs/src/main/asciidoc/quick-start.adoc b/docs/src/main/asciidoc/quick-start.adoc
index da5866a4..9fd61e97 100644
--- a/docs/src/main/asciidoc/quick-start.adoc
+++ b/docs/src/main/asciidoc/quick-start.adoc
@@ -17,7 +17,7 @@ make sure Docker is installed on your operating system.
 
 === Terminology
 
-Before starting, we will give a quick overview of **central** Storm concepts and terminology, you need to know before starting with StormCrawler:
+Before starting, we will give a quick overview of **essential** Storm concepts and terminology that you need to know before starting with StormCrawler:
 
 - *Topology*: A topology is the overall data processing graph in Storm, consisting of spouts and bolts connected together to perform continuous, real-time computations.
 
@@ -59,7 +59,7 @@ The archetype will generate a fully-structured project including:
 * A pre-configured `pom.xml` with the necessary dependencies
 * Default resource files
 * A sample `crawler.flux` configuration
-* A basic configuration file
+* A sample YAML configuration file (`crawler-conf.yaml`)
 
 After generation, navigate into the newly created directory (named after the `artifactId` you specified).
 
@@ -90,7 +90,7 @@ services:
   supervisor:
     image: storm:latest
     container_name: supervisor
-    command: storm supervisor -c worker.childopts=-Xmx%HEAP-MEM%m
+    command: storm supervisor -c worker.childopts=-Xmx2g
     depends_on:
       - nimbus
       - zookeeper
@@ -129,14 +129,14 @@ After setting up your Docker Compose, you should start it up:
 docker compose up -d
 ----
 
-Check the logs and see, if every service is up and running:
+Check the logs and see if every service is up and running:
 
 [source,shell]
 ----
 docker compose logs -f
 ----
 
-Next, access the Storm UI via `https://localhost:8080` and check, that a Storm Nimbus as well as a Storm Supervisor is available.
+Next, access the Storm UI via `http://localhost:8080` and check that a Storm Nimbus as well as a Storm Supervisor is available.
 
 ==== Compile