(stormcrawler) 01/02: fix(docs): correct configuration defaults and class references in adoc files

[rz...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

rzo1 pushed a commit to branch doc-improvements
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git

commit 5ad67b4caafc5f5330ea53582c43b563527c2fe6
Author: Richard Zowalla <rz...@apache.org>
AuthorDate: Thu Mar 26 19:21:56 2026 +0100

    fix(docs): correct configuration defaults and class references in adoc files
    
    Fix inaccurate default values, wrong config key names, and a
    nonexistent class reference in the documentation:
    - protocols: add missing "file" to default
    - redirections.allowed → http.allow.redirects (default false, not true)
    - fetchInterval.error: -1, not 44640
    - cacheConfigParamName → robots.cache.spec
    - errorcacheConfigParamName → robots.error.cache.spec
    - http.accept and http.accept.language: add actual defaults
    - protocol.md.prefix: default is "protocol.", not empty
    - FrontierSpout → Spout (class doesn't exist)
---
 docs/src/main/asciidoc/architecture.adoc  |  2 +-
 docs/src/main/asciidoc/configuration.adoc | 18 +++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/docs/src/main/asciidoc/architecture.adoc b/docs/src/main/asciidoc/architecture.adoc
index bbacfae6..040daa54 100644
--- a/docs/src/main/asciidoc/architecture.adoc
+++ b/docs/src/main/asciidoc/architecture.adoc
@@ -29,7 +29,7 @@ Black arrows show the *main data flow*, gray arrows represent URLs *taken from t
 
 The crawl begins with the *Frontier*, which is responsible for scheduling,
 prioritization, politeness, and retry logic. URLs are emitted by a
-`FrontierSpout` and partitioned by the `URLPartitioner`, typically using the
+`Spout` (e.g., `AggregationSpout`, `SQLSpout`) and partitioned by the `URLPartitioner`, typically using the
 host as a key to enforce politeness constraints.
 
 The `Fetcher` retrieves web resources and emits both the fetched content and
diff --git a/docs/src/main/asciidoc/configuration.adoc b/docs/src/main/asciidoc/configuration.adoc
index b5cfbaca..f2fcdef5 100644
--- a/docs/src/main/asciidoc/configuration.adoc
+++ b/docs/src/main/asciidoc/configuration.adoc
@@ -201,14 +201,14 @@ implementation.
 | http.robots.file.skip | false | Ignore robots.txt rules (1.17+).
 | http.skip.robots | false | Deprecated (replaced by http.robots.file.skip).
 | http.store.headers | false | Whether to store response headers.
-| http.store.responsetime | true | Not yet implemented — store response time in Metadata.
+| http.store.responsetime | true | Not yet implemented — intended to store response time in Metadata. Not present in default configuration.
 | http.timeout | 10000 | Connection timeout (ms).
 | http.use.cookies | false | Use cookies in subsequent requests.
 | https.protocol.implementation | org.apache.stormcrawler.protocol.httpclient.HttpProtocol | HTTPS Protocol
 implementation.
 | partition.url.mode | byHost | Defines how URLs are partitioned: byHost, byDomain, or byIP.
-| protocols | http,https | Supported protocols.
-| redirections.allowed | true | Allow URL redirects.
+| protocols | http,https,file | Supported protocols.
+| http.allow.redirects | false | Allow URL redirects.
 | sitemap.discovery | false | Enable automatic sitemap discovery.
 |===
 
@@ -218,13 +218,13 @@ implementation.
 |===
 | key | default value | description
 
-| cacheConfigParamName | maximumSize=10000,expireAfterWrite=6h | CacheBuilder configuration for robots cache.
-| errorcacheConfigParamName | maximumSize=10000,expireAfterWrite=1h | CacheBuilder configuration for error cache.
+| robots.cache.spec | maximumSize=10000,expireAfterWrite=6h | CacheBuilder configuration for robots cache.
+| robots.error.cache.spec | maximumSize=10000,expireAfterWrite=1h | CacheBuilder configuration for error cache.
 | file.encoding | UTF-8 | Encoding for FileProtocol.
 | http.custom.headers | - | Custom HTTP headers.
-| http.accept | - | HTTP Accept
+| http.accept | text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 | HTTP Accept
 header.
-| http.accept.language | - | HTTP Accept-Language
+| http.accept.language | en-us,en-gb,en;q=0.7,*;q=0.3 | HTTP Accept-Language
 header.
 | http.content.partial.as.trimmed | false | Accepts partially fetched content in OKHTTP.
 | http.trust.everything | true | If true, trust all SSL/TLS connections.
@@ -266,7 +266,7 @@ that is being calculated by a link:https://github.com/apache/stormcrawler/blob/m
 
 | fetchInterval.default | 1440 | Default revisit interval (minutes). Used by DefaultScheduler
 .
-| fetchInterval.error | 44640 | Revisit interval for error pages (minutes).
+| fetchInterval.error | -1 | Revisit interval for error pages (minutes). -1 means never refetch.
 | fetchInterval.fetch.error | 120 | Revisit interval for fetch errors (minutes).
 | status.updater.cache.spec | maximumSize=10000, expireAfterAccess=1h | Cache specification
 .
@@ -312,5 +312,5 @@ minimising metadata clashes
 | metadata.track.path | true | Track URL path history in metadata.
 | metadata.transfer | - | Metadata to transfer to outlinks.
 | metadata.transfer.class | org.apache.stormcrawler.util.MetadataTransfer | Class handling metadata transfer.
-| protocol.md.prefix | - | Prefix for remote metadata keys to avoid collisions.
+| protocol.md.prefix | protocol. | Prefix for remote metadata keys to avoid collisions.
 |===