(stormcrawler) branch main updated: #2001 Document adaptive scheduler metadata (#2002)

[jn...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

jnioche pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git


The following commit(s) were added to refs/heads/main by this push:
     new eaa6869e #2001 Document adaptive scheduler metadata (#2002)
eaa6869e is described below

commit eaa6869ea60454f09e009c89c9e2d8528594f2f9
Author: Yurii Bakurov <45...@users.noreply.github.com>
AuthorDate: Fri Jul 17 22:34:33 2026 +0200

    #2001 Document adaptive scheduler metadata (#2002)
    
    * Document adaptive scheduler metadata (#2001)
    
    * #2001 Clarify conditional request metadata
---
 docs/src/main/asciidoc/internals.adoc | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/docs/src/main/asciidoc/internals.adoc b/docs/src/main/asciidoc/internals.adoc
index f4287fc9..7dc32d5c 100644
--- a/docs/src/main/asciidoc/internals.adoc
+++ b/docs/src/main/asciidoc/internals.adoc
@@ -56,9 +56,17 @@ scheduler.adaptive.fetchInterval.min: 60
 scheduler.adaptive.fetchInterval.max: 20160
 scheduler.adaptive.fetchInterval.rate.incr: .5
 scheduler.adaptive.fetchInterval.rate.decr: .5
+
+metadata.persist:
+  - signature
+  - fetch.statusCode
+  - fetchInterval
+  - signatureChangeDate
+  - last-modified
+  - protocol.etag
 ----
 
-The `AdaptiveScheduler` requires the `MD5SignatureParseFilter` to be configured so that the `signature` metadata field is populated.
+The `AdaptiveScheduler` requires the `MD5SignatureParseFilter` to be configured so that the `signature` metadata field is populated. The scheduler also persists its current `fetchInterval` and the metadata used to detect content changes. The `last-modified` field (written when `scheduler.adaptive.setLastModified` is enabled) and `protocol.etag` allow conditional requests (`If-Modified-Since` / `If-None-Match`).
 
 
 === Bolts