You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "GGraziadei (via GitHub)" <gi...@apache.org> on 2026/09/24 09:36:51 UTC

[PR] #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl [stormcrawler]

GGraziadei opened a new pull request, #2186:
URL: https://github.com/apache/stormcrawler/pull/2186

   Fixes #2185
   
   With URLFrontier the order of the crawl is decided by the frontier: rotation between host queues, FIFO within each queue, depth plays no role. This PR adds observability of the depth of what the frontier hands out, without changing the behaviour of the crawl.
   
   ## Changes
   
   - `Spout` registers a `depth` counter with one scope per depth level read from the `depth` metadata (`depth.0`, `depth.1`, ...), an `unknown` scope for URLs without a numeric depth, and a single `N+` scope from `urlfrontier.depth.metric.max` (default 10) upwards, so that the number of scopes stays bounded when no `MaxDepthFilter` is configured.
   - A `depth_le` counter with cumulative counts in the style of Prometheus histograms: `depth_le.X` is the number of URLs with depth <= X, `depth_le.inf` the number of URLs with a valid depth, so P(depth <= X) is a plain ratio in a dashboard.
   - `probabilityDepthAtMost(int)` on the spout returns that ratio in constant time from an in-memory copy of the cumulative counts, since the spout was opened.
   - `DepthWindow`, a ring of ten time slots covering `urlfrontier.depth.window.secs` (default 300), and `windowProbabilityDepthAtMost(int)` returning the same ratio over the recent URLs only, for a future control loop in `populateBuffer`.
   - README section documenting the metrics and the two configuration keys.
   
   ## Tests
   
   - `SpoutDepthMetricTest`: counts per level, `unknown` and `N+` buckets, configurable cap, cumulative counters, probability values, NaN without data, window wiring.
   - `DepthWindowTest`: recent data, cap, expiry beyond the window, slot reuse after a full turn, NaN when empty.
   - `mvn -pl external/urlfrontier verify -Dskip.format.code=false`: 91 tests, formatter and Checkstyle clean.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl [stormcrawler]

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on PR #2186:
URL: https://github.com/apache/stormcrawler/pull/2186#issuecomment-5812977032

   Hi @jnioche thanks a lot for your comment. I’ll come back in a couple of days with a refactoring. Moving this to draft  and continuing the conversation here #2185 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl [stormcrawler]

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei closed pull request #2186: #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl
URL: https://github.com/apache/stormcrawler/pull/2186


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] #2185 URLFrontier Spout: depth metrics to observe the breadth of the crawl [stormcrawler]

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on PR #2186:
URL: https://github.com/apache/stormcrawler/pull/2186#issuecomment-5812951828

   I checked the queue selection in url-frontier on `master` (HEAD 43c58c3, 2026-09-22) and on the 2.3.1, 2.5 and 2.6 tags, since the behaviour matters for how this is framed.
   
   **Between queues** the frontier does not look at `nextFetchDate`: `getURLs` takes the first queue of the insertion-ordered map and moves it to the tail (`rotateFirstEntry()` on `ConcurrentInsertionOrderMap` since 2.6, `pollFirstEntry()` + `put` in 2.5, `iterator.remove()` + `put` in 2.3.1). The only gate is politeness (`blockedUntil`, `delay`, `lastProduced`); `QueueInterface` has no notion of a date. So the selection is round-robin across queues, in every release.
   
   **Within a queue** you are right and my description was imprecise: URLs are served by `nextFetchDate` (RocksDB scheduling key `crawl_queue_paddedDate_url`, the iterator stops at the first future date), not FIFO. Discovered URLs get `Instant.now()` on insertion, so among them it is arrival order, which is why depth ends up correlated with the position in the queue; known URLs with a refetch date interleave by date. I will fix the wording in the issue and in the description.
   
   On the rest I agree: steering belongs in url-frontier, and this PR should stay pure observability. On where to observe: `StatusMetricsBolt` polls the store on tick tuples, while the depth has to be read from the tuples the spout emits, so the equivalent is a bolt subscribed to the spout's default stream that reads the `depth` metadata, counts and acks. That keeps the URLFrontier spout untouched, works with any backend (OpenSearch, SQL, ...) and can live in `core`. Unless you prefer otherwise I will rework the PR that way: `DepthMetricsBolt` in `core` with the `depth` / `depth_le` counters and the windowed P(depth <= X) exposed as a gauge, nothing left in the spout.
   
   Measuring inside url-frontier would need the frontier to know which metadata key carries the depth (it is a StormCrawler key); I can open an issue on crawler-commons for a configurable key, as a complement rather than a replacement, since a frontier-side metric could not be broken down per topology.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org