You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "jnioche (via GitHub)" <gi...@apache.org> on 2026/04/01 07:55:43 UTC

[PR] Add bolt-level timeout for fetcher threads [stormcrawler]

jnioche opened a new pull request, #1861:
URL: https://github.com/apache/stormcrawler/pull/1861

   ## Summary
   - Adds a configurable hard timeout (`fetcher.thread.timeout`) that wraps `protocol.getProtocolOutput()` calls in both `FetcherBolt` and `SimpleFetcherBolt`
   - When a fetch exceeds the timeout, the protocol thread is interrupted, the URL is marked as `FETCH_ERROR`, and the fetcher thread moves on to the next item
   - Disabled by default (`-1`), relying on protocol-level socket timeouts only
   
   Previously, if the protocol layer hung beyond socket timeouts (due to client library bugs, misconfigured timeouts, or misbehaving proxies), fetcher threads would block indefinitely with no recovery mechanism.
   
   Fixes #996
   
   ## Test plan
   - [x] New `testThreadTimeout` test in `AbstractFetcherBoltTest` (inherited by both `FetcherBoltTest` and `SimpleFetcherBoltTest`) — uses WireMock with a 10s delay and a 2s bolt timeout to verify the thread is reclaimed and a `FETCH_ERROR` is emitted
   - [x] All 221 core tests pass
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add bolt-level timeout for fetcher threads [stormcrawler]

Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche merged PR #1861:
URL: https://github.com/apache/stormcrawler/pull/1861


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org