[PR] MultiProxyManager: thread-safe round-robin rotation [stormcrawler]

["dpol1 (via GitHub)" <gi...@apache.org>]

dpol1 opened a new pull request, #2228:
URL: https://github.com/apache/stormcrawler/pull/2228

   With `http.proxy.manager: org.apache.stormcrawler.proxy.MultiProxyManager` and `http.proxy.rotation` left at its default `ROUND_ROBIN`, some fetches fail when the rotation wraps around the end of the proxy list. The URL gets the status `FETCH_ERROR` with `fetch.exception: java.lang.ArrayIndexOutOfBoundsException` in its metadata. When it happens on a robots.txt fetch, the log shows `Couldn't get robots.txt for ...` and the host gets allow-all rules until the robots error cache expires (1 hour by default).
   
   The fetcher threads share the proxy manager of the protocol instance they use. `getRoundRobin()` checks the index and increments it in two separate steps. Two threads can both pass the check on the last index, and the second one then reads past the end of the proxy array.
   
   The new test in `MultiProxyManagerTest` calls `getProxy()` from 8 threads. It failed in 3 runs out of 3 on main with `Index 3 out of bounds for length 3`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] MultiProxyManager: thread-safe round-robin rotation [stormcrawler]

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 merged PR #2228:
URL: https://github.com/apache/stormcrawler/pull/2228


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org