[jira] [Commented] (NUTCH-3177) Fetcher to report idle threads not as hung threads

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080773#comment-18080773 ] 

ASF GitHub Bot commented on NUTCH-3177:
---------------------------------------

sebastian-nagel opened a new pull request, #915:
URL: https://github.com/apache/nutch/pull/915

   - reset reprUrl in FetcherThread after fetch is finished
   - report idle threads properly
   




> Fetcher to report idle threads not as hung threads
> --------------------------------------------------
>
>                 Key: NUTCH-3177
>                 URL: https://issues.apache.org/jira/browse/NUTCH-3177
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.22
>            Reporter: Sebastian Nagel
>            Priority: Minor
>             Fix For: 1.23
>
>
> If there is no URL fetched during half of the MapReduce task timeout, Fetcher is shutting down to avoid that the fetcher map task fails because of missing progress. Before the shut-down Fetcher reports the remaining FetcherThreads as "hung threads" together with the fetched URL. This should allow to debug the URLs / pages causing timeouts. For the reporting the field {{reprUrl}} of FetcherThread is used. However, the field is not reset after a fetch is done. In consequence, the reported URL is not necessarily one where the fetch is in process. It might a the URL that was fetched last, but the thread is no idle and waiting for the next fetch item to be ready. This happens if there are still fetch queues, but with long delays because of a robots.txt Crawl-delay or a longer delay because of the exponential back-off.
> FetcherThread should reset the {{reprUrl}} once a fetch is finished. Idle FetcherThread shouldn't be reported as hanging.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)