[jira] [Updated] (NUTCH-2014) Fetcher hang-up on completion

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sebastian Nagel updated NUTCH-2014:
-----------------------------------
    Attachment: NUTCH-2014-v1.patch

The reason is a mix-up of the counters for active threads and fetch errors: the former counter was incremented to 11 just after an error according to the logs:
{noformat}
2015-05-15 21:51:37,399 INFO  fetcher.Fetcher - -activeThreads=10, ...
...
2015-05-15 21:51:38,279 INFO  fetcher.FetcherThread - fetch of ... failed with: ...
...
2015-05-15 21:51:38,399 INFO  fetcher.Fetcher - -activeThreads=11, ...
{noformat}

> Fetcher hang-up on completion
> -----------------------------
>
>                 Key: NUTCH-2014
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2014
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.11
>            Reporter: Sebastian Nagel
>            Priority: Critical
>             Fix For: 1.11
>
>         Attachments: NUTCH-2014-v1.patch
>
>
> Although fetcher has done its work it does not shut down and exit but continues to log (and before reports its status to the task tracker):
> {noformat}
> -activeThreads=11, spinWaiting=0, fetchQueues.totalSize=33, fetchQueues.getQueueCount=1
> -activeThreads=11, spinWaiting=10, fetchQueues.totalSize=26, fetchQueues.getQueueCount=1
> -activeThreads=11, spinWaiting=9, fetchQueues.totalSize=9, fetchQueues.getQueueCount=1
> -activeThreads=9, spinWaiting=7, fetchQueues.totalSize=0, fetchQueues.getQueueCount=1
> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=0
> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=0
> ...
> (last message continues)
> {noformat}
> A possible hint: activeThreads should never exceed 10 (configured per default). Looks like the corresponding variable is lost/mixed-up during fetcher refactorization (NUTCH-1934).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)