[jira] [Created] (NUTCH-3058) Fetcher: counter for hung threads

["Sebastian Nagel (Jira)" <ji...@apache.org>]

Sebastian Nagel created NUTCH-3058:
--------------------------------------

             Summary: Fetcher: counter for hung threads
                 Key: NUTCH-3058
                 URL: https://issues.apache.org/jira/browse/NUTCH-3058
             Project: Nutch
          Issue Type: Improvement
          Components: fetcher
    Affects Versions: 1.20
            Reporter: Sebastian Nagel
            Assignee: Sebastian Nagel
             Fix For: 1.21


The Fetcher class defines a "hard" timeout defined as 50% of the MapReduce task timeout, see {{mapreduce.task.timeout}} and {{fetcher.threads.timeout.divisor}}. If there are fetcher threads running but without any progress during the timeout period (in terms of newly started fetch items), Fetcher is shut down to avoid that the task timeout is reached and the fetcher job is failed. The "hung threads" are logged together with the URL being fetched and (DEBUG level) the Java stack.

In addition to logging, a job counter should indicate the number of hung threads. This would allow to see on the job level whether there are issues with hung threads. To trace the issues it's still required to look into the Hadoop task logs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)