You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Issam Maamria <ma...@gmail.com> on 2014/11/28 15:36:26 UTC
Nutch 1.9 Fetchers Hung
Hi all,
I am running the crawl command with depth 2 using a seed file containing
120 urls (about 2000 documents). Halfway through, the following output is
logged:
*-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=51,
fetchQueues.getQueueCount=24*
And after a while:
*Aborting with 50 hung threads.*
I am trying exactly the same thing using 1.8, and it is *working fine*.
Please note that I am not applying any customisations apart from the
following nutch-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>http.agent.name</name>
<value>MyAgent</value>
</property>
<property>
<name>http.robots.agents</name>
<value> MyAgent,*</value>
<description>The agent strings we'll look for in robots.txt files,
comma-separated, in decreasing order of precedence. You should
put the value of http.agent.name as the first agent name, and keep the
default * at the end of the list. E.g.: BlurflDev,Blurfl,*
</description>
</property>
<!-- HTTP properties -->
<property>
<name>http.redirect.max</name>
<value>2</value>
<description>The maximum number of redirects the fetcher will follow when
trying to fetch a page. If set to negative or 0, fetcher won't immediately
follow redirected URLs, instead it will record them for later fetching.
</description>
</property>
<property>
<name>http.content.limit</name>
<value>-1</value>
<description>The length limit for downloaded content using the http://
protocol, in bytes. If this value is nonnegative (>=0), content longer
than it will be truncated; otherwise, no truncation at all. Do not
confuse this setting with the file.content.limit setting.
</description>
</property>
<!-- web db properties -->
<!-- fetcher properties -->
<property>
<name>fetcher.server.delay</name>
<value>4.0</value>
<description>The number of seconds the fetcher will delay between
successive requests to the same server.</description>
</property>
<property>
<name>fetcher.threads.fetch</name>
<value>20</value>
<description>The number of FetcherThreads the fetcher should use.
This is also determines the maximum number of requests that are
made at once (each FetcherThread handles one connection). The total
number of threads running in distributed mode will be the number of
fetcher threads * number of nodes as fetcher has one map task per node.
</description>
</property>
<property>
<name>fetcher.threads.per.queue</name>
<value>10</value>
<description>This number is the maximum number of threads that
should be allowed to access a queue at one time.
</description>
</property>
<!-- plugin properties -->
<property>
<name>plugin.includes</name>
<value>protocol-http|parse-(html|tika)|index-(basic|anchor)|urlnormalizer
-(pass|regex|basic)</value>
<!-- <value>protocol-http|urlfilter-regex|parse-(html|tika
)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
-->
<description>Regular expression naming plugin directory names to
include. Any plugin not matching this expression is excluded.
In any case you need at least include the nutch-extensionpoints plugin. By
default Nutch includes crawling just HTML and plain text via HTTP,
and basic indexing and search plugins. In order to use HTTPS please
enable
protocol-httpclient, but be aware of possible intermittent problems with
the
underlying commons-httpclient library.
</description>
</property>
<!-- parser properties -->
<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
<description>The character encoding to fall back to when no other
information
is available</description>
</property>
<property>
<name>parser.timeout</name>
<value>-1</value>
<description>Timeout in seconds for the parsing of a document, otherwise
treats it as an exception and
moves on the the following documents. This parameter is applied to any
Parser implementation.
Set to -1 to deactivate, bearing in mind that this could cause
the parsing to crash because of a very long or corrupted document.
</description>
</property>
</configuration>
----
Help is greatly appreciated.
Kind regards,
Issam
Re: Nutch 1.9 Fetchers Hung
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi Issam, hi Markus,
the warning that there are hung threads is shown also in 1.8.
With NUTCH-1182 the hung threads are logged (if they are alive):
- URL in process / being fetched
- with DEBUG logging: stack where thread is hanging
If the problem persists, would it be possible to see more
context from the log file? Ideally with debug logging turned on
in $NUTCH_HOME/conf/log4j.properties :
log4j.logger.org.apache.nutch.fetcher.Fetcher=DEBUG,cmdstdout
Thanks,
Sebastian
On 11/28/2014 03:41 PM, Markus Jelsma wrote:
> I think your're looking at, https://issues.apache.org/jira/browse/NUTCH-1182 logging of hung threads was added in 1.9, so it should happen in 1.8 as well, but not being logged.
> Markus
>
>
>
> -----Original message-----
>> From:Issam Maamria <ma...@gmail.com>
>> Sent: Friday 28th November 2014 15:37
>> To: user@nutch.apache.org
>> Cc: Mourad K <mo...@gmail.com>; Vitaly Savicks <v....@gmail.com>
>> Subject: Nutch 1.9 Fetchers Hung
>>
>> Hi all,
>>
>> I am running the crawl command with depth 2 using a seed file containing
>> 120 urls (about 2000 documents). Halfway through, the following output is
>> logged:
>>
>> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=51,
>> fetchQueues.getQueueCount=24*
>>
>> And after a while:
>>
>> *Aborting with 50 hung threads.*
>>
>> I am trying exactly the same thing using 1.8, and it is *working fine*.
>> Please note that I am not applying any customisations apart from the
>> following nutch-site.xml:
>>
>> <?xml version="1.0"?>
>>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>>
>> <!-- Put site-specific property overrides in this file. -->
>>
>>
>> <configuration>
>>
>> <property>
>>
>> <name>http.agent.name</name>
>>
>> <value>MyAgent</value>
>>
>> </property>
>>
>>
>> <property>
>>
>> <name>http.robots.agents</name>
>>
>> <value> MyAgent,*</value>
>>
>> <description>The agent strings we'll look for in robots.txt files,
>>
>> comma-separated, in decreasing order of precedence. You should
>>
>> put the value of http.agent.name as the first agent name, and keep the
>>
>> default * at the end of the list. E.g.: BlurflDev,Blurfl,*
>>
>> </description>
>>
>> </property>
>>
>>
>> <!-- HTTP properties -->
>>
>>
>> <property>
>>
>> <name>http.redirect.max</name>
>>
>> <value>2</value>
>>
>> <description>The maximum number of redirects the fetcher will follow when
>>
>> trying to fetch a page. If set to negative or 0, fetcher won't immediately
>>
>> follow redirected URLs, instead it will record them for later fetching.
>>
>> </description>
>>
>> </property>
>>
>>
>> <property>
>>
>> <name>http.content.limit</name>
>>
>> <value>-1</value>
>>
>> <description>The length limit for downloaded content using the http://
>>
>> protocol, in bytes. If this value is nonnegative (>=0), content longer
>>
>> than it will be truncated; otherwise, no truncation at all. Do not
>>
>> confuse this setting with the file.content.limit setting.
>>
>> </description>
>>
>> </property>
>>
>>
>> <!-- web db properties -->
>>
>>
>> <!-- fetcher properties -->
>>
>>
>> <property>
>>
>> <name>fetcher.server.delay</name>
>>
>> <value>4.0</value>
>>
>> <description>The number of seconds the fetcher will delay between
>>
>> successive requests to the same server.</description>
>>
>> </property>
>>
>>
>> <property>
>>
>> <name>fetcher.threads.fetch</name>
>>
>> <value>20</value>
>>
>> <description>The number of FetcherThreads the fetcher should use.
>>
>> This is also determines the maximum number of requests that are
>>
>> made at once (each FetcherThread handles one connection). The total
>>
>> number of threads running in distributed mode will be the number of
>>
>> fetcher threads * number of nodes as fetcher has one map task per node.
>>
>> </description>
>>
>> </property>
>>
>>
>> <property>
>>
>> <name>fetcher.threads.per.queue</name>
>>
>> <value>10</value>
>>
>> <description>This number is the maximum number of threads that
>>
>> should be allowed to access a queue at one time.
>>
>> </description>
>>
>> </property>
>>
>>
>> <!-- plugin properties -->
>>
>>
>> <property>
>>
>> <name>plugin.includes</name>
>>
>> <value>protocol-http|parse-(html|tika)|index-(basic|anchor)|urlnormalizer
>> -(pass|regex|basic)</value>
>>
>> <!-- <value>protocol-http|urlfilter-regex|parse-(html|tika
>> )|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
>> -->
>>
>> <description>Regular expression naming plugin directory names to
>>
>> include. Any plugin not matching this expression is excluded.
>>
>> In any case you need at least include the nutch-extensionpoints plugin. By
>>
>> default Nutch includes crawling just HTML and plain text via HTTP,
>>
>> and basic indexing and search plugins. In order to use HTTPS please
>> enable
>>
>> protocol-httpclient, but be aware of possible intermittent problems with
>> the
>>
>> underlying commons-httpclient library.
>>
>> </description>
>>
>> </property>
>>
>>
>> <!-- parser properties -->
>>
>>
>> <property>
>>
>> <name>parser.character.encoding.default</name>
>>
>> <value>utf-8</value>
>>
>> <description>The character encoding to fall back to when no other
>> information
>>
>> is available</description>
>>
>> </property>
>>
>>
>> <property>
>>
>> <name>parser.timeout</name>
>>
>> <value>-1</value>
>>
>> <description>Timeout in seconds for the parsing of a document, otherwise
>> treats it as an exception and
>>
>> moves on the the following documents. This parameter is applied to any
>> Parser implementation.
>>
>> Set to -1 to deactivate, bearing in mind that this could cause
>>
>> the parsing to crash because of a very long or corrupted document.
>>
>> </description>
>>
>> </property>
>>
>> </configuration>
>>
>> ----
>>
>> Help is greatly appreciated.
>>
>> Kind regards,
>>
>> Issam
>>
RE: Nutch 1.9 Fetchers Hung
Posted by Markus Jelsma <ma...@openindex.io>.
I think your're looking at, https://issues.apache.org/jira/browse/NUTCH-1182 logging of hung threads was added in 1.9, so it should happen in 1.8 as well, but not being logged.
Markus
-----Original message-----
> From:Issam Maamria <ma...@gmail.com>
> Sent: Friday 28th November 2014 15:37
> To: user@nutch.apache.org
> Cc: Mourad K <mo...@gmail.com>; Vitaly Savicks <v....@gmail.com>
> Subject: Nutch 1.9 Fetchers Hung
>
> Hi all,
>
> I am running the crawl command with depth 2 using a seed file containing
> 120 urls (about 2000 documents). Halfway through, the following output is
> logged:
>
> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=51,
> fetchQueues.getQueueCount=24*
>
> And after a while:
>
> *Aborting with 50 hung threads.*
>
> I am trying exactly the same thing using 1.8, and it is *working fine*.
> Please note that I am not applying any customisations apart from the
> following nutch-site.xml:
>
> <?xml version="1.0"?>
>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
>
> <!-- Put site-specific property overrides in this file. -->
>
>
> <configuration>
>
> <property>
>
> <name>http.agent.name</name>
>
> <value>MyAgent</value>
>
> </property>
>
>
> <property>
>
> <name>http.robots.agents</name>
>
> <value> MyAgent,*</value>
>
> <description>The agent strings we'll look for in robots.txt files,
>
> comma-separated, in decreasing order of precedence. You should
>
> put the value of http.agent.name as the first agent name, and keep the
>
> default * at the end of the list. E.g.: BlurflDev,Blurfl,*
>
> </description>
>
> </property>
>
>
> <!-- HTTP properties -->
>
>
> <property>
>
> <name>http.redirect.max</name>
>
> <value>2</value>
>
> <description>The maximum number of redirects the fetcher will follow when
>
> trying to fetch a page. If set to negative or 0, fetcher won't immediately
>
> follow redirected URLs, instead it will record them for later fetching.
>
> </description>
>
> </property>
>
>
> <property>
>
> <name>http.content.limit</name>
>
> <value>-1</value>
>
> <description>The length limit for downloaded content using the http://
>
> protocol, in bytes. If this value is nonnegative (>=0), content longer
>
> than it will be truncated; otherwise, no truncation at all. Do not
>
> confuse this setting with the file.content.limit setting.
>
> </description>
>
> </property>
>
>
> <!-- web db properties -->
>
>
> <!-- fetcher properties -->
>
>
> <property>
>
> <name>fetcher.server.delay</name>
>
> <value>4.0</value>
>
> <description>The number of seconds the fetcher will delay between
>
> successive requests to the same server.</description>
>
> </property>
>
>
> <property>
>
> <name>fetcher.threads.fetch</name>
>
> <value>20</value>
>
> <description>The number of FetcherThreads the fetcher should use.
>
> This is also determines the maximum number of requests that are
>
> made at once (each FetcherThread handles one connection). The total
>
> number of threads running in distributed mode will be the number of
>
> fetcher threads * number of nodes as fetcher has one map task per node.
>
> </description>
>
> </property>
>
>
> <property>
>
> <name>fetcher.threads.per.queue</name>
>
> <value>10</value>
>
> <description>This number is the maximum number of threads that
>
> should be allowed to access a queue at one time.
>
> </description>
>
> </property>
>
>
> <!-- plugin properties -->
>
>
> <property>
>
> <name>plugin.includes</name>
>
> <value>protocol-http|parse-(html|tika)|index-(basic|anchor)|urlnormalizer
> -(pass|regex|basic)</value>
>
> <!-- <value>protocol-http|urlfilter-regex|parse-(html|tika
> )|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
> -->
>
> <description>Regular expression naming plugin directory names to
>
> include. Any plugin not matching this expression is excluded.
>
> In any case you need at least include the nutch-extensionpoints plugin. By
>
> default Nutch includes crawling just HTML and plain text via HTTP,
>
> and basic indexing and search plugins. In order to use HTTPS please
> enable
>
> protocol-httpclient, but be aware of possible intermittent problems with
> the
>
> underlying commons-httpclient library.
>
> </description>
>
> </property>
>
>
> <!-- parser properties -->
>
>
> <property>
>
> <name>parser.character.encoding.default</name>
>
> <value>utf-8</value>
>
> <description>The character encoding to fall back to when no other
> information
>
> is available</description>
>
> </property>
>
>
> <property>
>
> <name>parser.timeout</name>
>
> <value>-1</value>
>
> <description>Timeout in seconds for the parsing of a document, otherwise
> treats it as an exception and
>
> moves on the the following documents. This parameter is applied to any
> Parser implementation.
>
> Set to -1 to deactivate, bearing in mind that this could cause
>
> the parsing to crash because of a very long or corrupted document.
>
> </description>
>
> </property>
>
> </configuration>
>
> ----
>
> Help is greatly appreciated.
>
> Kind regards,
>
> Issam
>
Re: Nutch 1.9 Fetchers Hung
Posted by Vitaly Savicks <v....@gmail.com>.
Hi Issam,
I've had a pretty fast performance yesterday with 10-15 threads and 2
threads per queue. Also, I've set the server delay to 0, but with a
different property. Don't remember its name now. You can find it in fetcher
properties of the nutch-defaults.XML
Looking at your configuration it looks like it's not used though. You
should make sure nutch uses it instead of the defaults. Also, some
parameters from 1.7 are absent or may be different from 1.9. I'd scrap them
and set the new ones according to default file
Cheers,
Vitaly
On 28 Nov 2014 14:36, "Issam Maamria" <ma...@gmail.com> wrote:
> Hi all,
>
> I am running the crawl command with depth 2 using a seed file containing
> 120 urls (about 2000 documents). Halfway through, the following output is
> logged:
>
> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=51,
> fetchQueues.getQueueCount=24*
>
> And after a while:
>
> *Aborting with 50 hung threads.*
>
> I am trying exactly the same thing using 1.8, and it is *working fine*.
> Please note that I am not applying any customisations apart from the
> following nutch-site.xml:
>
> <?xml version="1.0"?>
>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
>
> <!-- Put site-specific property overrides in this file. -->
>
>
> <configuration>
>
> <property>
>
> <name>http.agent.name</name>
>
> <value>MyAgent</value>
>
> </property>
>
>
> <property>
>
> <name>http.robots.agents</name>
>
> <value> MyAgent,*</value>
>
> <description>The agent strings we'll look for in robots.txt files,
>
> comma-separated, in decreasing order of precedence. You should
>
> put the value of http.agent.name as the first agent name, and keep the
>
> default * at the end of the list. E.g.: BlurflDev,Blurfl,*
>
> </description>
>
> </property>
>
>
> <!-- HTTP properties -->
>
>
> <property>
>
> <name>http.redirect.max</name>
>
> <value>2</value>
>
> <description>The maximum number of redirects the fetcher will follow
> when
>
> trying to fetch a page. If set to negative or 0, fetcher won't
> immediately
>
> follow redirected URLs, instead it will record them for later fetching.
>
> </description>
>
> </property>
>
>
> <property>
>
> <name>http.content.limit</name>
>
> <value>-1</value>
>
> <description>The length limit for downloaded content using the http://
>
> protocol, in bytes. If this value is nonnegative (>=0), content longer
>
> than it will be truncated; otherwise, no truncation at all. Do not
>
> confuse this setting with the file.content.limit setting.
>
> </description>
>
> </property>
>
>
> <!-- web db properties -->
>
>
> <!-- fetcher properties -->
>
>
> <property>
>
> <name>fetcher.server.delay</name>
>
> <value>4.0</value>
>
> <description>The number of seconds the fetcher will delay between
>
> successive requests to the same server.</description>
>
> </property>
>
>
> <property>
>
> <name>fetcher.threads.fetch</name>
>
> <value>20</value>
>
> <description>The number of FetcherThreads the fetcher should use.
>
> This is also determines the maximum number of requests that are
>
> made at once (each FetcherThread handles one connection). The total
>
> number of threads running in distributed mode will be the number of
>
> fetcher threads * number of nodes as fetcher has one map task per node.
>
> </description>
>
> </property>
>
>
> <property>
>
> <name>fetcher.threads.per.queue</name>
>
> <value>10</value>
>
> <description>This number is the maximum number of threads that
>
> should be allowed to access a queue at one time.
>
> </description>
>
> </property>
>
>
> <!-- plugin properties -->
>
>
> <property>
>
> <name>plugin.includes</name>
>
> <value>protocol-http|parse-(html|tika)|index-(basic|anchor)|
> urlnormalizer-(pass|regex|basic)</value>
>
> <!-- <value>protocol-http|urlfilter-regex|parse-(html|tika
> )|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
> -->
>
> <description>Regular expression naming plugin directory names to
>
> include. Any plugin not matching this expression is excluded.
>
> In any case you need at least include the nutch-extensionpoints plugin.
> By
>
> default Nutch includes crawling just HTML and plain text via HTTP,
>
> and basic indexing and search plugins. In order to use HTTPS please
> enable
>
> protocol-httpclient, but be aware of possible intermittent problems
> with the
>
> underlying commons-httpclient library.
>
> </description>
>
> </property>
>
>
> <!-- parser properties -->
>
>
> <property>
>
> <name>parser.character.encoding.default</name>
>
> <value>utf-8</value>
>
> <description>The character encoding to fall back to when no other
> information
>
> is available</description>
>
> </property>
>
>
> <property>
>
> <name>parser.timeout</name>
>
> <value>-1</value>
>
> <description>Timeout in seconds for the parsing of a document,
> otherwise treats it as an exception and
>
> moves on the the following documents. This parameter is applied to any
> Parser implementation.
>
> Set to -1 to deactivate, bearing in mind that this could cause
>
> the parsing to crash because of a very long or corrupted document.
>
> </description>
>
> </property>
>
> </configuration>
>
> ----
>
> Help is greatly appreciated.
>
> Kind regards,
>
> Issam
>