You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Eric Martin <er...@makethembite.com> on 2010/11/03 02:40:02 UTC
Logs Spin - Active Thread - Spin Waiting - Basic
Hi,
I am getting these logs and I have no idea what they mean. I have searched
google and found very little documentation on it. That doesn't mean it
doesn't exist just that I have a hard time finding it. I may have missed an
obvious discussion of this and I am sorry if I did. Can someone point me to
the documentation or an answer? I'm a law student. Thanks!
Apache/CentOS 5 e Dual Intel Processor 12GB Ram. Running Solr 1.4, Drupal
and Nutch 1.2
2010-11-02 20:32:36,155 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:37,158 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:37,371 INFO fetcher.Fetcher - fetching
http://www.docstoc.com/docs/22884575/Sense-and-Sensibility
2010-11-02 20:32:38,160 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=2499
2010-11-02 20:32:39,162 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:40,164 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:41,166 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:42,168 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:43,161 INFO fetcher.Fetcher - fetching
http://www.docstoc.com/docs/59126421/sp1.convertro.com
2010-11-02 20:32:43,170 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=49, fetchQueues.totalSize=2499
2010-11-02 20:32:44,172 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:45,174 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:46,175 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
2010-11-02 20:32:47,178 INFO fetcher.Fetcher - -activeThreads=50,
spinWaiting=50, fetchQueues.totalSize=2500
Eric
RE: Logs Spin - Active Thread - Spin Waiting - Basic
Posted by Eric Martin <er...@makethembite.com>.
Thank you. I have not used those settings live. I used defaults. :-) I'm a default guy, look at results, research and modify, then I use HELP. THEN, I commit. :-)
Ok, I will adjust the n settings in my file to 1. I emailed Sigram.com to begin talks for some vertical customization. I am grateful for your comments as I spent the last three weeks understanding the process so I can effectively communicate with an expert.
In RE: "Simply rerun it using the same locations."
I use Nutch/Solr/Drupal (just a law student!) so I am not sure how to get that done without losing my crawls. So off to google I go!
-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org]
Sent: Wednesday, November 03, 2010 9:25 AM
To: user@nutch.apache.org
Subject: Re: Logs Spin - Active Thread - Spin Waiting - Basic
On 2010-11-03 17:12, Eric Martin wrote:
> Thank you very much. I can understand what you wrote! My crawl has been running for days. Here are some new settings I would like to put into nutch-site.xml.
>
> 1.) Does anyone see them as rude?
See below.
> 2.) How can I stop the crawler without losing the crawled data (I'm running 1.2 and should be able to use kill-nutch but can't find out where to do that)
It's not possible in this version of Nutch.
> 3.) how do I restart the currently stopped crawler with the new nutch-site.xml settings?
Simply rerun it using the same locations.
> <name>http.threads.per.host</name>
> <value>50</value>
Uh, oh... I'm pretty sure people will be mad at you for this. It's
considered impolite to exceed value of 1.
Also, you can set the property generate.max.count.per.host to e.g. 500
to limit the number of urls in a fetchlist from any given host.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Logs Spin - Active Thread - Spin Waiting - Basic
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-11-03 17:12, Eric Martin wrote:
> Thank you very much. I can understand what you wrote! My crawl has been running for days. Here are some new settings I would like to put into nutch-site.xml.
>
> 1.) Does anyone see them as rude?
See below.
> 2.) How can I stop the crawler without losing the crawled data (I'm running 1.2 and should be able to use kill-nutch but can't find out where to do that)
It's not possible in this version of Nutch.
> 3.) how do I restart the currently stopped crawler with the new nutch-site.xml settings?
Simply rerun it using the same locations.
> <name>http.threads.per.host</name>
> <value>50</value>
Uh, oh... I'm pretty sure people will be mad at you for this. It's
considered impolite to exceed value of 1.
Also, you can set the property generate.max.count.per.host to e.g. 500
to limit the number of urls in a fetchlist from any given host.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Logs Spin - Active Thread - Spin Waiting - Basic
Posted by Julien Nioche <li...@gmail.com>.
You can also specify a non-default value for
<property>
> <name>fetcher.timelimit.mins</name>
> <value>-1</value>
> <description>This is the number of minutes allocated to the fetching.
> Once this value is reached, any remaining entry from the input URL list
> is skipped
> and all active queues are emptied. The default value of -1 deactivates
> the time limit.
> </description>
> </property>
>
which will prevent the fetching from taking too long, typically when a
single host is holding everything up, which I think was your case
HTH
Julien
On 3 November 2010 16:12, Eric Martin <er...@makethembite.com> wrote:
> Thank you very much. I can understand what you wrote! My crawl has been
> running for days. Here are some new settings I would like to put into
> nutch-site.xml.
>
> 1.) Does anyone see them as rude?
> 2.) How can I stop the crawler without losing the crawled data (I'm running
> 1.2 and should be able to use kill-nutch but can't find out where to do
> that)
> 3.) how do I restart the currently stopped crawler with the new
> nutch-site.xml settings?
>
> <name>fetcher.server.min.delay</name>
> <value>0.5</value>
>
> <name>fetcher.server.delay</name>
> <value>1.0</value>
>
> <name>fetcher.threads.per.host</name>
> <value>50</value>
>
> <name>fetcher.max.crawl.delay</name>
> <value>1</value>
>
> <name>http.threads.per.host</name>
> <value>50</value>
>
> <name>http.max.delays</name>
> <value>1</value>
>
> Target Sites:
>
> http://www.ecasebriefs.com/blog/law/
> http://www.lawnix.com/cases/cases-index/
> http://www.oyez.org/
> http://www.4lawnotes.com/
> http://www.docstoc.com/documents/education/law-school/case-briefs
> http://www.lawschoolcasebriefs.com/
> http://dictionary.findlaw.com
>
> -----Original Message-----
> From: Andrzej Bialecki [mailto:ab@getopt.org]
> Sent: Wednesday, November 03, 2010 2:55 AM
> To: user@nutch.apache.org
> Subject: Re: Logs Spin - Active Thread - Spin Waiting - Basic
>
> On 2010-11-03 02:40, Eric Martin wrote:
> > Hi,
> >
> >
> >
> > I am getting these logs and I have no idea what they mean. I have
> searched
> > google and found very little documentation on it. That doesn't mean it
> > doesn't exist just that I have a hard time finding it. I may have missed
> an
> > obvious discussion of this and I am sorry if I did. Can someone point me
> to
> > the documentation or an answer? I'm a law student. Thanks!
>
> Given the composition of your fetch list (all remaining URLs in the
> queue are from the same host) what you see is perfectly normal. There
> are 50 fetching threads that can fetch items from any host. However, all
> remaining items are from the same single host. Due to the politeness
> limits Nutch won't make more than one connection to the host, and it
> will space its requests N seconds apart - otherwise a multi-threaded
> distributed crawler could easily overwhelm the target host.
>
> So the logs indicate that only one thread is fetching at any given time,
> there are at least 2500 items in the queue, and every N seconds the
> thread is allowed to fetch an item. All other threads are spinning idle.
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
Re: Logs Spin - Active Thread - Spin Waiting - Basic
Posted by Dogacan Guney <do...@gmail.com>.
Hi,
On Nov 3, 2010, at 9:12 AM, Eric Martin wrote:
> Thank you very much. I can understand what you wrote! My crawl has been running for days. Here are some new settings I would like to put into nutch-site.xml.
>
> 1.) Does anyone see them as rude?
> 2.) How can I stop the crawler without losing the crawled data (I'm running 1.2 and should be able to use kill-nutch but can't find out where to do that)
> 3.) how do I restart the currently stopped crawler with the new nutch-site.xml settings?
>
> <name>fetcher.server.min.delay</name>
> <value>0.5</value>
>
> <name>fetcher.server.delay</name>
> <value>1.0</value>
>
> <name>fetcher.threads.per.host</name>
> <value>50</value>
If fetcher.threads.per.host > 1, then a fetcher will make more than 1 active request to servers. I don't know
about the sites below but that is generally frowned upon.
>
> <name>fetcher.max.crawl.delay</name>
> <value>1</value>
>
> <name>http.threads.per.host</name>
> <value>50</value>
>
> <name>http.max.delays</name>
> <value>1</value>
>
> Target Sites:
>
> http://www.ecasebriefs.com/blog/law/
> http://www.lawnix.com/cases/cases-index/
> http://www.oyez.org/
> http://www.4lawnotes.com/
> http://www.docstoc.com/documents/education/law-school/case-briefs
> http://www.lawschoolcasebriefs.com/
> http://dictionary.findlaw.com
>
> -----Original Message-----
> From: Andrzej Bialecki [mailto:ab@getopt.org]
> Sent: Wednesday, November 03, 2010 2:55 AM
> To: user@nutch.apache.org
> Subject: Re: Logs Spin - Active Thread - Spin Waiting - Basic
>
> On 2010-11-03 02:40, Eric Martin wrote:
>> Hi,
>>
>>
>>
>> I am getting these logs and I have no idea what they mean. I have searched
>> google and found very little documentation on it. That doesn't mean it
>> doesn't exist just that I have a hard time finding it. I may have missed an
>> obvious discussion of this and I am sorry if I did. Can someone point me to
>> the documentation or an answer? I'm a law student. Thanks!
>
> Given the composition of your fetch list (all remaining URLs in the
> queue are from the same host) what you see is perfectly normal. There
> are 50 fetching threads that can fetch items from any host. However, all
> remaining items are from the same single host. Due to the politeness
> limits Nutch won't make more than one connection to the host, and it
> will space its requests N seconds apart - otherwise a multi-threaded
> distributed crawler could easily overwhelm the target host.
>
> So the logs indicate that only one thread is fetching at any given time,
> there are at least 2500 items in the queue, and every N seconds the
> thread is allowed to fetch an item. All other threads are spinning idle.
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
RE: Logs Spin - Active Thread - Spin Waiting - Basic
Posted by Eric Martin <er...@makethembite.com>.
Thank you very much. I can understand what you wrote! My crawl has been running for days. Here are some new settings I would like to put into nutch-site.xml.
1.) Does anyone see them as rude?
2.) How can I stop the crawler without losing the crawled data (I'm running 1.2 and should be able to use kill-nutch but can't find out where to do that)
3.) how do I restart the currently stopped crawler with the new nutch-site.xml settings?
<name>fetcher.server.min.delay</name>
<value>0.5</value>
<name>fetcher.server.delay</name>
<value>1.0</value>
<name>fetcher.threads.per.host</name>
<value>50</value>
<name>fetcher.max.crawl.delay</name>
<value>1</value>
<name>http.threads.per.host</name>
<value>50</value>
<name>http.max.delays</name>
<value>1</value>
Target Sites:
http://www.ecasebriefs.com/blog/law/
http://www.lawnix.com/cases/cases-index/
http://www.oyez.org/
http://www.4lawnotes.com/
http://www.docstoc.com/documents/education/law-school/case-briefs
http://www.lawschoolcasebriefs.com/
http://dictionary.findlaw.com
-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org]
Sent: Wednesday, November 03, 2010 2:55 AM
To: user@nutch.apache.org
Subject: Re: Logs Spin - Active Thread - Spin Waiting - Basic
On 2010-11-03 02:40, Eric Martin wrote:
> Hi,
>
>
>
> I am getting these logs and I have no idea what they mean. I have searched
> google and found very little documentation on it. That doesn't mean it
> doesn't exist just that I have a hard time finding it. I may have missed an
> obvious discussion of this and I am sorry if I did. Can someone point me to
> the documentation or an answer? I'm a law student. Thanks!
Given the composition of your fetch list (all remaining URLs in the
queue are from the same host) what you see is perfectly normal. There
are 50 fetching threads that can fetch items from any host. However, all
remaining items are from the same single host. Due to the politeness
limits Nutch won't make more than one connection to the host, and it
will space its requests N seconds apart - otherwise a multi-threaded
distributed crawler could easily overwhelm the target host.
So the logs indicate that only one thread is fetching at any given time,
there are at least 2500 items in the queue, and every N seconds the
thread is allowed to fetch an item. All other threads are spinning idle.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Logs Spin - Active Thread - Spin Waiting - Basic
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-11-03 02:40, Eric Martin wrote:
> Hi,
>
>
>
> I am getting these logs and I have no idea what they mean. I have searched
> google and found very little documentation on it. That doesn't mean it
> doesn't exist just that I have a hard time finding it. I may have missed an
> obvious discussion of this and I am sorry if I did. Can someone point me to
> the documentation or an answer? I'm a law student. Thanks!
Given the composition of your fetch list (all remaining URLs in the
queue are from the same host) what you see is perfectly normal. There
are 50 fetching threads that can fetch items from any host. However, all
remaining items are from the same single host. Due to the politeness
limits Nutch won't make more than one connection to the host, and it
will space its requests N seconds apart - otherwise a multi-threaded
distributed crawler could easily overwhelm the target host.
So the logs indicate that only one thread is fetching at any given time,
there are at least 2500 items in the queue, and every N seconds the
thread is allowed to fetch an item. All other threads are spinning idle.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com