You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by axi <ax...@gmail.com> on 2010/01/20 16:16:15 UTC
Nutch 1.0 slow crawls
Hi to all,
I'm a novice user of Nutch, I have it on a debian machine, and I have probe
the latest release of nutch 1.0 with very slow results in crawling, I have a
10 Megabytes/s connection and it only crawls at 300 Kb/s with peaks of 1
Mb/s. I tweaked everything, dns, linux tcp settings, thread numbers, java
conf etc.. but anything have effect. There are a lot of spin waiting threads
there, only 10-20 of them working and I have injected 1M different hosts, so
politeness is not the problem. I swithched back to 0.9 nutch, and then it
works like a charm at good speeds 5-6 Mb/s with the bottleneck on machine
cpu.
¿Are this issues solved on dev version of nutch or why is this happens?
Thanks in advance,
--
View this message in context: http://old.nabble.com/Nutch-1.0-slow-crawls-tp27243302p27243302.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 1.0 slow crawls
Posted by MilleBii <mi...@gmail.com>.
Well politeness can still be the problem... If for instance your are
crawling blogs like wordpress or blogspot, they are all different url
but with same IP so the fetcher will wait.
2010/1/20, axi <ax...@gmail.com>:
>
> Hi to all,
> I'm a novice user of Nutch, I have it on a debian machine, and I have probe
> the latest release of nutch 1.0 with very slow results in crawling, I have a
> 10 Megabytes/s connection and it only crawls at 300 Kb/s with peaks of 1
> Mb/s. I tweaked everything, dns, linux tcp settings, thread numbers, java
> conf etc.. but anything have effect. There are a lot of spin waiting threads
> there, only 10-20 of them working and I have injected 1M different hosts, so
> politeness is not the problem. I swithched back to 0.9 nutch, and then it
> works like a charm at good speeds 5-6 Mb/s with the bottleneck on machine
> cpu.
>
> ¿Are this issues solved on dev version of nutch or why is this happens?
>
> Thanks in advance,
>
> --
> View this message in context:
> http://old.nabble.com/Nutch-1.0-slow-crawls-tp27243302p27243302.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>
--
-MilleBii-
Re: Nutch 1.0 slow crawls
Posted by Julien Nioche <li...@gmail.com>.
Hi,
See https://issues.apache.org/jira/browse/NUTCH-721 for an
explanation. This has been fixed in the SVN version.
Julien
--
DigitalPebble Ltd
http://www.digitalpebble.com
2010/1/20 axi <ax...@gmail.com>:
>
> Hi to all,
> I'm a novice user of Nutch, I have it on a debian machine, and I have probe
> the latest release of nutch 1.0 with very slow results in crawling, I have a
> 10 Megabytes/s connection and it only crawls at 300 Kb/s with peaks of 1
> Mb/s. I tweaked everything, dns, linux tcp settings, thread numbers, java
> conf etc.. but anything have effect. There are a lot of spin waiting threads
> there, only 10-20 of them working and I have injected 1M different hosts, so
> politeness is not the problem. I swithched back to 0.9 nutch, and then it
> works like a charm at good speeds 5-6 Mb/s with the bottleneck on machine
> cpu.
>
> ¿Are this issues solved on dev version of nutch or why is this happens?
>
> Thanks in advance,
>
> --
> View this message in context: http://old.nabble.com/Nutch-1.0-slow-crawls-tp27243302p27243302.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>