You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Kai_testing Middleton <ka...@yahoo.com> on 2007/06/30 04:10:29 UTC
Interrupting a nutch crawl -- or use topN?
I am running a nutch crawl of 19 sites. I wish to let this crawl go for about two days then gracefully stop it (I don't expect it to complete by then). Is there a way to do this? I want it to stop crawling then build the lucene index. Note that I used a simple nutch crawl command, rather than the "whole web" crawling methodology:
nutch crawl urls.txt -dir /usr/tmp/19sites -depth 10
Or is it better to use the -topN option?
Some documentation for topN:
http://www.mail-archive.com/nutch-user@lucene.apache.org/msg03916.html
"You can limit the number of pages by using the -topN parameter. This
limits the number of pages fetched in each round. Pages are prioritized
by how well-linked they are. The maximum number of pages that can be
fetched is topN*depth."
Or from the tutorial:
-topN N determines the maximum number of pages that
will be retrieved at each level up to the depth.
For example, a typical call might be:
bin/nutch crawl urls -dir crawl -depth 3 -topN 50
Typically one starts testing one's configuration by crawling at
shallow depths, sharply limiting the number of pages fetched at each
level (-topN), and watching the output to check that
desired pages are fetched and undesirable pages are not. Once one is
confident of the configuration, then an appropriate depth for a full
crawl is around 10. The number of pages per level
(-topN) for a full crawl can be from tens of thousands to
millions, depending on your resources.
____________________________________________________________________________________
Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469
RE: Interrupting a nutch crawl -- or use topN?
Posted by Gal Nitzan <ga...@gmail.com>.
Hi,
In the past Andrzej put some stuff related to your issue in the Jira. Try to
look it up there.
Found it :) http://issues.apache.org/jira/browse/NUTCH-368
HTH,
Gal
> -----Original Message-----
> From: Ian Holsman [mailto:lists@holsman.net]
> Sent: Sunday, July 01, 2007 2:38 AM
> To: nutch-user@lucene.apache.org
> Subject: Re: Interrupting a nutch crawl -- or use topN?
>
> Kai_testing Middleton wrote:
> > I am running a nutch crawl of 19 sites. I wish to let this crawl go for
> about two days then gracefully stop it (I don't expect it to complete by
> then). Is there a way to do this? I want it to stop crawling then build
> the lucene index. Note that I used a simple nutch crawl command, rather
> than the "whole web" crawling methodology:
> >
> > nutch crawl urls.txt -dir /usr/tmp/19sites -depth 10
> >
> I use a iterative approach using a script similar to what Sami blogs
> about here:
> http://blog.foofactory.fi/2007/02/online-indexing-integrating-nutch-
> with.html
>
> I then issue a crawl of 10,000 URLs at a time, and just repeat the
> process for as long as the window available. because I use solr to store
> the crawl results
> It makes the index available during the crawl window.
>
> but I'm a relative newbie as well, so look forward what the experts say.
>
>
> regards
> Ian
Re: Interrupting a nutch crawl -- or use topN?
Posted by Ian Holsman <li...@holsman.net>.
Kai_testing Middleton wrote:
> I am running a nutch crawl of 19 sites. I wish to let this crawl go for about two days then gracefully stop it (I don't expect it to complete by then). Is there a way to do this? I want it to stop crawling then build the lucene index. Note that I used a simple nutch crawl command, rather than the "whole web" crawling methodology:
>
> nutch crawl urls.txt -dir /usr/tmp/19sites -depth 10
>
I use a iterative approach using a script similar to what Sami blogs
about here:
http://blog.foofactory.fi/2007/02/online-indexing-integrating-nutch-with.html
I then issue a crawl of 10,000 URLs at a time, and just repeat the
process for as long as the window available. because I use solr to store
the crawl results
It makes the index available during the crawl window.
but I'm a relative newbie as well, so look forward what the experts say.
regards
Ian