You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Kai_testing Middleton <ka...@yahoo.com> on 2007/06/30 04:10:29 UTC

Interrupting a nutch crawl -- or use topN?

I am running a nutch crawl of 19 sites.  I wish to let this crawl go for about two days then gracefully stop it (I don't expect it to complete by then).  Is there a way to do this?  I want it to stop crawling then build the lucene index.  Note that I used a simple nutch crawl command, rather than the "whole web" crawling methodology:

nutch crawl urls.txt -dir /usr/tmp/19sites -depth 10

Or is it better to use the -topN option?

Some documentation for topN:

http://www.mail-archive.com/nutch-user@lucene.apache.org/msg03916.html

"You can limit the number of pages by using the -topN parameter.  This 
limits the number of pages fetched in each round.  Pages are prioritized 
by how well-linked they are.  The maximum number of pages that can be 
fetched is topN*depth."

Or from the tutorial:


-topN N determines the maximum number of pages that
will be retrieved at each level up to the depth.
For example, a typical call might be:

bin/nutch crawl urls -dir crawl -depth 3 -topN 50

Typically one starts testing one's configuration by crawling at
shallow depths, sharply limiting the number of pages fetched at each
level (-topN), and watching the output to check that
desired pages are fetched and undesirable pages are not.  Once one is
confident of the configuration, then an appropriate depth for a full
crawl is around 10.  The number of pages per level
(-topN) for a full crawl can be from tens of thousands to
millions, depending on your resources.





       
____________________________________________________________________________________
Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469

RE: Interrupting a nutch crawl -- or use topN?

Posted by Gal Nitzan <ga...@gmail.com>.
Hi,

In the past Andrzej put some stuff related to your issue in the Jira. Try to 
look it up there.

Found it :) http://issues.apache.org/jira/browse/NUTCH-368

HTH,

Gal

> -----Original Message-----
> From: Ian Holsman [mailto:lists@holsman.net]
> Sent: Sunday, July 01, 2007 2:38 AM
> To: nutch-user@lucene.apache.org
> Subject: Re: Interrupting a nutch crawl -- or use topN?
>
> Kai_testing Middleton wrote:
> > I am running a nutch crawl of 19 sites.  I wish to let this crawl go for
> about two days then gracefully stop it (I don't expect it to complete by
> then).  Is there a way to do this?  I want it to stop crawling then build
> the lucene index.  Note that I used a simple nutch crawl command, rather
> than the "whole web" crawling methodology:
> >
> > nutch crawl urls.txt -dir /usr/tmp/19sites -depth 10
> >
> I use a iterative approach using a script similar to what Sami blogs
> about here:
> http://blog.foofactory.fi/2007/02/online-indexing-integrating-nutch-
> with.html
>
> I then issue a crawl of 10,000 URLs at a time, and just repeat the
> process for as long as the window available. because I use solr to store
> the crawl results
> It makes the index available during the crawl window.
>
> but I'm a relative newbie as well, so look forward what the experts say.
>
>
> regards
> Ian



Re: Interrupting a nutch crawl -- or use topN?

Posted by Ian Holsman <li...@holsman.net>.
Kai_testing Middleton wrote:
> I am running a nutch crawl of 19 sites.  I wish to let this crawl go for about two days then gracefully stop it (I don't expect it to complete by then).  Is there a way to do this?  I want it to stop crawling then build the lucene index.  Note that I used a simple nutch crawl command, rather than the "whole web" crawling methodology:
>
> nutch crawl urls.txt -dir /usr/tmp/19sites -depth 10
>   
I use a iterative approach using a script similar to what Sami blogs 
about here: 
http://blog.foofactory.fi/2007/02/online-indexing-integrating-nutch-with.html

I then issue a crawl of 10,000 URLs at a time, and just repeat the 
process for as long as the window available. because I use solr to store 
the crawl results
It makes the index available during the crawl window.

but I'm a relative newbie as well, so look forward what the experts say.


regards
Ian