You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Matt Kangas <ka...@gmail.com> on 2005/12/01 21:22:06 UTC

Re: [Nutch-dev] incremental crawling

#2 should be a pluggable/hookable parameter. "high-scoring" sounds  
like a reasonable default basis for choosing recrawl intervals, but  
I'm sure that nearly everyone will think of a way to improve upon  
that for their particular system.

e.g. "high-scoring" ain't gonna cut it for my needs. (0.5 wink ;)

--matt

On Dec 1, 2005, at 2:15 PM, Doug Cutting wrote:

> It would be good to improve the support for incremental crawling  
> added to Nutch.  Here are some ideas about how we might implement  
> it.  Andrzej has posted in the past about this, so he probably has  
> better ideas.
>
> Incremental crawling could proceed as follows:
>
> 1. Bootstrap with a batch crawl, using the 'crawl' command.  Modify  
> CrawlDatum to store the MD5Hash of the content of fetched urls.
>
> 2. Reduce the fetch interval for high-scoring urls.  If the default  
> is monthly, then the top-scoring 1% of urls might be set to daily,  
> and the top-scoring 10% of urls might be set to weekly.
>
> 3. Generate a fetch list & fetch it.  When the url has been  
> previously fetched, and its content is unchanged, increase its  
> fetch interval by an amount, e.g., 50%.  If the content is changed,  
> decrease the fetch interval.  The percentage of increase and  
> decrease might be influenced by the url's score.
>
> 4. Update the crawl db & link db, index the new segment, dedup,  
> etc. When updating the crawl db, scores for existing urls should  
> not change, since the scoring method we're using (OPIC) assumes  
> each page is fetched only once.
>
> Steps 3 & 4 can be packaged as an 'update' command.  Step 2 can be  
> included in the 'crawl' command, so that crawled indexes are always  
> ready for update.
>
> Comments?
>
> Doug

--
Matt Kangas / kangas@gmail.com



Re: [Nutch-dev] incremental crawling

Posted by Jérôme Charron <je...@gmail.com>.
Sounds really good (and it is requested by a lot of nutch users!).
+1

Jérôme

On 12/1/05, Doug Cutting <cu...@nutch.org> wrote:
>
> Matt Kangas wrote:
> > #2 should be a pluggable/hookable parameter. "high-scoring" sounds  like
> > a reasonable default basis for choosing recrawl intervals, but  I'm sure
> > that nearly everyone will think of a way to improve upon  that for their
> > particular system.
> >
> > e.g. "high-scoring" ain't gonna cut it for my needs. (0.5 wink ;)
>
> In NUTCH-61, Andrzej has a pluggable FetchSchedule.  That looks like a
> good idea.
>
> http://issues.apache.org/jira/browse/NUTCH-61
>
> Doug
>



--
http://motrech.free.fr/
http://www.frutch.org/

Re: [Nutch-dev] incremental crawling

Posted by Doug Cutting <cu...@nutch.org>.
Matt Kangas wrote:
> #2 should be a pluggable/hookable parameter. "high-scoring" sounds  like 
> a reasonable default basis for choosing recrawl intervals, but  I'm sure 
> that nearly everyone will think of a way to improve upon  that for their 
> particular system.
> 
> e.g. "high-scoring" ain't gonna cut it for my needs. (0.5 wink ;)

In NUTCH-61, Andrzej has a pluggable FetchSchedule.  That looks like a 
good idea.

http://issues.apache.org/jira/browse/NUTCH-61

Doug