You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Matt Kangas <ka...@gmail.com> on 2005/12/01 21:22:06 UTC
Re: [Nutch-dev] incremental crawling
#2 should be a pluggable/hookable parameter. "high-scoring" sounds
like a reasonable default basis for choosing recrawl intervals, but
I'm sure that nearly everyone will think of a way to improve upon
that for their particular system.
e.g. "high-scoring" ain't gonna cut it for my needs. (0.5 wink ;)
--matt
On Dec 1, 2005, at 2:15 PM, Doug Cutting wrote:
> It would be good to improve the support for incremental crawling
> added to Nutch. Here are some ideas about how we might implement
> it. Andrzej has posted in the past about this, so he probably has
> better ideas.
>
> Incremental crawling could proceed as follows:
>
> 1. Bootstrap with a batch crawl, using the 'crawl' command. Modify
> CrawlDatum to store the MD5Hash of the content of fetched urls.
>
> 2. Reduce the fetch interval for high-scoring urls. If the default
> is monthly, then the top-scoring 1% of urls might be set to daily,
> and the top-scoring 10% of urls might be set to weekly.
>
> 3. Generate a fetch list & fetch it. When the url has been
> previously fetched, and its content is unchanged, increase its
> fetch interval by an amount, e.g., 50%. If the content is changed,
> decrease the fetch interval. The percentage of increase and
> decrease might be influenced by the url's score.
>
> 4. Update the crawl db & link db, index the new segment, dedup,
> etc. When updating the crawl db, scores for existing urls should
> not change, since the scoring method we're using (OPIC) assumes
> each page is fetched only once.
>
> Steps 3 & 4 can be packaged as an 'update' command. Step 2 can be
> included in the 'crawl' command, so that crawled indexes are always
> ready for update.
>
> Comments?
>
> Doug
--
Matt Kangas / kangas@gmail.com
Re: [Nutch-dev] incremental crawling
Posted by Jérôme Charron <je...@gmail.com>.
Sounds really good (and it is requested by a lot of nutch users!).
+1
Jérôme
On 12/1/05, Doug Cutting <cu...@nutch.org> wrote:
>
> Matt Kangas wrote:
> > #2 should be a pluggable/hookable parameter. "high-scoring" sounds like
> > a reasonable default basis for choosing recrawl intervals, but I'm sure
> > that nearly everyone will think of a way to improve upon that for their
> > particular system.
> >
> > e.g. "high-scoring" ain't gonna cut it for my needs. (0.5 wink ;)
>
> In NUTCH-61, Andrzej has a pluggable FetchSchedule. That looks like a
> good idea.
>
> http://issues.apache.org/jira/browse/NUTCH-61
>
> Doug
>
--
http://motrech.free.fr/
http://www.frutch.org/
Re: [Nutch-dev] incremental crawling
Posted by Doug Cutting <cu...@nutch.org>.
Matt Kangas wrote:
> #2 should be a pluggable/hookable parameter. "high-scoring" sounds like
> a reasonable default basis for choosing recrawl intervals, but I'm sure
> that nearly everyone will think of a way to improve upon that for their
> particular system.
>
> e.g. "high-scoring" ain't gonna cut it for my needs. (0.5 wink ;)
In NUTCH-61, Andrzej has a pluggable FetchSchedule. That looks like a
good idea.
http://issues.apache.org/jira/browse/NUTCH-61
Doug