Adaptive sync with the time of page change

[Pascal Dimassimo <th...@hotmail.com>]

Hi,

I'm using the AdaptiveFetchSchedule to crawl my sites. So far, it works as
expected. The pages are re-crawled at different interval depending if the
pages were modified or not since the last visit.

But I have trouble understanding the "db.fetch.schedule.adaptive.sync_delta"
setting. When that setting is set to false, the next fetch is always set to
fetchtime + interval. But when that setting is activated, the next fetch is
reftime + interval. The reftime is computed by subtracting a percentage of
the delta (fetch time - modified time) to the fetch time. 

Can someone explains me in which case this setting is useful?
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Adaptive-sync-with-the-time-of-page-change-tp870842p870842.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Adaptive sync with the time of page change

[Pascal Dimassimo <th...@hotmail.com>]

Hi,

I have written a blog post to summarize my understanding of the Adaptive
Fetch Schedule. 

http://pascaldimassimo.com/2010/06/11/how-to-re-crawl-with-nutch/

But, like I mention at the end of it, I have not fully understand the
usefulness of the db.fetch.schedule.adaptive.sync_delta setting. Anyone
knows more about it?

And feel free to provide me feedback about errors or imprecisions in that
blog post.
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Adaptive-sync-with-the-time-of-page-change-tp870842p897234.html
Sent from the Nutch - User mailing list archive at Nabble.com.