You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by atawfik <co...@gmail.com> on 2014/08/30 20:10:47 UTC

Re: Nutch re-crawl step

Nutch usually re-fetches  links if they match the default interval time. So,
in your case, even if the page is not modified, it is a half day old.
Therefore, Nutch re-fetches it.

The adaptive fetcher is useful for re-fetching pages before meeting the
default interval time.

Regards
Ameer



--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-re-crawl-step-tp4154940p4155991.html
Sent from the Nutch - User mailing list archive at Nabble.com.