You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by "R.Mayoran" <ma...@team-lab.com> on 2006/01/11 10:01:36 UTC

About Fetching

Hello,

I want to know what algorithm is used in Fetching and how Nutch creates the
Fetchlist?

For example, if a Fetchlist has 1000urls of a single host, then while
crawling the host will be accessed continuously and it might cause some
trouble to the host. So I want to know how to avoid this kind of problem and
if is it possible I want to make the Fetchlists which have entirely
different hosts.

Thank you.