You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Gaurav Bagga <gb...@gmail.com> on 2011/08/31 19:42:48 UTC
Redirect Handling
I couldn't find much documentation regarding what happens with redirects.
There are two types of redirects.
Temp Redirect and Perm Redirect
What happens in each of the cases.
Does the fetcher fetches the page in the same crawl cycle or leaves it to be
fetched later?
Eg. If xyz.com is redirected to abc.com
What happens to the original URL's. Are they kept or changed to the
redirected URL's.
Or both are added to DB.
And is the page fetched during the same fetching cycle or is left to be
fetched in next cycle.
It would be great if somebody can give a detailed explanation for redirect
handling or redirect me to right documentation.
Regards
Gaurav