You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2011/06/25 14:54:49 UTC
[jira] [Closed] (NUTCH-962) max. redirects not handled correctly:
fetcher stops at max-1 redirects
[ https://issues.apache.org/jira/browse/NUTCH-962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma closed NUTCH-962.
-------------------------------
Bulk close of resolved issues for 1.3.
> max. redirects not handled correctly: fetcher stops at max-1 redirects
> ----------------------------------------------------------------------
>
> Key: NUTCH-962
> URL: https://issues.apache.org/jira/browse/NUTCH-962
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.2, 1.3, 2.0
> Reporter: Sebastian Nagel
> Assignee: Andrzej Bialecki
> Fix For: 1.3, 2.0
>
> Attachments: Fetcher_redir.patch
>
>
> The fetcher stops following redirects one redirect before the max. redirects is reached.
> The description of http.redirect.max
> > The maximum number of redirects the fetcher will follow when
> > trying to fetch a page. If set to negative or 0, fetcher won't immediately
> > follow redirected URLs, instead it will record them for later fetching.
> suggests that if set to 1 that one redirect will be followed.
> I tried to crawl two documents the first redirecting by
> <meta http-equiv="refresh" content="0; URL=./to/meta_refresh_target.html">
> to the second with http.redirect.max = 1
> The second document is not fetched and the URL has state GONE in CrawlDb.
> fetching file:/test/redirects/meta_refresh.html
> redirectCount=0
> -finishing thread FetcherThread, activeThreads=1
> - content redirect to file:/test/redirects/to/meta_refresh_target.html (fetching now)
> - redirect count exceeded file:/test/redirects/to/meta_refresh_target.html
> The attached patch would fix this: if http.redirect.max is 1 : one redirect is followed.
> Of course, this would mean there is no possibility to skip redirects at all since 0
> (as well as negative values) means "treat redirects as ordinary links".
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira