[jira] Resolved: (NUTCH-962) max. redirects not handled correctly: fetcher stops at max-1 redirects

["Andrzej Bialecki (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki  resolved NUTCH-962.
-------------------------------------

       Resolution: Fixed
    Fix Version/s: 2.0
                   1.3
         Assignee: Andrzej Bialecki 

> max. redirects not handled correctly: fetcher stops at max-1 redirects
> ----------------------------------------------------------------------
>
>                 Key: NUTCH-962
>                 URL: https://issues.apache.org/jira/browse/NUTCH-962
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.2, 1.3, 2.0
>            Reporter: Sebastian Nagel
>            Assignee: Andrzej Bialecki 
>             Fix For: 1.3, 2.0
>
>         Attachments: Fetcher_redir.patch
>
>
> The fetcher stops following redirects one redirect before the max. redirects is reached.
> The description of http.redirect.max
> > The maximum number of redirects the fetcher will follow when
> > trying to fetch a page. If set to negative or 0, fetcher won't immediately
> > follow redirected URLs, instead it will record them for later fetching.
> suggests that if set to 1 that one redirect will be followed.
> I tried to crawl two documents the first redirecting by
>  <meta http-equiv="refresh" content="0; URL=./to/meta_refresh_target.html">
> to the second with http.redirect.max = 1
> The second document is not fetched and the URL has state GONE in CrawlDb.
> fetching file:/test/redirects/meta_refresh.html
> redirectCount=0
> -finishing thread FetcherThread, activeThreads=1
>  - content redirect to file:/test/redirects/to/meta_refresh_target.html (fetching now)
>  - redirect count exceeded file:/test/redirects/to/meta_refresh_target.html
> The attached patch would fix this: if http.redirect.max is 1 : one redirect is followed.
> Of course, this would mean there is no possibility to skip redirects at all since 0
> (as well as negative values) means "treat redirects as ordinary links".

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira