[jira] [Commented] (NUTCH-1422) reset signature for redirects

["Markus Jelsma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033850#comment-14033850 ] 

Markus Jelsma commented on NUTCH-1422:
--------------------------------------

Yes, agreed. Although right now we don't have a feature that can change a fetcher's status to being a redirect at the parsing stage. We are going to build that at some point anyway because we want to detect HTML meta and Javascript redirects, which already kind of works. 

This then should be implemented when updating the DB.

> reset signature for redirects
> -----------------------------
>
>                 Key: NUTCH-1422
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1422
>             Project: Nutch
>          Issue Type: Bug
>          Components: crawldb, fetcher
>    Affects Versions: 1.4
>            Reporter: Sebastian Nagel
>            Priority: Critical
>             Fix For: 1.9
>
>         Attachments: NUTCH-1422_redir_notmodified_log.txt
>
>
> In a long running continuous crawl with Nutch 1.4 URLs with a HTTP redirect (http.redirect.max = 0) are kept as not-modified in the CrawlDb. Short protocol (cf. attached dumped segment / CrawlDb data):
>  2012-02-23 :  injected
>  2012-02-24 :  fetched
>  2012-03-30 :  re-fetched, signature changed
>  2012-04-20 :  re-fetched, redirected
>  2012-04-24 :  in CrawlDb as db_notmodified, still indexed with old content!
> The signature of a previously fetched document is not reset when the URL/doc is changed to a redirect at a later time. CrawlDbReducer.reduce then sets the status to db_notmodified because the new signature in with fetch status is identical to the old one.
> Possible fixes (??):
> * reset the signature in Fetcher
> * handle this case in CrawlDbReducer.reduce



--
This message was sent by Atlassian JIRA
(v6.2#6252)