[jira] [Updated] (NUTCH-2456) Allow to index pages/URLs not contained in CrawlDb

["Yossi Tamari (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yossi Tamari updated NUTCH-2456:
--------------------------------
    Summary: Allow to index pages/URLs not contained in CrawlDb  (was: Redirected documents are not indexed)

> Allow to index pages/URLs not contained in CrawlDb
> --------------------------------------------------
>
>                 Key: NUTCH-2456
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2456
>             Project: Nutch
>          Issue Type: Bug
>          Components: indexer
>    Affects Versions: 1.13
>            Reporter: Yossi Tamari
>            Priority: Critical
>
> If http.redirect.max is set to a positive value, the Fetcher will follow redirects, creating a new CrawlDatum.
> If the redirected URL is fetched and parsed, during indexing for it we have a special case: dbDatum is null. This means that in [https://github.com/apache/nutch/blob/6199492f5e1e8811022257c88dbf63f1e1c739d0/src/java/org/apache/nutch/indexer/IndexerMapReduce.java#L259] the document is not indexed, as it is assumed it only has inlinks (actually it has everything but dbDatum).
> I'm not sure what the correct fix is here. It seems to me the condition should use AND instead of OR anyway, but I may not understand the original intent. It is clear that it is too strict as is.
> However, the code following that line assumes all 4 objects are not null, so a patch would need to change more than just the condition.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)