You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Yossi Tamari (JIRA)" <ji...@apache.org> on 2017/11/06 18:09:00 UTC
[jira] [Updated] (NUTCH-2456) Redirected documents are not indexed
[ https://issues.apache.org/jira/browse/NUTCH-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yossi Tamari updated NUTCH-2456:
--------------------------------
Description:
If http.redirect.max is set to a positive value, the Fetcher will follow redirects, creating a new CrawlDatum.
If the redirected URL is fetched and parsed, during indexing for it we have a special case: dbDatum is null. This means that in [https://github.com/apache/nutch/blob/6199492f5e1e8811022257c88dbf63f1e1c739d0/src/java/org/apache/nutch/indexer/IndexerMapReduce.java#L259] the document is not indexed, as it is assumed it only has inlinks (actually it has everything but dbDatum).
I'm not sure what the correct fix is here. It seems to me the condition should use AND instead of OR anyway, but I may not understand the original intent. It is clear that it is too strict as is.
However, the code following that line assumes all 4 objects are not null, so a patch would need to change more than just the condition.
was:
If http.redirect.max is set to a positive value, the Fetcher will follow redirects, creating a new CrawlDatum.
If the redirected URL is fetched and parsed, during indexing for it we have a special case: dbDatum is null. This means that in [https://github.com/apache/nutch/blob/6199492f5e1e8811022257c88dbf63f1e1c739d0/src/java/org/apache/nutch/indexer/IndexerMapReduce.java#L259] the document is not indexed, as it is assumed it only has inlinks (actually it has everything but dbDatum).
I'm not sure what the correct fix is here. It seems to me the condition should use AND instead of OR anyway, but I may not understand the original intent. It is clear that it is too strict as is.
> Redirected documents are not indexed
> ------------------------------------
>
> Key: NUTCH-2456
> URL: https://issues.apache.org/jira/browse/NUTCH-2456
> Project: Nutch
> Issue Type: Bug
> Components: indexer
> Affects Versions: 1.13
> Reporter: Yossi Tamari
> Priority: Critical
>
> If http.redirect.max is set to a positive value, the Fetcher will follow redirects, creating a new CrawlDatum.
> If the redirected URL is fetched and parsed, during indexing for it we have a special case: dbDatum is null. This means that in [https://github.com/apache/nutch/blob/6199492f5e1e8811022257c88dbf63f1e1c739d0/src/java/org/apache/nutch/indexer/IndexerMapReduce.java#L259] the document is not indexed, as it is assumed it only has inlinks (actually it has everything but dbDatum).
> I'm not sure what the correct fix is here. It seems to me the condition should use AND instead of OR anyway, but I may not understand the original intent. It is clear that it is too strict as is.
> However, the code following that line assumes all 4 objects are not null, so a patch would need to change more than just the condition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)