[jira] [Commented] (NUTCH-2275) MD5Signature by default doesn't take in account parse

["Sebastian Nagel (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980217#comment-16980217 ] 

Sebastian Nagel commented on NUTCH-2275:
----------------------------------------

The problem is that the "feed" plugin emits one document for each feed item without the need to actually fetch the URL/page of the item. There should be no signature at all for yet unfetched URLs in the CrawlDb. Everything done on the signature - deduplication, not-modified recognition - does not make sense.

> MD5Signature by default doesn't take in account parse
> -----------------------------------------------------
>
>                 Key: NUTCH-2275
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2275
>             Project: Nutch
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.11
>            Reporter: Francesco Capponi
>            Priority: Major
>             Fix For: 1.17
>
>
> I'm testing Apache Nutch with the feed's plugin. I've noticed that for each page it generates the same digest/signature, therefore the dedup cleans everything up from the database.
> I'm wondering why the class MD5Signature is the default one instead of TextMD5Signature.
> Anyhow now I've modified a little bit the MD5Signature to let it work with the feed plugin



--
This message was sent by Atlassian Jira
(v8.3.4#803005)