You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Sebastian Nagel (JIRA)" <ji...@apache.org> on 2018/07/27 13:32:00 UTC

[jira] [Created] (NUTCH-2628) Fetcher: optionally generate signature of unparsed content

Sebastian Nagel created NUTCH-2628:
--------------------------------------

             Summary: Fetcher: optionally generate signature of unparsed content
                 Key: NUTCH-2628
                 URL: https://issues.apache.org/jira/browse/NUTCH-2628
             Project: Nutch
          Issue Type: Improvement
          Components: fetcher
            Reporter: Sebastian Nagel
             Fix For: 1.16


To generate a document signature (MD5 digest) of the binary content requires that documents are parsed during the parse or fetch step. The signature is required for deduplication and detection of unmodified content and should be always available, also in a workflow which does not require that documents are parsed, e.g., because HTML content is exported to WARC files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)