You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (Jira)" <ji...@apache.org> on 2021/06/08 12:27:00 UTC
[jira] [Updated] (NUTCH-2865) WARC exporter support for metadata
and dropping empty responses
[ https://issues.apache.org/jira/browse/NUTCH-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-2865:
---------------------------------
Description:
WARCExporter is a handy tool to dump the segments. Unfortunately it also emits WARC records for status' other than success of notmodified, which accounts for a decent number in each crawl cycle. It also doesn't emit parsed metadata or extracted text. It does now.
This patch adds three switches:
* -includeOnlySuccessfulResponses to only emit records of success or notmodified
* -includeParseData to also emit parse metadata as WARC metadata record
* -includeParseText to also emit extracted text as WARC metadata
Both metadata objects are stored in the same WARC metadata record to save space.
was:
WARCExporter is a handy tool to dump the segments. Unfortunately it also emits WARC records for status' other than success of notmodified, which accounts for a decent number in each crawl cycle. It also doesn't emit parsed metadata or extracted text. It does now.
This patch adds three switches:
* -omitEmptyResponses to only emit records of success or notmodified
* -includeParseData to also emit parse metadata as WARC metadata record
* -includeParseText to also emit extracted text as WARC metadata
Both metadata objects are stored in the same WARC metadata record to save space.
> WARC exporter support for metadata and dropping empty responses
> ---------------------------------------------------------------
>
> Key: NUTCH-2865
> URL: https://issues.apache.org/jira/browse/NUTCH-2865
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.19
>
> Attachments: NUTCH-2865.patch, NUTCH-2865.patch
>
>
> WARCExporter is a handy tool to dump the segments. Unfortunately it also emits WARC records for status' other than success of notmodified, which accounts for a decent number in each crawl cycle. It also doesn't emit parsed metadata or extracted text. It does now.
>
> This patch adds three switches:
> * -includeOnlySuccessfulResponses to only emit records of success or notmodified
> * -includeParseData to also emit parse metadata as WARC metadata record
> * -includeParseText to also emit extracted text as WARC metadata
> Both metadata objects are stored in the same WARC metadata record to save space.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)