[jira] [Commented] (NUTCH-2865) WARC exporter support for metadata and dropping empty responses

["Sebastian Nagel (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446676#comment-17446676 ] 

Sebastian Nagel commented on NUTCH-2865:
----------------------------------------

Hi [~markus17], looks good and tests went successfully. I've made two changes in [6bb30c7|https://github.com/sebastian-nagel/nutch/commit/6bb30c719550f6d7b8efc127c052ed1a21816321]:
- add the Content-Type to metadata and text records. So, it's now:
  -- metadata: {{WARC-Type: metadata\r\nContent-Type: application/json}}, cf. [WARC "metadata" record|https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#metadata]
  -- text: {{WARC-Type: conversion\r\nContent-Type: text/plain}}, cf. [WARC "conversion" record|https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#conversion]
- apply code formats

If there are no objections I'll commit this. 

> WARC exporter support for metadata and dropping empty responses
> ---------------------------------------------------------------
>
>                 Key: NUTCH-2865
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2865
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>            Priority: Minor
>             Fix For: 1.19
>
>         Attachments: NUTCH-2865.patch, NUTCH-2865.patch, NUTCH-2865.patch
>
>
> WARCExporter is a handy tool to dump the segments. Unfortunately it also emits WARC records for status' other than success of notmodified, which accounts for a decent number in each crawl cycle. It also doesn't emit parsed metadata or extracted text. It does now.
>  
> This patch adds three switches:
>  * -includeOnlySuccessfulResponses to only emit records of success or notmodified
>  * -includeParseData to also emit parse metadata as WARC metadata record
>  * -includeParseText to also emit extracted text as WARC metadata
> Both metadata objects are stored in the same WARC metadata record to save space.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)