[jira] [Commented] (NUTCH-2250) CommonCrawlDumper : Invalid format + skipped parts

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240877#comment-15240877 ] 

ASF GitHub Bot commented on NUTCH-2250:
---------------------------------------

GitHub user thammegowda opened a pull request:

    https://github.com/apache/nutch/pull/103

    NUTCH-2250 : CommonCrawlDumper : Invalid format and skipped parts

    ### Summary
    + Reads all parts of segments
    + writes single document per dump file
    
    
    ### What was the fix?
    1. Lists all files insides segments and selects all the parts matching to pattern part-[0-9]{5} under the content directory. 
    2. CommonCrawlFormat implementation for "JACKSON" type does not clear state, instead it keeps appending documents. This fix uses a new instance for each document (actually it was hard to resolve the state issue because of too many options such as compressing and archiving).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/thammegowda/nutch NUTCH-2250

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nutch/pull/103.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #103
    
----
commit 47cc4e27a41a78753ed431685010da1987bca269
Author: Thamme Gowda <tg...@gmail.com>
Date:   2016-04-14T09:38:30Z

    NUTCH-2250 : CommonCrawlDumper : Invalid format and skipped parts
    
    + Reads all parts of segements
    + FIX : writes only one document to dump file

----


>  CommonCrawlDumper : Invalid format + skipped parts
> ---------------------------------------------------
>
>                 Key: NUTCH-2250
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2250
>             Project: Nutch
>          Issue Type: Sub-task
>          Components: commoncrawl
>    Affects Versions: 1.12
>         Environment: Linux x64
> Java 7
> Nutch 1.12
>            Reporter: Thamme Gowda N
>             Fix For: 1.10
>
>
> The following issues are found with CommonCrawlDumper;
> 1. Documents get duplicated in dump files
> How to reproduce 
> {code}
> bin/nutch commoncrawldump  -segment .../segments -outputDir testdump -SimpleDateFormat -epochFilename -jsonArray -reverseKey
> {code}
> The first ever written will contain 1 document.
> second file includes two documents
> third file includes first three documents and this grows linearly.
> 2.If a segment has many parts (part-00000, part-00001,...) only the first part (part-00000 ) is being dumped
> How to reproduce ?
> Create segment with two parts (part-00000 and part-00001)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)