You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Prasanth Iyer (JIRA)" <ji...@apache.org> on 2014/11/26 22:48:12 UTC

[jira] [Created] (NUTCH-1892) Update the FileDumper tool to fetch only those URLs with status db_fetched in nutch

Prasanth Iyer created NUTCH-1892:
------------------------------------

             Summary: Update the FileDumper tool to fetch only those URLs with status db_fetched in nutch
                 Key: NUTCH-1892
                 URL: https://issues.apache.org/jira/browse/NUTCH-1892
             Project: Nutch
          Issue Type: Improvement
          Components: nutchNewbie
    Affects Versions: 2.2.1
            Reporter: Prasanth Iyer


The FileDumper tool is a tool that reads the crawled data from Nutch and dumps this data into its raw files. This tool currently dumps every single file irrespective of status, duplicates etc. This cause files that are fetched in error or files that have not been fetched because they were made unavailable by the server to also be dumped. 

The fix should be to fetch only those files that were fetched with status db_fetched by Nutch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)