[jira] [Created] (NUTCH-2791) domainstats, protocolstats and crawlcomplete do not handle GCS URLs

["Patrick Mézard (Jira)" <ji...@apache.org>]

Patrick Mézard created NUTCH-2791:
-------------------------------------

             Summary: domainstats, protocolstats and crawlcomplete do not handle GCS URLs
                 Key: NUTCH-2791
                 URL: https://issues.apache.org/jira/browse/NUTCH-2791
             Project: Nutch
          Issue Type: Bug
          Components: util
    Affects Versions: 1.17
            Reporter: Patrick Mézard


I am running Nutch in GCP Dataproc. The domainstats, protocolstats and crawlcomplete commands do not resolve crawldb paths correctly from GCS URLs.

Also:
 * protocolstats has an off-by-one error when resolving the numReducers argument
 * crawlcomplete -inputDirs is inconsistent with other command: where domainstats expect directories containing a "current" path, crawlcomplete looks for directories containing a "crawldb/current" path.

I will send a PR after creating the issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)