You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Arthur B (JIRA)" <ji...@apache.org> on 2016/10/19 11:24:59 UTC

[jira] [Issue Comment Deleted] (NUTCH-2328) GeneratorJob does not generate anything on second run

     [ https://issues.apache.org/jira/browse/NUTCH-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arthur B updated NUTCH-2328:
----------------------------
    Comment: was deleted

(was: Thanks for the heads up.
>> * local, per-reducer limit = topN / number of reducers - I guess if you do not care about missing generating  {{ topN mod mapred.reduce.tasks }}  pages this would work as well. I do not know how expensive the call to the marked counter would be (if you care about exact numbers)
)

> GeneratorJob does not generate anything on second run
> -----------------------------------------------------
>
>                 Key: NUTCH-2328
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2328
>             Project: Nutch
>          Issue Type: Bug
>          Components: generator
>    Affects Versions: 2.2, 2.3, 2.2.1, 2.3.1
>         Environment: Ubuntu 16.04 / Hadoop 2.7.1
>            Reporter: Arthur B
>              Labels: fails, generator, subsequent
>             Fix For: 2.4
>
>         Attachments: generator-issue-static-count.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Given a topN parameter (ie 10) the GeneratorJob will fail to generate anything new on the subsequent runs within the same process space.
> To reproduce the issue submit the GeneratorJob twice one after another to the M/R framework. Second time will say it generated 0 URLs.
> This issue is due to the usage of the static count field (org.apache.nutch.crawl.GeneratorReducer#count) to determine if the topN value has been reached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)