You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Arthur B (JIRA)" <ji...@apache.org> on 2016/10/18 16:14:58 UTC
[jira] [Comment Edited] (NUTCH-2328) GeneratorJob does not generate
anything on second run
[ https://issues.apache.org/jira/browse/NUTCH-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585843#comment-15585843 ]
Arthur B edited comment on NUTCH-2328 at 10/18/16 4:14 PM:
-----------------------------------------------------------
The environment that this was observed:
* spring batched jobs / spring data hadoop, submitting GeneratorJob M/R jobs, so obviously the GeneratorReducer#count survived as static;
* in a pseudo-distributed Hadoop setup
Maybe I was missing something but the reason was cautious about making count instance field because this counter would have to be seen across the whole Hadoop cluster, right? So thats why I relied on the M/R counter to sync across the Hadoop cluster the actual pages counted as processed.
was (Author: arthur-evozon):
The environment that this was observed:
* spring batched jobs / spring data hadoop, submitting GeneratorJob M/R jobs, so obviously the GeneratorReducer#count survived as static;
* in a pseudo-distributed Hadoop setup
Maybe I was missing something but the reason was cautious about making count instance field because this counter would have to be seen across the whole Hadoop cluster, right? So thats why I relied on the M/R counter to sync across the actual hbase pages counted as processed.
> GeneratorJob does not generate anything on second run
> -----------------------------------------------------
>
> Key: NUTCH-2328
> URL: https://issues.apache.org/jira/browse/NUTCH-2328
> Project: Nutch
> Issue Type: Bug
> Components: generator
> Affects Versions: 2.2, 2.3, 2.2.1, 2.3.1
> Environment: Ubuntu 16.04 / Hadoop 2.7.1
> Reporter: Arthur B
> Labels: fails, generator, subsequent
> Fix For: 2.4
>
> Attachments: generator-issue-static-count.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Given a topN parameter (ie 10) the GeneratorJob will fail to generate anything new on the subsequent runs within the same process space.
> To reproduce the issue submit the GeneratorJob twice one after another to the M/R framework. Second time will say it generated 0 URLs.
> This issue is due to the usage of the static count field (org.apache.nutch.crawl.GeneratorReducer#count) to determine if the topN value has been reached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)