[jira] [Commented] (NUTCH-2461) Generate passes the data to when maxCount == 0

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326454#comment-16326454 ] 

ASF GitHub Bot commented on NUTCH-2461:
---------------------------------------

lewismc closed pull request #249: fix for NUTCH-2461 generate with maxcount equals 0
URL: https://github.com/apache/nutch/pull/249
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/java/org/apache/nutch/crawl/Generator.java b/src/java/org/apache/nutch/crawl/Generator.java
index e5f4831d0..dc1f899da 100644
--- a/src/java/org/apache/nutch/crawl/Generator.java
+++ b/src/java/org/apache/nutch/crawl/Generator.java
@@ -377,7 +377,6 @@ public void reduce(FloatWritable key, Iterator<SelectorEntry> values,
               LOG.info("Generator: variable maxCount: {} for {}", variableMaxCount, hostname);
               maxCount = (int)variableMaxCount;
             }
-            
             if (fetchDelayExpr != null) {
               long variableFetchDelay = Math.round((double)fetchDelayExpr.evaluate(createContext(host)));
               LOG.info("Generator: variable fetchDelay: {} ms for {}", variableFetchDelay, hostname);
@@ -386,6 +385,10 @@ public void reduce(FloatWritable key, Iterator<SelectorEntry> values,
           }
         }
         
+        if(maxCount == 0){ 
+        	continue; 
+        }
+        
         // Got a non-zero variable fetch delay? Add it to the datum's metadata
         if (variableFetchDelayWritable != null) {
           entry.datum.getMetaData().put(variableFetchDelayKey, variableFetchDelayWritable);


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Generate passes the data to when maxCount  == 0
> -----------------------------------------------
>
>                 Key: NUTCH-2461
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2461
>             Project: Nutch
>          Issue Type: Bug
>          Components: generator
>    Affects Versions: 1.14
>            Reporter: Semyon Semyonov
>            Priority: Critical
>             Fix For: 1.15
>
>
> The generator checks condition 
> if (maxCount > 0) : line 421 and stop the generation when amount per host exceeds maxCount( continue : line 455)
> but when  maxCount == 0 it goes directly to line 465 :output.collect(key, entry);
> It is obviously not correct, the correct solution would be to add 
> if(maxCount == 0){
>         	continue;
> }
> at line 380.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)