You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Apache Wiki <wi...@apache.org> on 2016/05/19 12:41:25 UTC

[Nutch Wiki] Update of "bin/nutch generate" by SebastianNagel

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "bin/nutch generate" page has been changed by SebastianNagel:
https://wiki.apache.org/nutch/bin/nutch%20generate?action=diff&rev1=4&rev2=5

Comment:
Add information about scope (per segment / over all segments) of -topN and generate.max.count when multiple segments are generated

  
  '''<segments_dir>''': Path to the location of our segments directory where the Fetcher Segments are created.
  
- '''[-force]''': This arguement will force an update even if there appears to be a lock. /!\ : CAUTION: advised /!\
+ '''[-force]''': This argument will force an update even if there appears to be a lock. /!\ : CAUTION: advised /!\
  
  '''[-topN N]''': Where N is the number of top URLs to be selected. Normally, the "generate" command prepares a fetchlist out of all unfetched pages, or the ones where fetch interval already expired. But if you use -topN, then instead of all unfetched urls you only get N urls with the highest score - potentially the most interesting ones, which should be prioritized in fetching.
  
@@ -27, +27 @@

  
  '''[-noNorm]''': The exact same applies for normalisation parameter as does for the filtering option above.
  
- '''[-maxNumSegments num]''': The (maximum) number of segments to be generated. Default: 1
+ '''[-maxNumSegments num]''': The (maximum) number of segments to be generated. Default: 1 -- Note: if multiple segments are generated, the limit -topN applies to the total number of URLs for all segments taken together, while generate.max.count is applied to every generated segment individually. 
  
  ==== Configuration Files ====
   hadoop-default.xml<<BR>>