[Nutch Wiki] Update of "Nutch0.9-Hadoop0.10-Tutorial" by AndrzejBialecki

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The following page has been changed by AndrzejBialecki:
http://wiki.apache.org/nutch/Nutch0%2e9-Hadoop0%2e10-Tutorial

The comment on the change is:
Corrected some misunderstanding about the config files.

------------------------------------------------------------------------------
  }}}
  
  === Configure Hadoop ===
+ Edit the mapred-default.xml configuration file. If it's missing, create it, with the following content:
+ {{{
+ <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+ 
+ <configuration>
+ 
+ <property> 
+   <name>mapred.map.tasks</name>
+   <value>2</value>
+   <description>
+     This should be a prime number larger than multiple number of slave hosts,
+     e.g. for 3 nodes set this to 17
+   </description> 
+ </property> 
+ 
+ <property> 
+   <name>mapred.reduce.tasks</name>
+   <value>2</value>
+   <description>
+     This should be a prime number close to a low multiple of slave hosts,
+     e.g. for 3 nodes set this to 7
+   </description> 
+ </property> 
+ 
+ </configuration>
+ }}}
+ 
+ ''Note: do NOT put these properties in hadoop-site.xml. That file (see below) should only contain properties characteristic to the cluster, and not to the job. Misplacing these properties leads to strange and difficult to debug problems - e.g. inability to specify the number of map/reduce tasks programmatically (Generator and Fetcher depend on this).''
+ 
  Edit the hadoop-site.xml configuration file.
  {{{
  <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
@@ -115, +144 @@

    </description>
  </property>
  
- <property> 
+ <property>
-   <name>mapred.map.tasks</name>
+   <name>mapred.tasktracker.tasks.maximum</name>
    <value>2</value>
    <description>
-     define mapred.map tasks to be number of slave hosts
+     The maximum number of tasks that will be run simultaneously by
+     a task tracker. This should be adjusted according to the heap size
+     per task, the amount of RAM available, and CPU consumption of each task.
-   </description> 
+   </description>
- </property> 
+ </property>
  
- <property> 
+ <property>
-   <name>mapred.reduce.tasks</name>
+   <name>mapred.child.java.opts</name>
-   <value>2</value>
+   <value>-Xmx200m</value>
    <description>
-     define mapred.reduce tasks to be number of slave hosts
+     You can specify other Java options for each map or reduce task here,
+     but most likely you will want to adjust the heap size.
-   </description> 
+   </description>
- </property> 
+ </property>
  
  <property>
    <name>dfs.name.dir</name>
@@ -462, +494 @@

  
  = Comments =
  == Number of map reduce tasks ==
- I noticed that the number of map and reduce task has an impact on the performance of Hadoop. Many times after crawling a lot of pages the nodes reported 'java.lang.OutOfMemoryError: Java heap space' errors, this happend also in the indexing part. Increasing the number of maps solved these problems, with an index that has over 200.000 pages I needed 306 maps in total over 3 machines. By setting the mapred.maps.tasks property in hadoop-site.xml to 99 (much higher than what is advised in other tutorials and in the hadoop-site.xml file) that problem is solved.
+ I noticed that the number of map and reduce task has an impact on the performance of Hadoop. Many times after crawling a lot of pages the nodes reported 'java.lang.OutOfMemoryError: Java heap space' errors, this happened also in the indexing part. Increasing the number of maps solved these problems, with an index that has over 200.000 pages I needed 306 maps in total over 3 machines. By setting the mapred.maps.tasks property in hadoop-site.xml to 99 (much higher than what is advised in other tutorials and in the hadoop-site.xml file) that problem is solved.
+ 
+ ''(ab) As noted above, do NOT set the number of map/reduce tasks in hadoop-site.xml, put them in mapred-default.xml instead.''
  
  See [http://wiki.apache.org/lucene-hadoop/HowManyMapsAndReduces] for more info about the number of map reduce tasks.