[Nutch Wiki] Update of "NutchGotchas" by LewisJohnMcgibbney

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "NutchGotchas" page has been changed by LewisJohnMcgibbney:
http://wiki.apache.org/nutch/NutchGotchas?action=diff&rev1=4&rev2=5

Comment:
more gotchas

  
  == Current Gotchas and using them: ==
  
- '''No agents listed in 'http.agent.name' property.''': 
+ '''No agents listed in 'http.agent.name' property''': 
  
  Since 1.3 Nutch is called from either of the runtime dirs (runtime/local and runtime/deploy). The conf files should be modified in runtime/local/conf, not in $NUTCH_HOME/conf.
  
@@ -37, +37 @@

   * During the crawl command, as explained [[http://wiki.apache.org/nutch/RunningNutchAndSolr#A3._Crawl_your_first_website|here]].
   * or during the later stage of sending an individual solrindex command to Solr as explained [[http://wiki.apache.org/nutch/RunningNutchAndSolr#A6._Integrate_Solr_with_Nutch|here]]. 
  
+ '''DiskErrorException while fetching''':
+ 
+ Questions like this one arise fairly regularly on the user@ list
+ 
+ {{{
+ Hello,
+ 
+ I am getting some exception while fetching:
+ 
+ 2011-07-10 23:25:21,427 WARN  mapred.LocalJobRunner - job_local_0001
+ org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
+ taskTracker/jobcache/job_local_0001/attempt_local_0001_m_000000_0/output/spill0.out
+ in any of the configured local directories
+        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389)
+        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
+        at org.apache.hadoop.mapred.MapOutputFile.getSpillFile(MapOutputFile.java:94)
+        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1443)
+        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154)
+        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)
+        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
+        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
+ 2011-07-10 23:25:22,279 FATAL fetcher.Fetcher - Fetcher:
+ java.io.IOException: Job failed!
+        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
+        at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
+        at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:1145)
+        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
+        at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:1116)
+ 
+ What should I do? What happens if I restart the fetch job?
+ }}}
+ 
+ The answer we find addressed the situation is that you're most likely out of disk space in /tmp. Consider
+ using another location, or possibly another partition for hadoop.tmp.dir (which can be set in nutch-site.xml) with plenty of room for large transient files or using a Hadoop cluster.
+