OutofMemory Error with updatedb

[Sandeep Tata <sa...@gmail.com>]

I've been running nutch-0.9 on a cluster of 3 linux machines.

I've been able to crawl to crawl about 2M pages in several segments of
around 200-300K pages each.

The updatedb job now fails consistently with this error:

java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOf(Arrays.java:2786)
 at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
 at java.io.DataOutputStream.write(DataOutputStream.java:90)
 at org.apache.hadoop.io.Text.write(Text.java:243)
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:313)
 at org.apache.nutch.crawl.CrawlDbFilter.map(CrawlDbFilter.java:99)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
 at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)

I tried upping the java heap space from 1000M to 1500M, and that
doesn't seem to help. (Each of the machines have about 3G)

It seems to launch 31 map tasks for 3 machines. Only about 5 of them
fail with the outofmemory error. 26 of them actually complete
successfully. Should I modify the code to use fewer map tasks? Should
I increase the heap space further? Do let me know if you need any of
the other configuration parameters.

If anyone has suggestions, I'd appreciate that!

Sandeep

Re: OutofMemory Error with updatedb

[Dennis Kubes <ku...@apache.org>]

You may want to try increasing the number of map tasks so that each task 
has a smaller set if possible.  If it is all from the same host(s) that 
won't matter though.  Depending on what you are fetching (PDFs, web 
pages) you can change the http.content.limit and file.content.limit 
configurations to have less content to process.

You may also want to try upping your memory again.  Make sure you are 
setting the mapred.child.java.opts variable in your hadoop-site.xml file 
to something like -Xmx2000m and not just changing the conf/hadoop-env.sh 
  HADOOP_HEAPSIZE variable which only affects the hadoop servers and not 
the child tasks.

If nothing else works you can set the following variable(s) in your 
hadoop-site.xml file:

mapred.max.map.failures.percent
mapred.max.reduce.failures.percent

These variables allow a certain percentage of map or reduce tasks to 
fail without the entire job failing.  AFAIK, these configuration 
variables are not documented and both default to 0.  So for your setup 
if nothing else works you could set the map.failures to say 10 for 10 
percent.  If you do this I would set the keep.failed.task.files variable 
to true to allow you to see later what went wrong.  The tasks would 
still fail but the entire job would then complete.

Dennis

Sandeep Tata wrote:
> I've been running nutch-0.9 on a cluster of 3 linux machines.
> 
> I've been able to crawl to crawl about 2M pages in several segments of
> around 200-300K pages each.
> 
> The updatedb job now fails consistently with this error:
> 
> java.lang.OutOfMemoryError: Java heap space
>  at java.util.Arrays.copyOf(Arrays.java:2786)
>  at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
>  at java.io.DataOutputStream.write(DataOutputStream.java:90)
>  at org.apache.hadoop.io.Text.write(Text.java:243)
>  at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:313)
>  at org.apache.nutch.crawl.CrawlDbFilter.map(CrawlDbFilter.java:99)
>  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
>  at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
> 
> I tried upping the java heap space from 1000M to 1500M, and that
> doesn't seem to help. (Each of the machines have about 3G)
> 
> It seems to launch 31 map tasks for 3 machines. Only about 5 of them
> fail with the outofmemory error. 26 of them actually complete
> successfully. Should I modify the code to use fewer map tasks? Should
> I increase the heap space further? Do let me know if you need any of
> the other configuration parameters.
> 
> If anyone has suggestions, I'd appreciate that!
> 
> Sandeep