You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "P.ILAYARAJA" <il...@rediff.co.in> on 2008/11/03 15:57:09 UTC

Nutch/Hadoop: Crawl is crashing

Hi,

I started an internet crawl of 30 million pages in a single segment.
The crawl was crashing with the following exception:

java.lang.ArrayIndexOutOfBoundsException: 17
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:540)
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:607)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:193)
 at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)


Any idea on why is it happenning and what would be the soln.

... am using hadoop 0.15.3 and nutch 1.0 versions.

Regards,
Ilay