Map Task is failing with out of memory issue

[Pallavi Palleti <pa...@gmail.com>]

Hi,
 I have running map-reduce task on huge data on 40 Quad Core(16GB RAM)
Machine hadoop cluster. As my mapper machines sufficiently run on 1GB heap
size. I had set map reduce tasks heap size to 1GB. And, total number of
tasks per node is set to 9 ( 5 map tasks and 4 reduce tasks). It took huge
amount of time and finally after mapper completed 98%, the job failed with
below error:

java.lang.OutOfMemoryError: Java heap space
	at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:52)
	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
	at
org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java:1974)
	at
org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:3002)
	at
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2802)
	at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2511)
	at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1040)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)

I am not able to figure out where things went wrong. Can some one please
help me in this?

Thanks
Pallavi

-- 
View this message in context: http://www.nabble.com/Map-Task-is-failing-with-out-of-memory-issue-tp19202548p19202548.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Map Task is failing with out of memory issue

[Jason Venner <ja...@attributor.com>]

We have occasionally seen that when there is either damage to the 
sequence file or there was a read error.

What is happening is that the object size to be read is garbage, and the 
system trys to allocate 2^32 bytes to de-serialize the object.

In our case the .crc file was not catching the damage, but I don't 
recall the reason why at present, as it was some months back that we had 
trouble with this.


Pallavi Palleti wrote:
> Hi,
>  I have running map-reduce task on huge data on 40 Quad Core(16GB RAM)
> Machine hadoop cluster. As my mapper machines sufficiently run on 1GB heap
> size. I had set map reduce tasks heap size to 1GB. And, total number of
> tasks per node is set to 9 ( 5 map tasks and 4 reduce tasks). It took huge
> amount of time and finally after mapper completed 98%, the job failed with
> below error:
>
> java.lang.OutOfMemoryError: Java heap space
> 	at
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:52)
> 	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
> 	at
> org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java:1974)
> 	at
> org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:3002)
> 	at
> org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2802)
> 	at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2511)
> 	at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1040)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
>
> I am not able to figure out where things went wrong. Can some one please
> help me in this?
>
> Thanks
> Pallavi
>
>   
-- 
Jason Venner
Attributor - Program the Web <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers and coding wizards, contact if 
interested