Error while merging in reduce step

[Vivek Sehgal <vi...@compasslabs.com>]

Hi,
I am getting the following error in the reduce step while merging/coping is
happening:

java.io.IOException: Task: attempt_201306130308_0177_r_000002_0 - The
reduce copier failed
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:384)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
	at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.io.IOException: Intermediate merge failed
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2703)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2628)
Caused by: java.io.IOException: Rec# 13932: Failed to skip past record
of length: 129
	at org.apache.hadoop.mapred.IFile$InMemoryReader.next(IFile.java:542)
	at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:220)
	at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:330)
	at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
	at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2687)
	... 1 more

If, I look at the code at this location:


long skipped = dataIn
<http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#IFile.Reader.0dataIn>.skip
<http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/FilterInputStream.java#FilterInputStream.skip%28long%29>(recordLength);

541 <http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#541>

<http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#>

      if (skipped != recordLength) {

542 <http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#542>

<http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#>

        throw new IOException
<http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/IOException.java#IOException>("Rec#
" + recNo <http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#IFile.Reader.0recNo>
+ ": Failed to skip past record of length: " +

543 <http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#543>

<http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#>

                              recordLength);

544 <http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#544>

<http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/IFile.java#>

      }


Basically, somehow the in-memory read of the next record is
inconsistent. How is this possible? Is it because the buffer cannot
completely fit in memory and I need to increase the memory?


Thanks

Vivek