You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Paul Mahon <pm...@decarta.com> on 2014/01/06 21:21:00 UTC

Spill Failed Caused by ArrayIndexOutOfBoundsException

I have a hadoop program that I'm running with version 1.2.1 which 
fails in a peculiar place. Most mappers complete without error, but 
some fail with this stack trace:

java.io.IOException: Spill failed
	at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
	at 
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1793)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
	at 
org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
	at 
org.apache.hadoop.io.BooleanWritable$Comparator.compare(BooleanWritable.java:103)
	at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
	at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
	at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
	at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1404)
	at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:858)
	at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1349)

I've noticed that that array index is exactly the size of the bufvoid, 
but I'm not sure if that has any significance.

The exception isn't happening in my WritableComparable or any of my 
code, it's all in hadoop. I'm not sure what to do to track down what 
I'm doing to cause the problem. Has anyone seen a problem like this or 
have any suggestions of where to look for the problem in my code?

Re: Spill Failed Caused by ArrayIndexOutOfBoundsException

Posted by Hardik Pandya <sm...@gmail.com>.
The error is happening during Sort And Spill phase

org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill

It seems like you are trying to compare two Int values and it fails during
compare

Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
        at
org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
        at
org.apache.hadoop.io.BooleanWritable$Comparator.
compare(BooleanWritable.java:103)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
1404)


On Mon, Jan 6, 2014 at 3:21 PM, Paul Mahon <pm...@decarta.com> wrote:

> I have a hadoop program that I'm running with version 1.2.1 which
> fails in a peculiar place. Most mappers complete without error, but
> some fail with this stack trace:
>
> java.io.IOException: Spill failed
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
>         at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1793)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
>         at
>
> org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
>         at
>
> org.apache.hadoop.io.BooleanWritable$Comparator.compare(BooleanWritable.java:103)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
>         at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
>         at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1404)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:858)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1349)
>
> I've noticed that that array index is exactly the size of the bufvoid,
> but I'm not sure if that has any significance.
>
> The exception isn't happening in my WritableComparable or any of my
> code, it's all in hadoop. I'm not sure what to do to track down what
> I'm doing to cause the problem. Has anyone seen a problem like this or
> have any suggestions of where to look for the problem in my code?
>

Re: Spill Failed Caused by ArrayIndexOutOfBoundsException

Posted by Hardik Pandya <sm...@gmail.com>.
The error is happening during Sort And Spill phase

org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill

It seems like you are trying to compare two Int values and it fails during
compare

Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
        at
org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
        at
org.apache.hadoop.io.BooleanWritable$Comparator.
compare(BooleanWritable.java:103)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
1404)


On Mon, Jan 6, 2014 at 3:21 PM, Paul Mahon <pm...@decarta.com> wrote:

> I have a hadoop program that I'm running with version 1.2.1 which
> fails in a peculiar place. Most mappers complete without error, but
> some fail with this stack trace:
>
> java.io.IOException: Spill failed
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
>         at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1793)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
>         at
>
> org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
>         at
>
> org.apache.hadoop.io.BooleanWritable$Comparator.compare(BooleanWritable.java:103)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
>         at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
>         at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1404)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:858)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1349)
>
> I've noticed that that array index is exactly the size of the bufvoid,
> but I'm not sure if that has any significance.
>
> The exception isn't happening in my WritableComparable or any of my
> code, it's all in hadoop. I'm not sure what to do to track down what
> I'm doing to cause the problem. Has anyone seen a problem like this or
> have any suggestions of where to look for the problem in my code?
>

Re: Spill Failed Caused by ArrayIndexOutOfBoundsException

Posted by Hardik Pandya <sm...@gmail.com>.
The error is happening during Sort And Spill phase

org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill

It seems like you are trying to compare two Int values and it fails during
compare

Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
        at
org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
        at
org.apache.hadoop.io.BooleanWritable$Comparator.
compare(BooleanWritable.java:103)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
1404)


On Mon, Jan 6, 2014 at 3:21 PM, Paul Mahon <pm...@decarta.com> wrote:

> I have a hadoop program that I'm running with version 1.2.1 which
> fails in a peculiar place. Most mappers complete without error, but
> some fail with this stack trace:
>
> java.io.IOException: Spill failed
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
>         at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1793)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
>         at
>
> org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
>         at
>
> org.apache.hadoop.io.BooleanWritable$Comparator.compare(BooleanWritable.java:103)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
>         at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
>         at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1404)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:858)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1349)
>
> I've noticed that that array index is exactly the size of the bufvoid,
> but I'm not sure if that has any significance.
>
> The exception isn't happening in my WritableComparable or any of my
> code, it's all in hadoop. I'm not sure what to do to track down what
> I'm doing to cause the problem. Has anyone seen a problem like this or
> have any suggestions of where to look for the problem in my code?
>

Re: Spill Failed Caused by ArrayIndexOutOfBoundsException

Posted by Hardik Pandya <sm...@gmail.com>.
The error is happening during Sort And Spill phase

org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill

It seems like you are trying to compare two Int values and it fails during
compare

Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
        at
org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
        at
org.apache.hadoop.io.BooleanWritable$Comparator.
compare(BooleanWritable.java:103)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
1404)


On Mon, Jan 6, 2014 at 3:21 PM, Paul Mahon <pm...@decarta.com> wrote:

> I have a hadoop program that I'm running with version 1.2.1 which
> fails in a peculiar place. Most mappers complete without error, but
> some fail with this stack trace:
>
> java.io.IOException: Spill failed
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
>         at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1793)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 99614720
>         at
>
> org.apache.hadoop.io.WritableComparator.readInt(WritableComparator.java:158)
>         at
>
> org.apache.hadoop.io.BooleanWritable$Comparator.compare(BooleanWritable.java:103)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1116)
>         at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95)
>         at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1404)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:858)
>         at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1349)
>
> I've noticed that that array index is exactly the size of the bufvoid,
> but I'm not sure if that has any significance.
>
> The exception isn't happening in my WritableComparable or any of my
> code, it's all in hadoop. I'm not sure what to do to track down what
> I'm doing to cause the problem. Has anyone seen a problem like this or
> have any suggestions of where to look for the problem in my code?
>