You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Aseem Anand <as...@gmail.com> on 2013/02/05 14:11:01 UTC
QuickSort array out of bound exception during Secondary Sort
Hi,
I'm facing a problem with hadoop's secondary sort such that it is
displaying the following error message. The code I have used has been used
by me previously and had not given any issues for a large number of
intermediate output pairs. The number of mapper output values is around
175,000. Could someone please guide me on this .
13/02/05 18:34:27 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ArrayIndexOutOfBoundsException: 1
at org.skyz.basic.KeyComparator.compare(KeyComparator.java:23)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:106)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
Hoping for a positive response.
Thanks & Regards,
Aseem Anand
Re: QuickSort array out of bound exception during Secondary Sort
Posted by Robert Evans <ev...@yahoo-inc.com>.
Without access to org.skyz.basic.KeyComparator.compare it is hard to say why this is happening. It looks like you got an array with 0 entries and the given function could not handle that.
I don't really know why.
--Bobby
From: Aseem Anand <as...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 5, 2013 7:11 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: QuickSort array out of bound exception during Secondary Sort
Hi,
I'm facing a problem with hadoop's secondary sort such that it is displaying the following error message. The code I have used has been used by me previously and had not given any issues for a large number of intermediate output pairs. The number of mapper output values is around 175,000. Could someone please guide me on this .
13/02/05 18:34:27 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ArrayIndexOutOfBoundsException: 1
at org.skyz.basic.KeyComparator.compare(KeyComparator.java:23)
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:106)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
Hoping for a positive response.
Thanks & Regards,
Aseem Anand
Re: QuickSort array out of bound exception during Secondary Sort
Posted by Robert Evans <ev...@yahoo-inc.com>.
Without access to org.skyz.basic.KeyComparator.compare it is hard to say why this is happening. It looks like you got an array with 0 entries and the given function could not handle that.
I don't really know why.
--Bobby
From: Aseem Anand <as...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 5, 2013 7:11 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: QuickSort array out of bound exception during Secondary Sort
Hi,
I'm facing a problem with hadoop's secondary sort such that it is displaying the following error message. The code I have used has been used by me previously and had not given any issues for a large number of intermediate output pairs. The number of mapper output values is around 175,000. Could someone please guide me on this .
13/02/05 18:34:27 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ArrayIndexOutOfBoundsException: 1
at org.skyz.basic.KeyComparator.compare(KeyComparator.java:23)
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:106)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
Hoping for a positive response.
Thanks & Regards,
Aseem Anand
Re: QuickSort array out of bound exception during Secondary Sort
Posted by Robert Evans <ev...@yahoo-inc.com>.
Without access to org.skyz.basic.KeyComparator.compare it is hard to say why this is happening. It looks like you got an array with 0 entries and the given function could not handle that.
I don't really know why.
--Bobby
From: Aseem Anand <as...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 5, 2013 7:11 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: QuickSort array out of bound exception during Secondary Sort
Hi,
I'm facing a problem with hadoop's secondary sort such that it is displaying the following error message. The code I have used has been used by me previously and had not given any issues for a large number of intermediate output pairs. The number of mapper output values is around 175,000. Could someone please guide me on this .
13/02/05 18:34:27 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ArrayIndexOutOfBoundsException: 1
at org.skyz.basic.KeyComparator.compare(KeyComparator.java:23)
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:106)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
Hoping for a positive response.
Thanks & Regards,
Aseem Anand
Re: QuickSort array out of bound exception during Secondary Sort
Posted by Robert Evans <ev...@yahoo-inc.com>.
Without access to org.skyz.basic.KeyComparator.compare it is hard to say why this is happening. It looks like you got an array with 0 entries and the given function could not handle that.
I don't really know why.
--Bobby
From: Aseem Anand <as...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 5, 2013 7:11 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: QuickSort array out of bound exception during Secondary Sort
Hi,
I'm facing a problem with hadoop's secondary sort such that it is displaying the following error message. The code I have used has been used by me previously and had not given any issues for a large number of intermediate output pairs. The number of mapper output values is around 175,000. Could someone please guide me on this .
13/02/05 18:34:27 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ArrayIndexOutOfBoundsException: 1
at org.skyz.basic.KeyComparator.compare(KeyComparator.java:23)
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:106)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
Hoping for a positive response.
Thanks & Regards,
Aseem Anand