You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Samuel Guo (JIRA)" <ji...@apache.org> on 2008/12/01 08:56:46 UTC
[jira] Updated: (HAMA-116) ArrayIndexOutOfBoundsException at
BlockingMapRed
[ https://issues.apache.org/jira/browse/HAMA-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Samuel Guo updated HAMA-116:
----------------------------
Attachment: HAMA-116.patch
the bug is caused by grouping wrong *BlockID* together.
The *BlockID* 's raw comparator has implementation bug. so I discard them, we just used the *WritableComparable* 's compare method.
And in *BlockingMapRed*, I also discard *BlockRowId* and *GroupingComparator* these two classes, because the *SubMatrix* is working in a random-access way, so we don't need to sort them during reducing.
I test "mult -m 4 -r 4 20 20", it seems OK.
> bin/hama examples mult -m 4 -r 4 20 20
08/12/02 00:51:31 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/02 00:51:46 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randvvjvr
08/12/02 00:51:46 INFO hama.AbstractMatrix: Create the 20 * 20 random matrix : DenseMatrix_randvvjvr
08/12/02 00:51:47 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/02 00:51:57 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randssufx
08/12/02 00:51:57 INFO hama.AbstractMatrix: Create the 20 * 20 random matrix : DenseMatrix_randssufx
08/12/02 00:51:58 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
08/12/02 00:51:58 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/02 00:51:58 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/02 00:51:59 INFO mapred.JobClient: Running job: job_200812020041_0001
08/12/02 00:52:00 INFO mapred.JobClient: map 0% reduce 0%
08/12/02 00:52:10 INFO mapred.JobClient: map 50% reduce 0%
08/12/02 00:52:16 INFO mapred.JobClient: map 75% reduce 0%
08/12/02 00:52:19 INFO mapred.JobClient: map 100% reduce 0%
08/12/02 00:52:24 INFO mapred.JobClient: map 100% reduce 8%
08/12/02 00:52:25 INFO mapred.JobClient: map 100% reduce 29%
08/12/02 00:52:27 INFO mapred.JobClient: map 100% reduce 50%
08/12/02 00:52:34 INFO mapred.JobClient: map 100% reduce 75%
08/12/02 00:52:36 INFO mapred.JobClient: Job complete: job_200812020041_0001
08/12/02 00:52:36 INFO mapred.JobClient: Counters: 13
08/12/02 00:52:36 INFO mapred.JobClient: File Systems
08/12/02 00:52:36 INFO mapred.JobClient: Local bytes read=23346
08/12/02 00:52:36 INFO mapred.JobClient: Local bytes written=47144
08/12/02 00:52:36 INFO mapred.JobClient: Job Counters
08/12/02 00:52:36 INFO mapred.JobClient: Launched reduce tasks=4
08/12/02 00:52:36 INFO mapred.JobClient: Launched map tasks=4
08/12/02 00:52:36 INFO mapred.JobClient: Map-Reduce Framework
08/12/02 00:52:36 INFO mapred.JobClient: Reduce input groups=16
08/12/02 00:52:36 INFO mapred.JobClient: Combine output records=0
08/12/02 00:52:36 INFO mapred.JobClient: Map input records=33
08/12/02 00:52:36 INFO mapred.JobClient: Reduce output records=0
08/12/02 00:52:36 INFO mapred.JobClient: Map output bytes=22718
08/12/02 00:52:36 INFO mapred.JobClient: Map input bytes=0
08/12/02 00:52:36 INFO mapred.JobClient: Combine input records=0
08/12/02 00:52:36 INFO mapred.JobClient: Map output records=132
08/12/02 00:52:36 INFO mapred.JobClient: Reduce input records=132
08/12/02 00:52:36 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
08/12/02 00:52:36 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/02 00:52:36 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/02 00:52:37 INFO mapred.JobClient: Running job: job_200812020041_0002
08/12/02 00:52:38 INFO mapred.JobClient: map 0% reduce 0%
08/12/02 00:52:44 INFO mapred.JobClient: map 25% reduce 0%
08/12/02 00:52:48 INFO mapred.JobClient: map 50% reduce 0%
08/12/02 00:52:52 INFO mapred.JobClient: map 75% reduce 0%
08/12/02 00:52:57 INFO mapred.JobClient: map 75% reduce 2%
08/12/02 00:53:02 INFO mapred.JobClient: map 75% reduce 10%
08/12/02 00:53:12 INFO mapred.JobClient: map 75% reduce 12%
08/12/02 00:55:55 INFO mapred.JobClient: map 100% reduce 12%
08/12/02 00:55:57 INFO mapred.JobClient: map 100% reduce 31%
08/12/02 00:56:03 INFO mapred.JobClient: map 100% reduce 50%
08/12/02 00:56:08 INFO mapred.JobClient: map 100% reduce 75%
08/12/02 00:56:12 INFO mapred.JobClient: Job complete: job_200812020041_0002
08/12/02 00:56:12 INFO mapred.JobClient: Counters: 13
08/12/02 00:56:12 INFO mapred.JobClient: File Systems
08/12/02 00:56:12 INFO mapred.JobClient: Local bytes read=23346
08/12/02 00:56:12 INFO mapred.JobClient: Local bytes written=47144
08/12/02 00:56:12 INFO mapred.JobClient: Job Counters
08/12/02 00:56:12 INFO mapred.JobClient: Launched reduce tasks=4
08/12/02 00:56:12 INFO mapred.JobClient: Launched map tasks=4
08/12/02 00:56:12 INFO mapred.JobClient: Map-Reduce Framework
08/12/02 00:56:12 INFO mapred.JobClient: Reduce input groups=16
08/12/02 00:56:12 INFO mapred.JobClient: Combine output records=0
08/12/02 00:56:12 INFO mapred.JobClient: Map input records=33
08/12/02 00:56:12 INFO mapred.JobClient: Reduce output records=0
08/12/02 00:56:12 INFO mapred.JobClient: Map output bytes=22718
08/12/02 00:56:12 INFO mapred.JobClient: Map input bytes=0
08/12/02 00:56:12 INFO mapred.JobClient: Combine input records=0
08/12/02 00:56:12 INFO mapred.JobClient: Map output records=132
08/12/02 00:56:12 INFO mapred.JobClient: Reduce input records=132
08/12/02 00:56:12 INFO hama.AbstractMatrix: Initializing the matrix storage.
08/12/02 00:56:22 INFO hama.AbstractMatrix: Create Matrix DenseMatrix_randvxlnn
08/12/02 00:56:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
08/12/02 00:56:22 WARN mapred.JobClient: Use genericOptions for the option -libjars
08/12/02 00:56:23 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/12/02 00:56:23 INFO mapred.JobClient: Running job: job_200812020041_0003
08/12/02 00:56:24 INFO mapred.JobClient: map 0% reduce 0%
08/12/02 00:56:33 INFO mapred.JobClient: map 100% reduce 0%
08/12/02 00:56:43 INFO mapred.JobClient: map 100% reduce 25%
08/12/02 00:56:49 INFO mapred.JobClient: map 100% reduce 50%
08/12/02 00:56:51 INFO mapred.JobClient: map 100% reduce 75%
08/12/02 00:56:55 INFO mapred.JobClient: Job complete: job_200812020041_0003
08/12/02 00:56:55 INFO mapred.JobClient: Counters: 13
08/12/02 00:56:55 INFO mapred.JobClient: File Systems
08/12/02 00:56:55 INFO mapred.JobClient: Local bytes read=24952
08/12/02 00:56:55 INFO mapred.JobClient: Local bytes written=50104
08/12/02 00:56:55 INFO mapred.JobClient: Job Counters
08/12/02 00:56:55 INFO mapred.JobClient: Launched reduce tasks=4
08/12/02 00:56:55 INFO mapred.JobClient: Launched map tasks=2
08/12/02 00:56:55 INFO mapred.JobClient: Map-Reduce Framework
08/12/02 00:56:55 INFO mapred.JobClient: Reduce input groups=4
08/12/02 00:56:55 INFO mapred.JobClient: Combine output records=0
08/12/02 00:56:55 INFO mapred.JobClient: Map input records=4
08/12/02 00:56:55 INFO mapred.JobClient: Reduce output records=80
08/12/02 00:56:55 INFO mapred.JobClient: Map output bytes=24448
08/12/02 00:56:55 INFO mapred.JobClient: Map input bytes=0
08/12/02 00:56:55 INFO mapred.JobClient: Combine input records=0
08/12/02 00:56:55 INFO mapred.JobClient: Map output records=64
08/12/02 00:56:55 INFO mapred.JobClient: Reduce input records=64
08/12/02 00:57:05 INFO client.HBaseAdmin: Disabled DenseMatrix_randvvjvr
08/12/02 00:57:05 INFO client.HBaseAdmin: Deleted DenseMatrix_randvvjvr
08/12/02 00:57:16 INFO client.HBaseAdmin: Disabled DenseMatrix_randssufx
08/12/02 00:57:16 INFO client.HBaseAdmin: Deleted DenseMatrix_randssufx
08/12/02 00:57:26 INFO client.HBaseAdmin: Disabled DenseMatrix_randvxlnn
08/12/02 00:57:27 INFO client.HBaseAdmin: Deleted DenseMatrix_randvxlnn
so attach my patch, and try this.
> ArrayIndexOutOfBoundsException at BlockingMapRed
> ------------------------------------------------
>
> Key: HAMA-116
> URL: https://issues.apache.org/jira/browse/HAMA-116
> Project: Hama
> Issue Type: Bug
> Components: mapred
> Reporter: Edward J. Yoon
> Fix For: 0.1.0
>
> Attachments: HAMA-116.patch
>
>
> [d8g053:/root/hama-trunk]# bin/hama examples mult -m 100 -r 100 1000 1000
> ----
> 08/11/30 11:56:07 INFO mapred.JobClient: map 100% reduce 1%
> 08/11/30 11:56:26 INFO mapred.JobClient: map 100% reduce 0%
> 08/11/30 11:56:26 INFO mapred.JobClient: Task Id : attempt_200811271544_0006_r_0
> 00001_1, Status : FAILED
> java.lang.ArrayIndexOutOfBoundsException: 10
> at org.apache.hama.SubMatrix.set(SubMatrix.java:61)
> at org.apache.hama.mapred.BlockingMapRed$BlockingReducer.reduce(Blocking
> MapRed.java:146)
> at org.apache.hama.mapred.BlockingMapRed$BlockingReducer.reduce(Blocking
> MapRed.java:127)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207
> )
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.