You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Keith Thompson <kt...@binghamton.edu> on 2011/10/15 23:26:52 UTC
implementing comparable
Hello,
I am trying to write my very first MapReduce code. When I try to run the
jar, I get this error:
11/10/15 17:17:30 INFO mapred.JobClient: Task Id :
attempt_201110151636_0003_m_000001_2, Status : FAILED
java.lang.ClassCastException: class edu.bing.vfi5.KeyList
at java.lang.Class.asSubclass(Class.java:3018)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:599)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
I assume this means that it has something to do with my implementation of
comparable. KeyList is a class for a 3-tuple key. The code is listed
below. Any hints would be greatly appreciated as I am trying to understand
how comparable is supposed to work. Also, do I need to implement Writable
as well? If so, should this be code for how the output is written to a file
in HDFS?
Thanks,
Keith
package edu.bing.vfi5;
public class KeyList implements Comparable<KeyList> {
private int[] keys;
public KeyList(int i, int j, int k) {
keys = new int[3];
keys[0] = i;
keys[0] = j;
keys[0] = k;
}
@Override
public int compareTo(KeyList k) {
// TODO Auto-generated method stub
if(this.keys[0] == k.keys[0] && this.keys[1] == k.keys[1] && this.keys[2] ==
k.keys[2])
return 0;
else if((this.keys[0]>k.keys[0])
||(this.keys[0]==k.keys[0]&&this.keys[1]>k.keys[1])
||(this.keys[0]==k.keys[0]&&this.keys[1]==k.keys[1]&&this.keys[2]>k.keys[2]))
return 1;
else
return -1;
}
}
Re: implementing comparable
Posted by Brock Noland <br...@cloudera.com>.
Hi,
Discussion, below.
On Sat, Oct 15, 2011 at 4:26 PM, Keith Thompson <kt...@binghamton.edu>wrote:
> Hello,
> I am trying to write my very first MapReduce code. When I try to run the
> jar, I get this error:
>
> 11/10/15 17:17:30 INFO mapred.JobClient: Task Id :
> attempt_201110151636_0003_m_000001_2, Status : FAILED
> java.lang.ClassCastException: class edu.bing.vfi5.KeyList
> at java.lang.Class.asSubclass(Class.java:3018)
> at
> org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:599)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> I assume this means that it has something to do with my implementation of
> comparable. KeyList is a class for a 3-tuple key. The code is listed
> below. Any hints would be greatly appreciated as I am trying to understand
> how comparable is supposed to work. Also, do I need to implement Writable
> as well? If so, should this be code for how the output is written to a
> file
> in HDFS?
>
> Thanks,
> Keith
>
> package edu.bing.vfi5;
>
> public class KeyList implements Comparable<KeyList> {
>
Key's need to be WritableComparable.
>
> private int[] keys;
> public KeyList(int i, int j, int k) {
> keys = new int[3];
> keys[0] = i;
> keys[0] = j;
> keys[0] = k;
> }
>
> @Override
> public int compareTo(KeyList k) {
> // TODO Auto-generated method stub
> if(this.keys[0] == k.keys[0] && this.keys[1] == k.keys[1] && this.keys[2]
> ==
> k.keys[2])
> return 0;
> else if((this.keys[0]>k.keys[0])
> ||(this.keys[0]==k.keys[0]&&this.keys[1]>k.keys[1])
>
> ||(this.keys[0]==k.keys[0]&&this.keys[1]==k.keys[1]&&this.keys[2]>k.keys[2]))
> return 1;
> else
> return -1;
> }
> }
>
Re: implementing comparable
Posted by Keith Thompson <kt...@binghamton.edu>.
Ahh ok ...I think I understand now. I added the default constructor (just
initializing all values to 0) and now it seems to be running. :-) Thanks
for your help.
On Sun, Oct 16, 2011 at 9:43 PM, Brock Noland <br...@cloudera.com> wrote:
> Hi,
>
> Inline..
>
> On Sun, Oct 16, 2011 at 9:40 PM, Keith Thompson <kthomps6@binghamton.edu
> >wrote:
>
> > Thanks. I went back and changed to WritableComparable instead of just
> > Comparable. So, I added the readFields and write methods. I also took
> > care of the typo in the constructor. :P
> >
> > Now I am getting this error:
> >
> > 11/10/16 21:34:08 INFO mapred.JobClient: Task Id :
> > attempt_201110162105_0002_m_000001_1, Status : FAILED
> > java.lang.RuntimeException: java.lang.NoSuchMethodException:
> > edu.bing.vfi5.KeyList.<init>()
> > at
> >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> > at
> >
> org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:84)
> > at
> >
> org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:70)
> > at
> org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:44)
> > at
> > org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:599)
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.lang.NoSuchMethodException:
> edu.bing.vfi5.KeyList.<init>()
> > at java.lang.Class.getConstructor0(Class.java:2706)
> > at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> > at
> >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> >
> > Is it saying it can't find the constructor?
> >
> >
> Writables and by extension WritableComparables need a default Constructor.
> This makes logical sense. If hadoop is going to call the readFields()
> method, it needs a previously constructed object.
>
> Brock
>
--
*Keith Thompson*
Graduate Research Associate, Xerox Corporation
SUNY Research Foundation
Dept. of Systems Science and Industrial Engineering
Binghamton University
work: 585-422-6587
Re: implementing comparable
Posted by Brock Noland <br...@cloudera.com>.
Hi,
Inline..
On Sun, Oct 16, 2011 at 9:40 PM, Keith Thompson <kt...@binghamton.edu>wrote:
> Thanks. I went back and changed to WritableComparable instead of just
> Comparable. So, I added the readFields and write methods. I also took
> care of the typo in the constructor. :P
>
> Now I am getting this error:
>
> 11/10/16 21:34:08 INFO mapred.JobClient: Task Id :
> attempt_201110162105_0002_m_000001_1, Status : FAILED
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> edu.bing.vfi5.KeyList.<init>()
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> at
> org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:84)
> at
> org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:70)
> at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:44)
> at
> org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:599)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.NoSuchMethodException: edu.bing.vfi5.KeyList.<init>()
> at java.lang.Class.getConstructor0(Class.java:2706)
> at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> at
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
>
> Is it saying it can't find the constructor?
>
>
Writables and by extension WritableComparables need a default Constructor.
This makes logical sense. If hadoop is going to call the readFields()
method, it needs a previously constructed object.
Brock
Re: implementing comparable
Posted by Keith Thompson <kt...@binghamton.edu>.
Thanks. I went back and changed to WritableComparable instead of just
Comparable. So, I added the readFields and write methods. I also took
care of the typo in the constructor. :P
Now I am getting this error:
11/10/16 21:34:08 INFO mapred.JobClient: Task Id :
attempt_201110162105_0002_m_000001_1, Status : FAILED
java.lang.RuntimeException: java.lang.NoSuchMethodException:
edu.bing.vfi5.KeyList.<init>()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:84)
at
org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:70)
at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:44)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:599)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.NoSuchMethodException: edu.bing.vfi5.KeyList.<init>()
at java.lang.Class.getConstructor0(Class.java:2706)
at java.lang.Class.getDeclaredConstructor(Class.java:1985)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
Is it saying it can't find the constructor?
On Sat, Oct 15, 2011 at 5:26 PM, Keith Thompson <kt...@binghamton.edu>wrote:
> Hello,
> I am trying to write my very first MapReduce code. When I try to run the
> jar, I get this error:
>
> 11/10/15 17:17:30 INFO mapred.JobClient: Task Id :
> attempt_201110151636_0003_m_000001_2, Status : FAILED
> java.lang.ClassCastException: class edu.bing.vfi5.KeyList
> at java.lang.Class.asSubclass(Class.java:3018)
> at
> org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:599)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> I assume this means that it has something to do with my implementation of
> comparable. KeyList is a class for a 3-tuple key. The code is listed
> below. Any hints would be greatly appreciated as I am trying to understand
> how comparable is supposed to work. Also, do I need to implement Writable
> as well? If so, should this be code for how the output is written to a file
> in HDFS?
>
> Thanks,
> Keith
>
> package edu.bing.vfi5;
>
> public class KeyList implements Comparable<KeyList> {
>
> private int[] keys;
> public KeyList(int i, int j, int k) {
> keys = new int[3];
> keys[0] = i;
> keys[1] = j;
> keys[2] = k;
> }
>
> @Override
> public int compareTo(KeyList k) {
> // TODO Auto-generated method stub
> if(this.keys[0] == k.keys[0] && this.keys[1] == k.keys[1] && this.keys[2]
> == k.keys[2])
> return 0;
> else if((this.keys[0]>k.keys[0])
> ||(this.keys[0]==k.keys[0]&&this.keys[1]>k.keys[1])
>
> ||(this.keys[0]==k.keys[0]&&this.keys[1]==k.keys[1]&&this.keys[2]>k.keys[2]))
> return 1;
> else
> return -1;
> }
> }
>
>
--
*Keith Thompson*
Graduate Research Associate, Xerox Corporation
SUNY Research Foundation
Dept. of Systems Science and Industrial Engineering
Binghamton University
work: 585-422-6587