You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Raymond Jennings III <ra...@yahoo.com> on 2010/06/03 19:13:11 UTC

Cutom partitioner question

I am trying to create my partitioner but I am getting an exception.  Is anything required other than providing the method "public int getPartition" and extending the Partitioner class?



java.lang.RuntimeException: java.lang.NoSuchMethodException: TSPmrV6$TSPPartitioner.<init>()
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.NoSuchMethodException: TSPmrV6$TSPPartitioner.<init>()
	at java.lang.Class.getConstructor0(Unknown Source)
	at java.lang.Class.getDeclaredConstructor(Unknown Source)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
	... 4 more

Re: Cutom partitioner question

Posted by Raymond Jennings III <ra...@yahoo.com>.
Hi Ted, that does not appear to be the problem I am having.  I tried adding it as you said but I get the same runtime error.  Here is my partitioner:

  public class MyPartitioner extends Partitioner<Text, Text> {
	  
	public MyPartitioner() {
		
	}
	  
	public int getPartition(Text key, Text value, int num_partitions) {
		String key2 = key.toString();
		int hash = key2.hashCode();
		
		hash = hash % num_partitions;
	
		return(hash);
	}	  
  }


and in my main I have:

job.setMapOutputValueClass(Text.class);
job.setMapOutputKeyClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);

job.setPartitionerClass(MyPartitioner.class);


Thanks.

--- On Thu, 6/3/10, Ted Yu <yu...@gmail.com> wrote:

> From: Ted Yu <yu...@gmail.com>
> Subject: Re: Cutom partitioner question
> To: common-user@hadoop.apache.org
> Date: Thursday, June 3, 2010, 2:10 PM
> An empty ctor is needed for your
> Partitioner class.
> 
> On Thu, Jun 3, 2010 at 10:13 AM, Raymond Jennings III
> <raymondjiii@yahoo.com
> > wrote:
> 
> > I am trying to create my partitioner but I am getting
> an exception.  Is
> > anything required other than providing the method
> "public int getPartition"
> > and extending the Partitioner class?
> >
> >
> >
> > java.lang.RuntimeException:
> java.lang.NoSuchMethodException:
> > TSPmrV6$TSPPartitioner.<init>()
> >        at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> >        at
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527)
> >        at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> >        at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >        at
> org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.lang.NoSuchMethodException:
> TSPmrV6$TSPPartitioner.<init>()
> >        at
> java.lang.Class.getConstructor0(Unknown Source)
> >        at
> java.lang.Class.getDeclaredConstructor(Unknown Source)
> >        at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> >        ... 4 more
> >
> >
> >
> >
> >
>

Re: Cutom partitioner question

Posted by Ted Yu <yu...@gmail.com>.
An empty ctor is needed for your Partitioner class.

On Thu, Jun 3, 2010 at 10:13 AM, Raymond Jennings III <raymondjiii@yahoo.com
> wrote:

> I am trying to create my partitioner but I am getting an exception.  Is
> anything required other than providing the method "public int getPartition"
> and extending the Partitioner class?
>
>
>
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> TSPmrV6$TSPPartitioner.<init>()
>        at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
>        at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.NoSuchMethodException: TSPmrV6$TSPPartitioner.<init>()
>        at java.lang.Class.getConstructor0(Unknown Source)
>        at java.lang.Class.getDeclaredConstructor(Unknown Source)
>        at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
>        ... 4 more
>
>
>
>
>