You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by samir das mohapatra <sa...@gmail.com> on 2013/01/23 17:58:49 UTC
Custom Partitioner is now working in CDH4
Hi All
Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
is note working under CDH4.
Can you tell me why it is like that but in CDH3 same code is working fine.
PARTITONER CODE
-------------------------------------------
public static class AgePartitioner extends Partitioner<Text, Text> {
static{
System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
}
@Override
public int getPartition(Text key, Text value, int numReduceTasks) {
System.out.println("&##############################AgePartitioner"+value.toString());
String [] nameAgeScore = value.toString().split("\t");
String age = nameAgeScore[1];
int ageInt = Integer.parseInt(age);
//this is done to avoid performing mod with 0
if(numReduceTasks == 0)
return 0;
//if the age is <20, assign partition 0
if(ageInt <=20){
return 0;
}
//else if the age is between 20 and 50, assign partition 1
if(ageInt >20 && ageInt <=50){
return 1 % numReduceTasks;
}
//otherwise assign partition 2
else
return 2 % numReduceTasks;
}
}
DRIVER LEVEL CODE
-------------------------------------------
job.setPartitionerClass(AgePartitioner.class);
job.setNumReduceTasks(3);
Re: Custom Partitioner is now working in CDH4
Posted by Harsh J <ha...@cloudera.com>.
Moving to cdh-user@ as you seem to indicate this is CDH-specific. Lets
continue there.
My response below:
>From your question/code below, it is not clear _what_ you aren't seeing as
working alright. Are you observing an error, a use of default partitioner,
etc.? Could you clarify on what exactly isn't working after upgrade
(assuming you've recompiled the code for CDH4 already)?
On Wed, Jan 23, 2013 at 10:28 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:
> Hi All
> Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
> is note working under CDH4.
>
> Can you tell me why it is like that but in CDH3 same code is working fine.
>
> PARTITONER CODE
> -------------------------------------------
> public static class AgePartitioner extends Partitioner<Text, Text> {
> static{
>
> System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
> }
> @Override
> public int getPartition(Text key, Text value, int numReduceTasks) {
>
> System.out.println("&##############################AgePartitioner"+value.toString());
> String [] nameAgeScore = value.toString().split("\t");
> String age = nameAgeScore[1];
> int ageInt = Integer.parseInt(age);
>
> //this is done to avoid performing mod with 0
> if(numReduceTasks == 0)
> return 0;
>
> //if the age is <20, assign partition 0
> if(ageInt <=20){
> return 0;
> }
> //else if the age is between 20 and 50, assign partition 1
> if(ageInt >20 && ageInt <=50){
>
> return 1 % numReduceTasks;
> }
> //otherwise assign partition 2
> else
> return 2 % numReduceTasks;
>
> }
> }
>
>
>
> DRIVER LEVEL CODE
> -------------------------------------------
>
> job.setPartitionerClass(AgePartitioner.class);
> job.setNumReduceTasks(3);
>
>
>
>
>
--
Harsh J
Re: Custom Partitioner is now working in CDH4
Posted by Harsh J <ha...@cloudera.com>.
Moving to cdh-user@ as you seem to indicate this is CDH-specific. Lets
continue there.
My response below:
>From your question/code below, it is not clear _what_ you aren't seeing as
working alright. Are you observing an error, a use of default partitioner,
etc.? Could you clarify on what exactly isn't working after upgrade
(assuming you've recompiled the code for CDH4 already)?
On Wed, Jan 23, 2013 at 10:28 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:
> Hi All
> Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
> is note working under CDH4.
>
> Can you tell me why it is like that but in CDH3 same code is working fine.
>
> PARTITONER CODE
> -------------------------------------------
> public static class AgePartitioner extends Partitioner<Text, Text> {
> static{
>
> System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
> }
> @Override
> public int getPartition(Text key, Text value, int numReduceTasks) {
>
> System.out.println("&##############################AgePartitioner"+value.toString());
> String [] nameAgeScore = value.toString().split("\t");
> String age = nameAgeScore[1];
> int ageInt = Integer.parseInt(age);
>
> //this is done to avoid performing mod with 0
> if(numReduceTasks == 0)
> return 0;
>
> //if the age is <20, assign partition 0
> if(ageInt <=20){
> return 0;
> }
> //else if the age is between 20 and 50, assign partition 1
> if(ageInt >20 && ageInt <=50){
>
> return 1 % numReduceTasks;
> }
> //otherwise assign partition 2
> else
> return 2 % numReduceTasks;
>
> }
> }
>
>
>
> DRIVER LEVEL CODE
> -------------------------------------------
>
> job.setPartitionerClass(AgePartitioner.class);
> job.setNumReduceTasks(3);
>
>
>
>
>
--
Harsh J
Re: Custom Partitioner is now working in CDH4
Posted by Harsh J <ha...@cloudera.com>.
Moving to cdh-user@ as you seem to indicate this is CDH-specific. Lets
continue there.
My response below:
>From your question/code below, it is not clear _what_ you aren't seeing as
working alright. Are you observing an error, a use of default partitioner,
etc.? Could you clarify on what exactly isn't working after upgrade
(assuming you've recompiled the code for CDH4 already)?
On Wed, Jan 23, 2013 at 10:28 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:
> Hi All
> Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
> is note working under CDH4.
>
> Can you tell me why it is like that but in CDH3 same code is working fine.
>
> PARTITONER CODE
> -------------------------------------------
> public static class AgePartitioner extends Partitioner<Text, Text> {
> static{
>
> System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
> }
> @Override
> public int getPartition(Text key, Text value, int numReduceTasks) {
>
> System.out.println("&##############################AgePartitioner"+value.toString());
> String [] nameAgeScore = value.toString().split("\t");
> String age = nameAgeScore[1];
> int ageInt = Integer.parseInt(age);
>
> //this is done to avoid performing mod with 0
> if(numReduceTasks == 0)
> return 0;
>
> //if the age is <20, assign partition 0
> if(ageInt <=20){
> return 0;
> }
> //else if the age is between 20 and 50, assign partition 1
> if(ageInt >20 && ageInt <=50){
>
> return 1 % numReduceTasks;
> }
> //otherwise assign partition 2
> else
> return 2 % numReduceTasks;
>
> }
> }
>
>
>
> DRIVER LEVEL CODE
> -------------------------------------------
>
> job.setPartitionerClass(AgePartitioner.class);
> job.setNumReduceTasks(3);
>
>
>
>
>
--
Harsh J
Re: Custom Partitioner is now working in CDH4
Posted by Harsh J <ha...@cloudera.com>.
Moving to cdh-user@ as you seem to indicate this is CDH-specific. Lets
continue there.
My response below:
>From your question/code below, it is not clear _what_ you aren't seeing as
working alright. Are you observing an error, a use of default partitioner,
etc.? Could you clarify on what exactly isn't working after upgrade
(assuming you've recompiled the code for CDH4 already)?
On Wed, Jan 23, 2013 at 10:28 PM, samir das mohapatra <
samir.helpdoc@gmail.com> wrote:
> Hi All
> Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
> is note working under CDH4.
>
> Can you tell me why it is like that but in CDH3 same code is working fine.
>
> PARTITONER CODE
> -------------------------------------------
> public static class AgePartitioner extends Partitioner<Text, Text> {
> static{
>
> System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
> }
> @Override
> public int getPartition(Text key, Text value, int numReduceTasks) {
>
> System.out.println("&##############################AgePartitioner"+value.toString());
> String [] nameAgeScore = value.toString().split("\t");
> String age = nameAgeScore[1];
> int ageInt = Integer.parseInt(age);
>
> //this is done to avoid performing mod with 0
> if(numReduceTasks == 0)
> return 0;
>
> //if the age is <20, assign partition 0
> if(ageInt <=20){
> return 0;
> }
> //else if the age is between 20 and 50, assign partition 1
> if(ageInt >20 && ageInt <=50){
>
> return 1 % numReduceTasks;
> }
> //otherwise assign partition 2
> else
> return 2 % numReduceTasks;
>
> }
> }
>
>
>
> DRIVER LEVEL CODE
> -------------------------------------------
>
> job.setPartitionerClass(AgePartitioner.class);
> job.setNumReduceTasks(3);
>
>
>
>
>
--
Harsh J