You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Ben K <bl...@uvic.ca> on 2013/11/18 21:11:59 UTC

Can't get a streaming job to use a custom partitioner

I need help. No matter what I do I can't seem to get Hadoop to find my custom partitioner.
Here is the command I am running:

../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
-libjars ./NumericPartitioner.jar \
-input /input -output /output/keys -mapper "map_threeJoin.py" -reducer "keycount.py" \
-partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file "keycount.py"

(The code of NumericPartitioner is very simple, and is here: http://pastebin.com/ZEK7N1RN)
But no matter what I do, it gives:

-partitioner : class not found : newjoin.NumericPartitioner

Does anyone have any idea why it might be going wrong?

Ben K



Re: Can't get a streaming job to use a custom partitioner

Posted by Adam Kawa <ka...@gmail.com>.
1) Could you print the output of:
$ jar tf ./NumericPartitioner.jar

2) Could you try:
$ export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:./NumericPartitioner.jar"

$ ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
-libjars ./NumericPartitioner.jar \
-input /input -output /output/keys -mapper "map_threeJoin.py" -reducer
"keycount.py" \
-partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file
"keycount.py"



2013/11/18 Ben K <bl...@uvic.ca>

> I need help. No matter what I do I can't seem to get Hadoop to find my
> custom partitioner.
> Here is the command I am running:
>
> ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
> -libjars ./NumericPartitioner.jar \
> -input /input -output /output/keys -mapper "map_threeJoin.py" -reducer
> "keycount.py" \
> -partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file
> "keycount.py"
>
> (The code of NumericPartitioner is very simple, and is here:
> http://pastebin.com/ZEK7N1RN)
> But no matter what I do, it gives:
>
> -partitioner : class not found : newjoin.NumericPartitioner
>
> Does anyone have any idea why it might be going wrong?
>
> Ben K
>
>
>

Re: Can't get a streaming job to use a custom partitioner

Posted by Adam Kawa <ka...@gmail.com>.
1) Could you print the output of:
$ jar tf ./NumericPartitioner.jar

2) Could you try:
$ export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:./NumericPartitioner.jar"

$ ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
-libjars ./NumericPartitioner.jar \
-input /input -output /output/keys -mapper "map_threeJoin.py" -reducer
"keycount.py" \
-partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file
"keycount.py"



2013/11/18 Ben K <bl...@uvic.ca>

> I need help. No matter what I do I can't seem to get Hadoop to find my
> custom partitioner.
> Here is the command I am running:
>
> ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
> -libjars ./NumericPartitioner.jar \
> -input /input -output /output/keys -mapper "map_threeJoin.py" -reducer
> "keycount.py" \
> -partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file
> "keycount.py"
>
> (The code of NumericPartitioner is very simple, and is here:
> http://pastebin.com/ZEK7N1RN)
> But no matter what I do, it gives:
>
> -partitioner : class not found : newjoin.NumericPartitioner
>
> Does anyone have any idea why it might be going wrong?
>
> Ben K
>
>
>

Re: Can't get a streaming job to use a custom partitioner

Posted by Adam Kawa <ka...@gmail.com>.
1) Could you print the output of:
$ jar tf ./NumericPartitioner.jar

2) Could you try:
$ export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:./NumericPartitioner.jar"

$ ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
-libjars ./NumericPartitioner.jar \
-input /input -output /output/keys -mapper "map_threeJoin.py" -reducer
"keycount.py" \
-partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file
"keycount.py"



2013/11/18 Ben K <bl...@uvic.ca>

> I need help. No matter what I do I can't seem to get Hadoop to find my
> custom partitioner.
> Here is the command I am running:
>
> ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
> -libjars ./NumericPartitioner.jar \
> -input /input -output /output/keys -mapper "map_threeJoin.py" -reducer
> "keycount.py" \
> -partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file
> "keycount.py"
>
> (The code of NumericPartitioner is very simple, and is here:
> http://pastebin.com/ZEK7N1RN)
> But no matter what I do, it gives:
>
> -partitioner : class not found : newjoin.NumericPartitioner
>
> Does anyone have any idea why it might be going wrong?
>
> Ben K
>
>
>

Re: Can't get a streaming job to use a custom partitioner

Posted by Adam Kawa <ka...@gmail.com>.
1) Could you print the output of:
$ jar tf ./NumericPartitioner.jar

2) Could you try:
$ export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:./NumericPartitioner.jar"

$ ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
-libjars ./NumericPartitioner.jar \
-input /input -output /output/keys -mapper "map_threeJoin.py" -reducer
"keycount.py" \
-partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file
"keycount.py"



2013/11/18 Ben K <bl...@uvic.ca>

> I need help. No matter what I do I can't seem to get Hadoop to find my
> custom partitioner.
> Here is the command I am running:
>
> ../bin/hadoop jar ../contrib/streaming/hadoop-streaming-1.2.1.jar \
> -libjars ./NumericPartitioner.jar \
> -input /input -output /output/keys -mapper "map_threeJoin.py" -reducer
> "keycount.py" \
> -partitioner newjoin.NumericPartitioner -file "map_threeJoin.py" -file
> "keycount.py"
>
> (The code of NumericPartitioner is very simple, and is here:
> http://pastebin.com/ZEK7N1RN)
> But no matter what I do, it gives:
>
> -partitioner : class not found : newjoin.NumericPartitioner
>
> Does anyone have any idea why it might be going wrong?
>
> Ben K
>
>
>