You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by somnath <so...@gmail.com> on 2011/06/14 15:36:50 UTC

How to access the Reducer id in partitioner

By using context,getTaskAttemptID().getTaskID(), we can have the  task id of
the reducer  but this code does not work for partitioner , So can anyone
tell me how to get the  Reducer id in partitioner. I am using Hadoop 0.21
version.

--
View this message in context: http://hadoop-common.472056.n3.nabble.com/How-to-access-the-Reducer-id-in-partitioner-tp3063040p3063040.html
Sent from the Users mailing list archive at Nabble.com.

Re: Handling external jars in EMR

Posted by madhu phatak <ph...@gmail.com>.
Its better to merge the library with ur code . Other wise u have to copy the
library to every lib folder of HADOOP in every node cluster. libjars is not
working for me also . I used maven shade plugin (eclipse) to get the merged
jar.

On Wed, Jun 15, 2011 at 12:20 AM, Mehmet Tepedelenlioglu <
mehmetsino@gmail.com> wrote:

> I am using the Guava library in my hadoop code through a jar file. With
> hadoop one has the -libjars option (although I could not get that working on
> 0.2 for some reason). Are there any easy options with EMR short of using a
> utility like jarjar or bootstrapping magic. Or is that what I'll need to do?
>
> Thanks,
>
> Mehmet T.

Handling external jars in EMR

Posted by Mehmet Tepedelenlioglu <me...@gmail.com>.
I am using the Guava library in my hadoop code through a jar file. With hadoop one has the -libjars option (although I could not get that working on 0.2 for some reason). Are there any easy options with EMR short of using a utility like jarjar or bootstrapping magic. Or is that what I'll need to do?

Thanks,

Mehmet T.

Re: How to access the Reducer id in partitioner

Posted by somnath <so...@gmail.com>.
Thanks for your reply. Yes you are right but for my implementation I need to
send the super keys or Indexes to a reducer and to continue the work I need
the reducer id so that I can use in later stage.My implemenattion is quite
complex   . The reducer will have some specific different keys and values
and they are grouped by based on the index key .....So each reducer will
have a specific index which in turn will conatin many keys and associated
values... 

--
View this message in context: http://hadoop-common.472056.n3.nabble.com/How-to-access-the-Reducer-id-in-partitioner-tp3063040p3064021.html
Sent from the Users mailing list archive at Nabble.com.

Re: How to access the Reducer id in partitioner

Posted by Harsh J <ha...@cloudera.com>.
Somnath,

The Partitioner is applied in the Map-phase. The number (< n) you'd
return would serve as the ReduceID the K-V pair has to go to. Why do
you seek a Reducer ID here?

Perhaps I did not get your question exactly, care to rephrase?

On Tue, Jun 14, 2011 at 7:06 PM, somnath <so...@gmail.com> wrote:
> By using context,getTaskAttemptID().getTaskID(), we can have the  task id of
> the reducer  but this code does not work for partitioner , So can anyone
> tell me how to get the  Reducer id in partitioner. I am using Hadoop 0.21
> version.
>
> --
> View this message in context: http://hadoop-common.472056.n3.nabble.com/How-to-access-the-Reducer-id-in-partitioner-tp3063040p3063040.html
> Sent from the Users mailing list archive at Nabble.com.
>



-- 
Harsh J