You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by somnath <so...@gmail.com> on 2011/06/14 15:36:50 UTC
How to access the Reducer id in partitioner
By using context,getTaskAttemptID().getTaskID(), we can have the task id of
the reducer but this code does not work for partitioner , So can anyone
tell me how to get the Reducer id in partitioner. I am using Hadoop 0.21
version.
--
View this message in context: http://hadoop-common.472056.n3.nabble.com/How-to-access-the-Reducer-id-in-partitioner-tp3063040p3063040.html
Sent from the Users mailing list archive at Nabble.com.
Re: Handling external jars in EMR
Posted by madhu phatak <ph...@gmail.com>.
Its better to merge the library with ur code . Other wise u have to copy the
library to every lib folder of HADOOP in every node cluster. libjars is not
working for me also . I used maven shade plugin (eclipse) to get the merged
jar.
On Wed, Jun 15, 2011 at 12:20 AM, Mehmet Tepedelenlioglu <
mehmetsino@gmail.com> wrote:
> I am using the Guava library in my hadoop code through a jar file. With
> hadoop one has the -libjars option (although I could not get that working on
> 0.2 for some reason). Are there any easy options with EMR short of using a
> utility like jarjar or bootstrapping magic. Or is that what I'll need to do?
>
> Thanks,
>
> Mehmet T.
Handling external jars in EMR
Posted by Mehmet Tepedelenlioglu <me...@gmail.com>.
I am using the Guava library in my hadoop code through a jar file. With hadoop one has the -libjars option (although I could not get that working on 0.2 for some reason). Are there any easy options with EMR short of using a utility like jarjar or bootstrapping magic. Or is that what I'll need to do?
Thanks,
Mehmet T.
Re: How to access the Reducer id in partitioner
Posted by somnath <so...@gmail.com>.
Thanks for your reply. Yes you are right but for my implementation I need to
send the super keys or Indexes to a reducer and to continue the work I need
the reducer id so that I can use in later stage.My implemenattion is quite
complex . The reducer will have some specific different keys and values
and they are grouped by based on the index key .....So each reducer will
have a specific index which in turn will conatin many keys and associated
values...
--
View this message in context: http://hadoop-common.472056.n3.nabble.com/How-to-access-the-Reducer-id-in-partitioner-tp3063040p3064021.html
Sent from the Users mailing list archive at Nabble.com.
Re: How to access the Reducer id in partitioner
Posted by Harsh J <ha...@cloudera.com>.
Somnath,
The Partitioner is applied in the Map-phase. The number (< n) you'd
return would serve as the ReduceID the K-V pair has to go to. Why do
you seek a Reducer ID here?
Perhaps I did not get your question exactly, care to rephrase?
On Tue, Jun 14, 2011 at 7:06 PM, somnath <so...@gmail.com> wrote:
> By using context,getTaskAttemptID().getTaskID(), we can have the task id of
> the reducer but this code does not work for partitioner , So can anyone
> tell me how to get the Reducer id in partitioner. I am using Hadoop 0.21
> version.
>
> --
> View this message in context: http://hadoop-common.472056.n3.nabble.com/How-to-access-the-Reducer-id-in-partitioner-tp3063040p3063040.html
> Sent from the Users mailing list archive at Nabble.com.
>
--
Harsh J