You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Vikas Jadhav <vi...@gmail.com> on 2013/03/13 07:15:17 UTC
How to shuffle (Key,Value) pair from mapper to multiple reducer
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one
reducer
I have use case where i need to shufffle same (key,value) pair to multiple
reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by samir das mohapatra <sa...@gmail.com>.
Use can use Custom Partitioner for that same.
Regards,
Samir.
On Wed, Mar 13, 2013 at 2:29 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example we have pair (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
> Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Ajay Srivastava <Aj...@guavus.com>.
Emit (key, value) twice from mapper by modifying key as key' = (key, partId) and record becomes (key', value)
>From custom partitioner, send record to reducer based on partId. Ignore partId field in reducer.
Regards,
Ajay Srivastava
On 13-Mar-2013, at 2:29 PM, Vikas Jadhav wrote:
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) = (1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com>> wrote:
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>> wrote:
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one reducer
I have use case where i need to shufffle same (key,value) pair to multiple reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
Thanx and Regards
Vikas Jadhav
--
Don't Grow Old, Grow Up... :-)
--
Thanx and Regards
Vikas Jadhav
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by samir das mohapatra <sa...@gmail.com>.
Use can use Custom Partitioner for that same.
Regards,
Samir.
On Wed, Mar 13, 2013 at 2:29 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example we have pair (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
> Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Artem Onuchin <on...@gmail.com>.
Hello Vikas!
Well you can duplicate your pair in mapper for each reducer, add reducer's
mark to keys and write a partitioner that will partition records acording
this marks.
I mean something like this:
In mapper you produce (key_r1 value) (key_r2 value) instead (key value)
But i cannot imagine why you need that.
WBR, Onuchin Artem
2013/3/13 Viral Bajaria <vi...@gmail.com>
> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>> Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Karthik Kambatla <ka...@cloudera.com>.
How about sending <0,x> to 0 and <1,x> to 1; reduce 0 can act based on the
value of x?
On Wed, Mar 13, 2013 at 2:29 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello I am not talking about custom partioner(custom partitioner is
> involved but i want to write same pair for more number times)
> i want it go to two reducer.
> for example i have partioning attribute two dimensional
> <x1,x2>
>
> singatue reduce
> <0,0> 0
> <0,1> 1
> <1,0> 2
> <1,1> 3
>
> for <1,0> it will goto reducer
> for <1,null> it should goto to reducer 2 and 3
> for <0,null> it should goto reducer 0 and 1
>
> On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <vi...@gmail.com>wrote:
>
>> Do you want the pair to go to both reducers or do you want it to go to
>> only one but in a random fashion ?
>>
>> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
>> 2nd is possible by just implementing your own partitioner which
>> randomizes where each key goes (not sure what you gain by that).
>>
>>
>> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>>
>>> Hi
>>> I am specifying requirement again with example.
>>>
>>>
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>>
>>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>>> reducer1) are there then
>>>
>>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>>> (1%2) )
>>>
>>>
>>> how i should shuffle this pair to both reducer.
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>> Thank you
>>>
>>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> you can use Job#setNumReduceTasks(int tasks) method to set the number
>>>> of reducer to output.
>>>>
>>>>
>>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vikascjadhav87@gmail.com
>>>> > wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> As by default Hadoop framework can shuffle (key,value) pair to only
>>>>> one reducer
>>>>>
>>>>> I have use case where i need to shufffle same (key,value) pair to
>>>>> multiple reducers
>>>>>
>>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>>
>>>>>
>>>>> Thank you
>>>>>
>>>>> --
>>>>> *
>>>>> *
>>>>> *
>>>>>
>>>>> Thanx and Regards*
>>>>> * Vikas Jadhav*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Don't Grow Old, Grow Up... :-)
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Karthik Kambatla <ka...@cloudera.com>.
How about sending <0,x> to 0 and <1,x> to 1; reduce 0 can act based on the
value of x?
On Wed, Mar 13, 2013 at 2:29 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello I am not talking about custom partioner(custom partitioner is
> involved but i want to write same pair for more number times)
> i want it go to two reducer.
> for example i have partioning attribute two dimensional
> <x1,x2>
>
> singatue reduce
> <0,0> 0
> <0,1> 1
> <1,0> 2
> <1,1> 3
>
> for <1,0> it will goto reducer
> for <1,null> it should goto to reducer 2 and 3
> for <0,null> it should goto reducer 0 and 1
>
> On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <vi...@gmail.com>wrote:
>
>> Do you want the pair to go to both reducers or do you want it to go to
>> only one but in a random fashion ?
>>
>> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
>> 2nd is possible by just implementing your own partitioner which
>> randomizes where each key goes (not sure what you gain by that).
>>
>>
>> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>>
>>> Hi
>>> I am specifying requirement again with example.
>>>
>>>
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>>
>>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>>> reducer1) are there then
>>>
>>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>>> (1%2) )
>>>
>>>
>>> how i should shuffle this pair to both reducer.
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>> Thank you
>>>
>>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> you can use Job#setNumReduceTasks(int tasks) method to set the number
>>>> of reducer to output.
>>>>
>>>>
>>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vikascjadhav87@gmail.com
>>>> > wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> As by default Hadoop framework can shuffle (key,value) pair to only
>>>>> one reducer
>>>>>
>>>>> I have use case where i need to shufffle same (key,value) pair to
>>>>> multiple reducers
>>>>>
>>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>>
>>>>>
>>>>> Thank you
>>>>>
>>>>> --
>>>>> *
>>>>> *
>>>>> *
>>>>>
>>>>> Thanx and Regards*
>>>>> * Vikas Jadhav*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Don't Grow Old, Grow Up... :-)
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Karthik Kambatla <ka...@cloudera.com>.
How about sending <0,x> to 0 and <1,x> to 1; reduce 0 can act based on the
value of x?
On Wed, Mar 13, 2013 at 2:29 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello I am not talking about custom partioner(custom partitioner is
> involved but i want to write same pair for more number times)
> i want it go to two reducer.
> for example i have partioning attribute two dimensional
> <x1,x2>
>
> singatue reduce
> <0,0> 0
> <0,1> 1
> <1,0> 2
> <1,1> 3
>
> for <1,0> it will goto reducer
> for <1,null> it should goto to reducer 2 and 3
> for <0,null> it should goto reducer 0 and 1
>
> On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <vi...@gmail.com>wrote:
>
>> Do you want the pair to go to both reducers or do you want it to go to
>> only one but in a random fashion ?
>>
>> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
>> 2nd is possible by just implementing your own partitioner which
>> randomizes where each key goes (not sure what you gain by that).
>>
>>
>> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>>
>>> Hi
>>> I am specifying requirement again with example.
>>>
>>>
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>>
>>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>>> reducer1) are there then
>>>
>>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>>> (1%2) )
>>>
>>>
>>> how i should shuffle this pair to both reducer.
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>> Thank you
>>>
>>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> you can use Job#setNumReduceTasks(int tasks) method to set the number
>>>> of reducer to output.
>>>>
>>>>
>>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vikascjadhav87@gmail.com
>>>> > wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> As by default Hadoop framework can shuffle (key,value) pair to only
>>>>> one reducer
>>>>>
>>>>> I have use case where i need to shufffle same (key,value) pair to
>>>>> multiple reducers
>>>>>
>>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>>
>>>>>
>>>>> Thank you
>>>>>
>>>>> --
>>>>> *
>>>>> *
>>>>> *
>>>>>
>>>>> Thanx and Regards*
>>>>> * Vikas Jadhav*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Don't Grow Old, Grow Up... :-)
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Karthik Kambatla <ka...@cloudera.com>.
How about sending <0,x> to 0 and <1,x> to 1; reduce 0 can act based on the
value of x?
On Wed, Mar 13, 2013 at 2:29 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello I am not talking about custom partioner(custom partitioner is
> involved but i want to write same pair for more number times)
> i want it go to two reducer.
> for example i have partioning attribute two dimensional
> <x1,x2>
>
> singatue reduce
> <0,0> 0
> <0,1> 1
> <1,0> 2
> <1,1> 3
>
> for <1,0> it will goto reducer
> for <1,null> it should goto to reducer 2 and 3
> for <0,null> it should goto reducer 0 and 1
>
> On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <vi...@gmail.com>wrote:
>
>> Do you want the pair to go to both reducers or do you want it to go to
>> only one but in a random fashion ?
>>
>> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
>> 2nd is possible by just implementing your own partitioner which
>> randomizes where each key goes (not sure what you gain by that).
>>
>>
>> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>>
>>> Hi
>>> I am specifying requirement again with example.
>>>
>>>
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>>
>>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>>> reducer1) are there then
>>>
>>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>>> (1%2) )
>>>
>>>
>>> how i should shuffle this pair to both reducer.
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>> Thank you
>>>
>>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> you can use Job#setNumReduceTasks(int tasks) method to set the number
>>>> of reducer to output.
>>>>
>>>>
>>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vikascjadhav87@gmail.com
>>>> > wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> As by default Hadoop framework can shuffle (key,value) pair to only
>>>>> one reducer
>>>>>
>>>>> I have use case where i need to shufffle same (key,value) pair to
>>>>> multiple reducers
>>>>>
>>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>>
>>>>>
>>>>> Thank you
>>>>>
>>>>> --
>>>>> *
>>>>> *
>>>>> *
>>>>>
>>>>> Thanx and Regards*
>>>>> * Vikas Jadhav*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Don't Grow Old, Grow Up... :-)
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Vikas Jadhav <vi...@gmail.com>.
Hello I am not talking about custom partioner(custom partitioner is
involved but i want to write same pair for more number times)
i want it go to two reducer.
for example i have partioning attribute two dimensional
<x1,x2>
singatue reduce
<0,0> 0
<0,1> 1
<1,0> 2
<1,1> 3
for <1,0> it will goto reducer
for <1,null> it should goto to reducer 2 and 3
for <0,null> it should goto reducer 0 and 1
On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <vi...@gmail.com>wrote:
> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>> Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Vikas Jadhav <vi...@gmail.com>.
Hello I am not talking about custom partioner(custom partitioner is
involved but i want to write same pair for more number times)
i want it go to two reducer.
for example i have partioning attribute two dimensional
<x1,x2>
singatue reduce
<0,0> 0
<0,1> 1
<1,0> 2
<1,1> 3
for <1,0> it will goto reducer
for <1,null> it should goto to reducer 2 and 3
for <0,null> it should goto reducer 0 and 1
On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <vi...@gmail.com>wrote:
> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>> Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Artem Onuchin <on...@gmail.com>.
Hello Vikas!
Well you can duplicate your pair in mapper for each reducer, add reducer's
mark to keys and write a partitioner that will partition records acording
this marks.
I mean something like this:
In mapper you produce (key_r1 value) (key_r2 value) instead (key value)
But i cannot imagine why you need that.
WBR, Onuchin Artem
2013/3/13 Viral Bajaria <vi...@gmail.com>
> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>> Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Artem Onuchin <on...@gmail.com>.
Hello Vikas!
Well you can duplicate your pair in mapper for each reducer, add reducer's
mark to keys and write a partitioner that will partition records acording
this marks.
I mean something like this:
In mapper you produce (key_r1 value) (key_r2 value) instead (key value)
But i cannot imagine why you need that.
WBR, Onuchin Artem
2013/3/13 Viral Bajaria <vi...@gmail.com>
> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>> Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Vikas Jadhav <vi...@gmail.com>.
Hello I am not talking about custom partioner(custom partitioner is
involved but i want to write same pair for more number times)
i want it go to two reducer.
for example i have partioning attribute two dimensional
<x1,x2>
singatue reduce
<0,0> 0
<0,1> 1
<1,0> 2
<1,1> 3
for <1,0> it will goto reducer
for <1,null> it should goto to reducer 2 and 3
for <0,null> it should goto reducer 0 and 1
On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <vi...@gmail.com>wrote:
> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>> Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Artem Onuchin <on...@gmail.com>.
Hello Vikas!
Well you can duplicate your pair in mapper for each reducer, add reducer's
mark to keys and write a partitioner that will partition records acording
this marks.
I mean something like this:
In mapper you produce (key_r1 value) (key_r2 value) instead (key value)
But i cannot imagine why you need that.
WBR, Onuchin Artem
2013/3/13 Viral Bajaria <vi...@gmail.com>
> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>> Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Vikas Jadhav <vi...@gmail.com>.
Hello I am not talking about custom partioner(custom partitioner is
involved but i want to write same pair for more number times)
i want it go to two reducer.
for example i have partioning attribute two dimensional
<x1,x2>
singatue reduce
<0,0> 0
<0,1> 1
<1,0> 2
<1,1> 3
for <1,0> it will goto reducer
for <1,null> it should goto to reducer 2 and 3
for <0,null> it should goto reducer 0 and 1
On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <vi...@gmail.com>wrote:
> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example we have pair (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause (key % numOfReducer) =
>> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>> Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Viral Bajaria <vi...@gmail.com>.
Do you want the pair to go to both reducers or do you want it to go to only
one but in a random fashion ?
AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
2nd is possible by just implementing your own partitioner which randomizes
where each key goes (not sure what you gain by that).
On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example we have pair (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
> Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Ajay Srivastava <Aj...@guavus.com>.
Emit (key, value) twice from mapper by modifying key as key' = (key, partId) and record becomes (key', value)
>From custom partitioner, send record to reducer based on partId. Ignore partId field in reducer.
Regards,
Ajay Srivastava
On 13-Mar-2013, at 2:29 PM, Vikas Jadhav wrote:
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) = (1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com>> wrote:
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>> wrote:
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one reducer
I have use case where i need to shufffle same (key,value) pair to multiple reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
Thanx and Regards
Vikas Jadhav
--
Don't Grow Old, Grow Up... :-)
--
Thanx and Regards
Vikas Jadhav
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Viral Bajaria <vi...@gmail.com>.
Do you want the pair to go to both reducers or do you want it to go to only
one but in a random fashion ?
AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
2nd is possible by just implementing your own partitioner which randomizes
where each key goes (not sure what you gain by that).
On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example we have pair (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
> Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Ajay Srivastava <Aj...@guavus.com>.
Emit (key, value) twice from mapper by modifying key as key' = (key, partId) and record becomes (key', value)
>From custom partitioner, send record to reducer based on partId. Ignore partId field in reducer.
Regards,
Ajay Srivastava
On 13-Mar-2013, at 2:29 PM, Vikas Jadhav wrote:
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) = (1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com>> wrote:
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>> wrote:
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one reducer
I have use case where i need to shufffle same (key,value) pair to multiple reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
Thanx and Regards
Vikas Jadhav
--
Don't Grow Old, Grow Up... :-)
--
Thanx and Regards
Vikas Jadhav
RE: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Samir Kumar Das Mohapatra <da...@adobe.com>.
Use can use Custom Partitioner for that same.
Regards,
Samir.
From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
Sent: 13 March 2013 14:29
To: user@hadoop.apache.org
Subject: Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) = (1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com>> wrote:
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>> wrote:
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one reducer
I have use case where i need to shufffle same (key,value) pair to multiple reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
Thanx and Regards
Vikas Jadhav
--
Don't Grow Old, Grow Up... :-)
--
Thanx and Regards
Vikas Jadhav
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Ajay Srivastava <Aj...@guavus.com>.
Emit (key, value) twice from mapper by modifying key as key' = (key, partId) and record becomes (key', value)
>From custom partitioner, send record to reducer based on partId. Ignore partId field in reducer.
Regards,
Ajay Srivastava
On 13-Mar-2013, at 2:29 PM, Vikas Jadhav wrote:
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) = (1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com>> wrote:
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>> wrote:
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one reducer
I have use case where i need to shufffle same (key,value) pair to multiple reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
Thanx and Regards
Vikas Jadhav
--
Don't Grow Old, Grow Up... :-)
--
Thanx and Regards
Vikas Jadhav
RE: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Samir Kumar Das Mohapatra <da...@adobe.com>.
Use can use Custom Partitioner for that same.
Regards,
Samir.
From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
Sent: 13 March 2013 14:29
To: user@hadoop.apache.org
Subject: Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) = (1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com>> wrote:
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>> wrote:
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one reducer
I have use case where i need to shufffle same (key,value) pair to multiple reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
Thanx and Regards
Vikas Jadhav
--
Don't Grow Old, Grow Up... :-)
--
Thanx and Regards
Vikas Jadhav
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by samir das mohapatra <sa...@gmail.com>.
Use can use Custom Partitioner for that same.
Regards,
Samir.
On Wed, Mar 13, 2013 at 2:29 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example we have pair (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
> Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Viral Bajaria <vi...@gmail.com>.
Do you want the pair to go to both reducers or do you want it to go to only
one but in a random fashion ?
AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
2nd is possible by just implementing your own partitioner which randomizes
where each key goes (not sure what you gain by that).
On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example we have pair (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
> Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
RE: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Samir Kumar Das Mohapatra <da...@adobe.com>.
Use can use Custom Partitioner for that same.
Regards,
Samir.
From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
Sent: 13 March 2013 14:29
To: user@hadoop.apache.org
Subject: Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) = (1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com>> wrote:
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>> wrote:
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one reducer
I have use case where i need to shufffle same (key,value) pair to multiple reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
Thanx and Regards
Vikas Jadhav
--
Don't Grow Old, Grow Up... :-)
--
Thanx and Regards
Vikas Jadhav
RE: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Samir Kumar Das Mohapatra <da...@adobe.com>.
Use can use Custom Partitioner for that same.
Regards,
Samir.
From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
Sent: 13 March 2013 14:29
To: user@hadoop.apache.org
Subject: Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) = (1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com>> wrote:
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>> wrote:
Hello,
As by default Hadoop framework can shuffle (key,value) pair to only one reducer
I have use case where i need to shufffle same (key,value) pair to multiple reducers
Also I willing to change the code of hadoop framework if Necessory.
Thank you
--
Thanx and Regards
Vikas Jadhav
--
Don't Grow Old, Grow Up... :-)
--
Thanx and Regards
Vikas Jadhav
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Viral Bajaria <vi...@gmail.com>.
Do you want the pair to go to both reducers or do you want it to go to only
one but in a random fashion ?
AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
2nd is possible by just implementing your own partitioner which randomizes
where each key goes (not sure what you gain by that).
On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example we have pair (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
> Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by samir das mohapatra <sa...@gmail.com>.
Use can use Custom Partitioner for that same.
Regards,
Samir.
On Wed, Mar 13, 2013 at 2:29 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example we have pair (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause (key % numOfReducer) =
> (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
> Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Vikas Jadhav <vi...@gmail.com>.
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple
reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and
reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) =
(1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
> Hi
>
> you can use Job#setNumReduceTasks(int tasks) method to set the number of
> reducer to output.
>
>
> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Hello,
>>
>> As by default Hadoop framework can shuffle (key,value) pair to only one
>> reducer
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>>
>> Thank you
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Vikas Jadhav <vi...@gmail.com>.
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple
reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and
reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) =
(1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
> Hi
>
> you can use Job#setNumReduceTasks(int tasks) method to set the number of
> reducer to output.
>
>
> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Hello,
>>
>> As by default Hadoop framework can shuffle (key,value) pair to only one
>> reducer
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>>
>> Thank you
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Vikas Jadhav <vi...@gmail.com>.
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple
reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and
reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) =
(1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
> Hi
>
> you can use Job#setNumReduceTasks(int tasks) method to set the number of
> reducer to output.
>
>
> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Hello,
>>
>> As by default Hadoop framework can shuffle (key,value) pair to only one
>> reducer
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>>
>> Thank you
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by Vikas Jadhav <vi...@gmail.com>.
Hi
I am specifying requirement again with example.
I have use case where i need to shufffle same (key,value) pair to multiple
reducers
For Example we have pair (1,"ABC") and two reducers (reducer0 and
reducer1) are there then
by default this pair will go to reduce1 (cause (key % numOfReducer) =
(1%2) )
how i should shuffle this pair to both reducer.
Also I willing to change the code of hadoop framework if Necessory.
Thank you
On Wed, Mar 13, 2013 at 12:51 PM, feng lu <am...@gmail.com> wrote:
> Hi
>
> you can use Job#setNumReduceTasks(int tasks) method to set the number of
> reducer to output.
>
>
> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Hello,
>>
>> As by default Hadoop framework can shuffle (key,value) pair to only one
>> reducer
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>>
>> Thank you
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by feng lu <am...@gmail.com>.
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of
reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
>
> As by default Hadoop framework can shuffle (key,value) pair to only one
> reducer
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
> Also I willing to change the code of hadoop framework if Necessory.
>
>
> Thank you
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Don't Grow Old, Grow Up... :-)
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by feng lu <am...@gmail.com>.
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of
reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
>
> As by default Hadoop framework can shuffle (key,value) pair to only one
> reducer
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
> Also I willing to change the code of hadoop framework if Necessory.
>
>
> Thank you
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Don't Grow Old, Grow Up... :-)
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by feng lu <am...@gmail.com>.
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of
reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
>
> As by default Hadoop framework can shuffle (key,value) pair to only one
> reducer
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
> Also I willing to change the code of hadoop framework if Necessory.
>
>
> Thank you
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Don't Grow Old, Grow Up... :-)
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Posted by feng lu <am...@gmail.com>.
Hi
you can use Job#setNumReduceTasks(int tasks) method to set the number of
reducer to output.
On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
>
> As by default Hadoop framework can shuffle (key,value) pair to only one
> reducer
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
> Also I willing to change the code of hadoop framework if Necessory.
>
>
> Thank you
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Don't Grow Old, Grow Up... :-)