You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Pankil Doshi <fo...@gmail.com> on 2009/09/26 02:48:47 UTC
Best Idea to deal with following situation
Hello everyone,
I have job whose result has only 5 keys but but each key has long list of
values like in 100000's .
What should be best way to deal with it. I feel few of my reducers get over
loaded as two or more keys go to same reduce and hence they have lots of
work to do.
So what should be best way out with this situation?
Pankil
RE: Best Idea to deal with following situation
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
Along with partitioner, try to plug in a combiner. It would provide significant performance gains. Not sure about the algo you use, but might have to tweak that a little to facilitate a combiner.
Thanks,
Amogh
-----Original Message-----
From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]
Sent: Tuesday, September 29, 2009 12:25 PM
To: common-user@hadoop.apache.org
Cc: core-user@hadoop.apache.org
Subject: Re: Best Idea to deal with following situation
you can write your custom partitioner instead of hash partitioner
On Sat, Sep 26, 2009 at 6:18 AM, Pankil Doshi <fo...@gmail.com> wrote:
> Hello everyone,
>
> I have job whose result has only 5 keys but but each key has long list of
> values like in 100000's .
> What should be best way to deal with it. I feel few of my reducers get over
> loaded as two or more keys go to same reduce and hence they have lots of
> work to do.
>
> So what should be best way out with this situation?
>
> Pankil
>
--
Thanks & Regards,
Chandra Prakash Bhagtani,
Re: Best Idea to deal with following situation
Posted by Chandraprakash Bhagtani <cp...@gmail.com>.
you can write your custom partitioner instead of hash partitioner
On Sat, Sep 26, 2009 at 6:18 AM, Pankil Doshi <fo...@gmail.com> wrote:
> Hello everyone,
>
> I have job whose result has only 5 keys but but each key has long list of
> values like in 100000's .
> What should be best way to deal with it. I feel few of my reducers get over
> loaded as two or more keys go to same reduce and hence they have lots of
> work to do.
>
> So what should be best way out with this situation?
>
> Pankil
>
--
Thanks & Regards,
Chandra Prakash Bhagtani,