v0.20.203: Do we need to set the number of reducers everytime?

[Piyush Kansal <pi...@gmail.com>]

Hi,

I am using Partitioner and Grouper class in my program. And lets say the
data which I want to process using MapReduce varies in size, it can be just
10MB or can go up to 10GB.
So, do I need to set the number of reducers depending on data size, every
time I want to run MapReduce. Or the Hadoop framework automatically invokes
the required number of reducers at run time?

-- 
Regards,
Piyush Kansal

Re: v0.20.203: Do we need to set the number of reducers everytime?

[Harsh J <ha...@cloudera.com>]

Piyush,

On Sat, Apr 7, 2012 at 3:02 AM, Piyush Kansal <pi...@gmail.com> wrote:
> So, do I need to set the number of reducers depending on data size, every
> time I want to run MapReduce. Or the Hadoop framework automatically invokes
> the required number of reducers at run time?

Plain Apache Hadoop does not auto-set # of reducers. You need to set
it manually.
However, tools like Pig and Hive have determining ability for the jobs
that are planned and launched, and set # of reducers based on data for
you.

-- 
Harsh J