You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Sudharsan Sampath <su...@ericsson.com> on 2011/05/27 12:16:01 UTC

No of reducers

Hi,

Is there a reference on how the number of reducers required for a job is calculated?

Thanks
Sudharsan S


Re: No of reducers

Posted by Thejas M Nair <te...@yahoo-inc.com>.
In pig 0.8 the default number of reducers changed from 1 to a value computed based on input data size -
http://pig.apache.org/docs/r0.8.1/cookbook.html#Use+the+Parallel+Features

-Thejas


On 5/27/11 6:46 AM, "Jonathan Coveney" <jc...@gmail.com> wrote:

SET default_parallel X; will set the PARALLEL keyword for all parallel
functions (ie set the reducers for the job)

I am not sure how the default is calculated...for a while it was set to 1 I
believe, ostensibly to force people to set it to something more reasonable.

2011/5/27 Harsh J <ha...@cloudera.com>

> The PARALLEL keyword controls the number of reducers used in the job.
> If unspecified, a default number is applied. Is this what you're
> looking for?
>
> On Fri, May 27, 2011 at 3:46 PM, Sudharsan Sampath
> <su...@ericsson.com> wrote:
> >
> > Hi,
> >
> > Is there a reference on how the number of reducers required for a job is
> calculated?
> >
> > Thanks
> > Sudharsan S
> >
> >
>
>
>
> --
> Harsh J
>



--


Re: No of reducers

Posted by Jonathan Coveney <jc...@gmail.com>.
SET default_parallel X; will set the PARALLEL keyword for all parallel
functions (ie set the reducers for the job)

I am not sure how the default is calculated...for a while it was set to 1 I
believe, ostensibly to force people to set it to something more reasonable.

2011/5/27 Harsh J <ha...@cloudera.com>

> The PARALLEL keyword controls the number of reducers used in the job.
> If unspecified, a default number is applied. Is this what you're
> looking for?
>
> On Fri, May 27, 2011 at 3:46 PM, Sudharsan Sampath
> <su...@ericsson.com> wrote:
> >
> > Hi,
> >
> > Is there a reference on how the number of reducers required for a job is
> calculated?
> >
> > Thanks
> > Sudharsan S
> >
> >
>
>
>
> --
> Harsh J
>

Re: No of reducers

Posted by Harsh J <ha...@cloudera.com>.
The PARALLEL keyword controls the number of reducers used in the job.
If unspecified, a default number is applied. Is this what you're
looking for?

On Fri, May 27, 2011 at 3:46 PM, Sudharsan Sampath
<su...@ericsson.com> wrote:
>
> Hi,
>
> Is there a reference on how the number of reducers required for a job is calculated?
>
> Thanks
> Sudharsan S
>
>



-- 
Harsh J