You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Mridul Muralidharan <mr...@yahoo-inc.com> on 2008/12/03 03:17:45 UTC

Re: use of reduce parallelism in queries

Not sure about types branch, but from what I remember, in trunk I used 
to put the PARALLEL after the generate.

The nested statements within the foreach are operating on a per tuple 
basis ... so parallel might not make sense there anyway imo.


Regards,
Mridul

Kevin Weil wrote:
> In the types branch, should nested statements be able to be parallelized?  I
> can do
> 
> B = DISTINCT A PARALLEL 20;
> 
> for example, but if I have a nested statement:
> 
> B = GROUP A BY whatever;
> C = FOREACH B {
>    statement1;
>    statement2;
>    GENERATE ...
> } PARALLEL 20;
> 
> the PigParser barfs when reading that PARALLEL statement.  I can't put
> PARALLEL after any of the nested statements without the same error.
> 
> Also, in general, it seems like there should be a way to specify reduce
> parallelism at least at a script/job level rather than a statement level.
> Or am I overlooking something?
> 
> Thank you,
> Kevin
>