You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Mridul Muralidharan <mr...@yahoo-inc.com> on 2008/12/03 03:17:45 UTC
Re: use of reduce parallelism in queries
Not sure about types branch, but from what I remember, in trunk I used
to put the PARALLEL after the generate.
The nested statements within the foreach are operating on a per tuple
basis ... so parallel might not make sense there anyway imo.
Regards,
Mridul
Kevin Weil wrote:
> In the types branch, should nested statements be able to be parallelized? I
> can do
>
> B = DISTINCT A PARALLEL 20;
>
> for example, but if I have a nested statement:
>
> B = GROUP A BY whatever;
> C = FOREACH B {
> statement1;
> statement2;
> GENERATE ...
> } PARALLEL 20;
>
> the PigParser barfs when reading that PARALLEL statement. I can't put
> PARALLEL after any of the nested statements without the same error.
>
> Also, in general, it seems like there should be a way to specify reduce
> parallelism at least at a script/job level rather than a statement level.
> Or am I overlooking something?
>
> Thank you,
> Kevin
>