You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Jeremy Hanna <je...@gmail.com> on 2011/07/04 21:24:46 UTC
set default_parallel or let pig set it
According to http://pig.apache.org/docs/r0.8.1/cookbook.html#Use+the+Parallel+Features there are two ways for pig to determine the number of reducers to use:
1- set default_parallel and/or PARALLEL
2- let pig calculate it
What do people generally use right now? Is there a preferred option?
Re: set default_parallel or let pig set it
Posted by Alan Gates <ga...@hortonworks.com>.
Definitely set it yourself. Pig does a poor job of calculating it. The Pig calculation is only provided as a backstop when users forget (or don't yet know) to set it. At some point in the future we'd like to provide good default parallel settings, but we have a ways to go before we get there.
Alan.
On Jul 4, 2011, at 12:24 PM, Jeremy Hanna wrote:
> According to http://pig.apache.org/docs/r0.8.1/cookbook.html#Use+the+Parallel+Features there are two ways for pig to determine the number of reducers to use:
> 1- set default_parallel and/or PARALLEL
> 2- let pig calculate it
>
> What do people generally use right now? Is there a preferred option?