You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by "Jie Li (JIRA)" <ji...@apache.org> on 2012/06/29 19:00:44 UTC

[jira] [Updated] (PIG-2779) Refactoring the code for setting number of reducers

     [ https://issues.apache.org/jira/browse/PIG-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jie Li updated PIG-2779:
------------------------

    Affects Version/s: 0.11
        Fix Version/s: 0.11
    
> Refactoring the code for setting number of reducers
> ---------------------------------------------------
>
>                 Key: PIG-2779
>                 URL: https://issues.apache.org/jira/browse/PIG-2779
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Jie Li
>             Fix For: 0.11
>
>
> As PIG-2652 observed, currently the code for setting number of reducers is a little messy. MapReduceOper.requestedParallelism seems being misused in some plases, and now we support runtime estimation of #reducer which further complicates the problem.
> For example, if we specify parallel 1 for the order-by, the estimated #reducer will be used. If we specify parallel 2 while it estimates 4, order-by will fail due to "Illegal partition for Null". If we specify parallel 4 while it estimates 2, then some reducers will have nothing to do. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira