[jira] [Updated] (PIG-2208) Restrict number of PIG generated Haddop counters

["Daniel Dai (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2208:
----------------------------

       Resolution: Fixed
    Fix Version/s: 0.10
     Hadoop Flags: Reviewed
           Status: Resolved  (was: Patch Available)

Thanks Dmitriy!

All unit tests pass.

Test-patch result:
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     -1 release audit.  The applied patch generated 464 release audit warnings (more than the trunk's current 462 warnings).

No new files added, ignore release audit warnings.

Patch committed to both trunk and 0.9 branch.
                
> Restrict number of PIG generated Haddop counters 
> -------------------------------------------------
>
>                 Key: PIG-2208
>                 URL: https://issues.apache.org/jira/browse/PIG-2208
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.8.1, 0.9.0
>            Reporter: Richard Ding
>            Assignee: Richard Ding
>             Fix For: 0.9.1, 0.10
>
>         Attachments: PIG-2208.patch
>
>
> PIG 8.0 implemented Hadoop counters to track the number of records read for each input and the number of records written for each output (PIG-1389 & PIG-1299). On the other hand, Hadoop has imposed limit on per job counters (MAPREDUCE-1943) and jobs will fail if the counters exceed the limit.
> Therefore we need a way to cap the number of PIG generated counters.
> Here are the two options:
> 1. Add a integer property (e.g., pig.counter.limit) to the pig property file (e.g., 20). If the number of inputs of a job exceeds this number, the input counters are disabled. Similarly, if the number of outputs of a job exceeds this number, the output counters are disabled.
> 2. Add a boolean property (e.g., pig.disable.counters) to the pig property file (default: false). If this property is set to true, then the PIG generated counters are disabled.
>   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira