[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild

["Karthik Kambatla (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054432#comment-14054432 ] 

Karthik Kambatla commented on MAPREDUCE-5875:
---------------------------------------------

Looks close. Couple of comments.
# Instead of doing string manipulation inline, can we add a helper method preferably where we are constructing these History URLs.
{code}
      final Path jobConfPath = new Path(jobFile.getParent(),  jobDetails[0]
          + "_" + jobDetails[1] + "_" + jobDetails[2] + "_conf.xml");
      Limits.reset(new JobConf(conf).loadFile(fs, jobConfPath));
{code}
# Nit: I am okay with not addressing this. We should probably skip isEnabled checks for INFO and WARN levels as the code doesn't have it anywhere else. 

> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5875
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster, client, task
>    Affects Versions: 2.4.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch, MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch, MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized asymmetrically: on the client side, and on the AM, job.xml is ignored whereas it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max counters/groups is only increased when needed. With the current Limits implementation relying on static constants, it's going to be challenging for tools that submit jobs concurrently  without resorting to class loading isolation.
> The patch that I am uploading is not perfect but demonstrates the issue. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)