[jira] [Created] (OOZIE-3604) Oozie Luncher & Map-Reduce Action Complete Successfully However Oozie Still Fails the Action

["houman babai (Jira)" <ji...@apache.org>]

houman babai created OOZIE-3604:
-----------------------------------

             Summary: Oozie Luncher & Map-Reduce Action Complete Successfully However Oozie Still Fails the Action
                 Key: OOZIE-3604
                 URL: https://issues.apache.org/jira/browse/OOZIE-3604
             Project: Oozie
          Issue Type: Bug
          Components: core
    Affects Versions: 5.1.0
         Environment: 5.1.0-cdh-6.3.1
5.1.0-cdh-6.3.2
            Reporter: houman babai


I am using oozie 5.1.0-cdh6.3.1

In my workflow I have a mapreduce action, which generates over 300 counters. The oozie launcher & the mapreduce job successfully complete, however, oozie reports that:
{code:java}
Error Code: LimitExceededException

LimitExceededException: Too many counters: 121 max=120
{code}
I have updated mapred-site.xml.

The log for the *mapreduce job* reports success, in fact I can see all the counters & the actual output of the mapreduce job on hdfs.

In the oozie *launcher log* I can see:
 * mapreduce.job.counters.max : 8192
 * mapreduce.job.counters.groups.max : 100

I set the oozie logs to TRACE, and I can see the following WARN:
{code:java}
WARN org.apache.oozie.command.wf.ActionEndXCommand: SERVER[SERVER-NAME] USER[MY-NAME] GROUP[-] TOKEN[] APP[APP-NAME] JOB[0000012-200714223028181-oozie-oozi-W] ACTION[0000012-200714223028181-oozie-oozi-W@ACTION-NAME] Error ending action [ACTION-NAME]. ErrorType [ERROR], ErrorCode [LimitExceededException], Message [LimitExceededException: Too many counters: 121 max=120]

WARN org.apache.oozie.command.wf.ActionEndXCommand: SERVER[SERVER-NAME] USER[MY-NAME] GROUP[-] TOKEN[] APP[APP-NAME] JOB[0000012-200714223028181-oozie-oozi-W] ACTION[0000012-200714223028181-oozie-oozi-W@ACTION-NAME] Setting Action Status to [ERROR]
{code}
Looking through the oozie source code, I think the issue may be traced to [line in MapReduceActionExecutor|https://github.com/apache/oozie/blob/a40ab5361372aa73c9e4aa386a9c81bd21742aa4/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java#L250]

I am guessing that for some reason, the context or the config passed into the [createJobClient|https://github.com/apache/oozie/blob/a40ab5361372aa73c9e4aa386a9c81bd21742aa4/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java#L242] lack the mapreduce.job.counters.max configuration.

I tried several other things hoping that it would somehow work:
 * I added the mapreduce.job.counters.max as configuration into the action (inside the workflow)
 * I added the mapreduce.job.counters.max to action config default.xml

Nothing has helped.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)