You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2015/12/11 00:56:11 UTC

[jira] [Commented] (PIG-4529) Pig on tez hit counter limit imposed by MR

    [ https://issues.apache.org/jira/browse/PIG-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051878#comment-15051878 ] 

Daniel Dai commented on PIG-4529:
---------------------------------

This won't work in Oozie since Limits is imposed already in Oozie job. Pig cannot change it. Actually the MR counter code is not needed. Open PIG-4760 to fix that.

> Pig on tez hit counter limit imposed by MR
> ------------------------------------------
>
>                 Key: PIG-4529
>                 URL: https://issues.apache.org/jira/browse/PIG-4529
>             Project: Pig
>          Issue Type: Bug
>          Components: tez
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.15.0
>
>         Attachments: PIG-4529-1.patch, PIG-4529-2.patch
>
>
> See the following stack in test:
> {code}
> org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 131 max=130
> 	at org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
> 	at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
> 	at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
> 	at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
> 	at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:87)
> 	at org.apache.hadoop.mapred.Counters$Group.addCounter(Counters.java:354)
> 	at org.apache.pig.tools.pigstats.tez.TezDAGStats.covertToHadoopCounters(TezDAGStats.java:276)
> 	at org.apache.pig.tools.pigstats.tez.TezDAGStats.accumulateStats(TezDAGStats.java:192)
> 	at org.apache.pig.tools.pigstats.tez.TezPigScriptStats.accumulateStats(TezPigScriptStats.java:181)
> 	at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:198)
> 	at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:174)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}
> Actually I've set tez.counters.max to a much higher value, TezLauncher should not be limited by MR limits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)