[jira] [Commented] (TEZ-3676) Desired number of tasks in TezGroupedSplitsInputFormat could be negative

["Gopal V (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947988#comment-15947988 ] 

Gopal V commented on TEZ-3676:
------------------------------

This looks like a repeat of HIVE-12957

> Desired number of tasks in TezGroupedSplitsInputFormat could be negative
> ------------------------------------------------------------------------
>
>                 Key: TEZ-3676
>                 URL: https://issues.apache.org/jira/browse/TEZ-3676
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Ming Ma
>
> RM sometimes returned negative headroom value to Tez AM, which then generated negative number of tasks, which will cause the job to fail. We need to look into why FairScheduler will return negative headroom, but it will be useful to make Tez more robust against this. For example TezGroupedSplitsInputFormat can use the value from the original input format in such scenario.
> {noformat}
> , java.lang.IllegalArgumentException
>   at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
>   at org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat.setDesiredNumberOfSplits(TezGroupedSplitsInputFormat.java:72)
>   at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateNewSplits(MRInputHelpers.java:442)
>   at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:329)
>   at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
>   at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
>   at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)