[jira] [Created] (TEZ-3676) Desired number of tasks in TezGroupedSplitsInputFormat could be negative

["Ming Ma (JIRA)" <ji...@apache.org>]

Ming Ma created TEZ-3676:
----------------------------

             Summary: Desired number of tasks in TezGroupedSplitsInputFormat could be negative
                 Key: TEZ-3676
                 URL: https://issues.apache.org/jira/browse/TEZ-3676
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Ming Ma


RM sometimes returned negative headroom value to Tez AM, which then generated negative number of tasks, which will cause the job to fail. We need to look into why FairScheduler will return negative headroom, but it will be useful to make Tez more robust against this. For example TezGroupedSplitsInputFormat can use the value from the original input format in such scenario.

{noformat}
, java.lang.IllegalArgumentException
  at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
  at org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat.setDesiredNumberOfSplits(TezGroupedSplitsInputFormat.java:72)
  at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateNewSplits(MRInputHelpers.java:442)
  at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:329)
  at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
  at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
  at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)