[jira] [Created] (HIVE-24707) Apply Sane Default for Tez Containers as Last Resort

["David Mollitor (Jira)" <ji...@apache.org>]

David Mollitor created HIVE-24707:
-------------------------------------

             Summary: Apply Sane Default for Tez Containers as Last Resort
                 Key: HIVE-24707
                 URL: https://issues.apache.org/jira/browse/HIVE-24707
             Project: Hive
          Issue Type: Improvement
            Reporter: David Mollitor


{code:java|title=DagUtils.java}
public static Resource getContainerResource(Configuration conf) {
    int memory = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCONTAINERSIZE) > 0 ?
      HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCONTAINERSIZE) :
      conf.getInt(MRJobConfig.MAP_MEMORY_MB, MRJobConfig.DEFAULT_MAP_MEMORY_MB);
    int cpus = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCPUVCORES) > 0 ?
      HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCPUVCORES) :
      conf.getInt(MRJobConfig.MAP_CPU_VCORES, MRJobConfig.DEFAULT_MAP_CPU_VCORES);
    return Resource.newInstance(memory, cpus);
  }
{code}

If Tez Container Size or VCores is an invalid value ( <= 0 ) then it falls back onto the MapReduce configurations, but if the MapReduce configurations have invalid values ( <= 0 ), they are excepted regardless and this will cause failures down the road.

This code should also check the MapReduce values and fall back to MapReduce default values if they are <= 0.

Also, some logging would be nice here too, reporting about where the configuration values came from.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)