[jira] [Commented] (HIVE-24707) Apply Sane Default for Tez Containers as Last Resort

["David Mollitor (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-24707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275917#comment-17275917 ] 

David Mollitor commented on HIVE-24707:
---------------------------------------

Also need to have this logic in one place and use throughout.  Like here:

https://github.com/apache/hive/blob/4446414f4478091db1eb20bc782a5c1825356153/ql/src/java/org/apache/hadoop/hive/ql/exec/MemoryInfo.java#L58-L64


> Apply Sane Default for Tez Containers as Last Resort
> ----------------------------------------------------
>
>                 Key: HIVE-24707
>                 URL: https://issues.apache.org/jira/browse/HIVE-24707
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: David Mollitor
>            Priority: Trivial
>
> {code:java|title=DagUtils.java}
> public static Resource getContainerResource(Configuration conf) {
>     int memory = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCONTAINERSIZE) > 0 ?
>       HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCONTAINERSIZE) :
>       conf.getInt(MRJobConfig.MAP_MEMORY_MB, MRJobConfig.DEFAULT_MAP_MEMORY_MB);
>     int cpus = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCPUVCORES) > 0 ?
>       HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCPUVCORES) :
>       conf.getInt(MRJobConfig.MAP_CPU_VCORES, MRJobConfig.DEFAULT_MAP_CPU_VCORES);
>     return Resource.newInstance(memory, cpus);
>   }
> {code}
> If Tez Container Size or VCores is an invalid value ( <= 0 ) then it falls back onto the MapReduce configurations, but if the MapReduce configurations have invalid values ( <= 0 ), they are excepted regardless and this will cause failures down the road.
> This code should also check the MapReduce values and fall back to MapReduce default values if they are <= 0.
> Also, some logging would be nice here too, reporting about where the configuration values came from.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)