[jira] [Commented] (TEZ-1793) Setting different container sizes for jobs in the same tez session can cause cluster underutilization

["Jeff Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217878#comment-14217878 ] 

Jeff Zhang commented on TEZ-1793:
---------------------------------

I think this can been resolved once YARN support container resize. Not sure when YARN-1197 will be resolved.  

> Setting different container sizes for jobs in the same tez session can cause cluster underutilization
> -----------------------------------------------------------------------------------------------------
>
>                 Key: TEZ-1793
>                 URL: https://issues.apache.org/jira/browse/TEZ-1793
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rajesh Balamohan
>
> set tez.am.container.idle.release-timeout-min.millis to 200000
> set tez.am.container.idle.release-timeout-max.millis to 500000
> set hive.tez.container.size to 4096 (similar to tez.task.resource.memory.mb)
> - Ran a hive query (ie with container size 4096) in tez mode.  In the small cluster (with 720 GB total memory), this launched 180 containers.
> - Reset container size to 8192 and rerun the query.  Now, it launched 90 containers.
> - Reset container size to 4096 and rerun the query.  Now, it reuses the same 90 containers.
> This could lead to cluster underutilization where a large job's settings can cause adverse impact (e.g concurrency) on the subsequent small jobs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)