[jira] [Created] (TEZ-1793) Setting different container sizes for jobs in the same tez session can cause cluster underutilization

["Rajesh Balamohan (JIRA)" <ji...@apache.org>]

Rajesh Balamohan created TEZ-1793:
-------------------------------------

             Summary: Setting different container sizes for jobs in the same tez session can cause cluster underutilization
                 Key: TEZ-1793
                 URL: https://issues.apache.org/jira/browse/TEZ-1793
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Rajesh Balamohan


set tez.am.container.idle.release-timeout-min.millis to 200000
set tez.am.container.idle.release-timeout-max.millis to 500000
set hive.tez.container.size to 4096 (similar to tez.task.resource.memory.mb)

- Ran a hive query (ie with container size 4096) in tez mode.  In the small cluster (with 720 GB total memory), this launched 180 containers.
- Reset container size to 8192 and rerun the query.  Now, it launched 90 containers.
- Reset container size to 4096 and rerun the query.  Now, it reuses the same 90 containers.

This could lead to cluster underutilization where a large job's settings can cause adverse impact (e.g concurrency) on the subsequent small jobs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)