Session max held containers

[Thaddeus Diamond <th...@gmail.com>]

I have specified the following:

      tezConfiguration.setBoolean(TEZ_AM_SESSION_MODE, true);
      tezConfiguration.setLong(TEZ_SESSION_AM_DAG_SUBMIT_TIMEOUT_SECS, 30 *
60);
      tezConfiguration.setLong(TEZ_AM_SESSION_MIN_HELD_CONTAINERS, 3);

tezConfiguration.setLong(TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS,
10 * 60 * 1000);

tezConfiguration.setLong(TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS, 15
* 60 * 1000);

For my session.  On a cluster with 220 containers when I launch a Tez job
that uses all 220 containers it holds onto all 220 containers after it's
done.  Is there a way to specify max held containers?  I want to retain
maybe like 10% of those used.

- Thad

RE: Session max held containers

[Bikas Saha <bi...@hortonworks.com>]

The containers are expected to expire (when idle) between min and max idle
timeouts. They will decay randomly between the min and max.

Looks like your time-outs are very high.  1 min and 10 min. (I am guessing
you have a typo in your email with min and max reversed).



What you want is being specified by min held containers. Do you have some
spurious tez-site that is misconfiguring the idle timeouts? If min timeout
is -1 then containers will be held forever.



Look for this log that will tell you the actual values being used.



    *LOG*.info("TaskScheduler initialized with configuration: " +

            "maxRMHeartbeatInterval: " + heartbeatIntervalMax +

            ", containerReuseEnabled: " + shouldReuseContainers +

            ", reuseRackLocal: " + reuseRackLocal +

            ", reuseNonLocal: " + reuseNonLocal +

            ", localitySchedulingDelay: " + localitySchedulingDelay +

            ", idleContainerMinTimeout=" + idleContainerTimeoutMin +

            ", idleContainerMaxTimeout=" + idleContainerTimeoutMax +

            ", sessionMinHeldContainers=" + sessionNumMinHeldContainers);



*From:* Thaddeus Diamond [mailto:thaddeus.diamond@gmail.com]
*Sent:* Thursday, August 14, 2014 9:04 AM
*To:* user@tez.apache.org
*Subject:* Session max held containers



I have specified the following:



      tezConfiguration.setBoolean(TEZ_AM_SESSION_MODE, true);

      tezConfiguration.setLong(TEZ_SESSION_AM_DAG_SUBMIT_TIMEOUT_SECS, 30 *
60);

      tezConfiguration.setLong(TEZ_AM_SESSION_MIN_HELD_CONTAINERS, 3);


tezConfiguration.setLong(TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS,
10 * 60 * 1000);


tezConfiguration.setLong(TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS, 15
* 60 * 1000);



For my session.  On a cluster with 220 containers when I launch a Tez job
that uses all 220 containers it holds onto all 220 containers after it's
done.  Is there a way to specify max held containers?  I want to retain
maybe like 10% of those used.



- Thad

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.