[jira] [Created] (TEZ-1086) Allow Tez sessions to decay smoothly instead of timeouts

["Gopal V (JIRA)" <ji...@apache.org>]

Gopal V created TEZ-1086:
----------------------------

             Summary: Allow Tez sessions to decay smoothly instead of timeouts
                 Key: TEZ-1086
                 URL: https://issues.apache.org/jira/browse/TEZ-1086
             Project: Apache Tez
          Issue Type: New Feature
            Reporter: Gopal V


With hiveserver2's session sharing, we hit a problem with session container delays.

If the timeout is too small, like 2 seconds, there is no real reuse of containers, even for small queries which hit the server.

If the timeout is too big, like 2 minutes, then a big query will hold onto idle containers for way too long causing smaller queries to stall.

An ideal middle ground would be to keep some containers around, while getting rid of a majority of containers owned by the session.

A simple heuristic could be to set the fractional-life of a session container to 2 seconds.

If the fraction is 0.5 (i.e half-life), then 50% of the idle capacity would become available for the small query in 2 seconds.

This could work better for the JIT as the mechanism can sort containers by their launchtime and kill off the newer ones always (or pick 1 per node to keep).



--
This message was sent by Atlassian JIRA
(v6.2#6252)