[jira] [Commented] (AMBARI-20225) Take care of hive-interactive-site's 'hive.tez.container.size' config during HDP upgrade from 2.5 to 2.6 and update default value for tez-interactive-site's config 'tez.am.resource.memory.mb'.

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMBARI-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889302#comment-15889302 ] 

Hadoop QA commented on AMBARI-20225:
------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12855244/AMBARI-20225.patch
  against trunk revision .

    {color:red}-1 patch{color}.  Top-level trunk compilation may be broken.

Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/10811//console

This message is automatically generated.

> Take care of hive-interactive-site's 'hive.tez.container.size' config during HDP upgrade from 2.5 to 2.6 and update default value for tez-interactive-site's config 'tez.am.resource.memory.mb'.
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: AMBARI-20225
>                 URL: https://issues.apache.org/jira/browse/AMBARI-20225
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: trunk, 2.5.0
>            Reporter: Swapan Shridhar
>            Assignee: Swapan Shridhar
>             Fix For: trunk, 2.5.0
>
>         Attachments: AMBARI-20225.patch
>
>
> *Background :*
> We are dealing with 2 specific configs here. 
> *(1).* 'tez.am.resource.memory.mb' for tez-interactive-site and 
> *(2).* 'hive.tez.container.size' for hive-interactive-site. 
> - 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4 onwards and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
> - 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4. Starting from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with a default value of 'SET_ON_FIRST_INVOCATION'.
> The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack Advisor (SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive config), so that config values for both can be calculated and set for the 1st time. Subsequesnt calls in the lifecycle, will just read the value set by SA or if later changed by user manually.
> *Problem :*
> 'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario. 
> *Fix:*
> Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of configs during HDP upgrade.
>  
> *Possible scenarios:*
> *1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py code would have been updated with new LLAP GA logic.*
>    - 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive (HSI) is ON or OFF after Ambari upgrade. 
>       - If HSI if OFF before the upgrade was done, nothing to do over there.
>       - If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic has been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist in hive-interactive-site at this point. It will show up in hive-interative-site aftre the HDP upgrade to 2.6
> *2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6*
>    - 'tez.am.resource.memory.mb' will already be there
>    - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got added, having current value picked from hive-site/hive.tez.container.size. SA code carries a calculation logic for config 'hive.tez.container.size', which is bound to yield different value compared to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated (1). when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations).    
>  
> *3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6*
>    - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
> *Note* that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb' is caluclated at below conditions:
> (1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations, or
> (2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)