You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Swapan Shridhar <ss...@hortonworks.com> on 2017/02/28 23:07:31 UTC
Review Request 57128: AMBARI-20225. Take care of
hive-interactive-site's
'hive.tez.container.size' config during HDP upgrade from 2.5 to 2.6 and
update default value for tez-interactive-site's config
'tez.am.resource.memory.mb'.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57128/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
Bugs: AMBARI-20225
https://issues.apache.org/jira/browse/AMBARI-20225
Repository: ambari
Description
-------
**Background :**
**----------**
We are dealing with 2 specific configs here.
**(1).** 'tez.am.resource.memory.mb' for tez-interactive-site and
**(2).** 'hive.tez.container.size' for hive-interactive-site.
- 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4 onwards and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
- 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4. Starting from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with a default value of 'SET_ON_FIRST_INVOCATION'.
The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack Advisor (SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive config), so that config values for both can be calculated and set for the 1st time. Subsequesnt calls in the lifecycle, will just read the value set by SA or if later changed by user manually.
**Problem :**
**---------**
'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario.
**Fix:**
**----**
Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of configs during HDP upgrade.
**Possible scenarios:**
**------------------**
**1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py code would have been updated with new LLAP GA logic.**
- 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive (HSI) is ON or OFF after Ambari upgrade.
- If HSI if OFF before the upgrade was done, nothing to do over there.
- If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic has been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist in hive-interactive-site at this point. It will show up in hive-interative-site aftre the HDP upgrade to 2.6
**2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6**
- 'tez.am.resource.memory.mb' will already be there
- As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got added, having current value picked from hive-site/hive.tez.container.size. SA code carries a calculation logic for config 'hive.tez.container.size', which is bound to yield different value compared to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated (1). when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations).
**3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
- Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
**Note** that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb' is caluclated at below conditions:
(1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations, or
(2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)
Diffs
-----
ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/service_advisor.py e1e03c1
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml 8008bef
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml 1c5117e
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 6f3dfa7
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml da334be
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml d396033
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml c01d11e
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py cf1c0ee
Diff: https://reviews.apache.org/r/57128/diff/
Testing
-------
Manually tested on cluster.
Thanks,
Swapan Shridhar
Re: Review Request 57128: AMBARI-20225. Take care of
hive-interactive-site's
'hive.tez.container.size' config during HDP upgrade from 2.5 to 2.6 and
update default value for tez-interactive-site's config
'tez.am.resource.memory.mb'.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57128/#review167176
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Feb. 28, 2017, 11:07 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57128/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2017, 11:07 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-20225
> https://issues.apache.org/jira/browse/AMBARI-20225
>
>
> Repository: ambari
>
>
> Description
> -------
>
> **Background :**
> **----------**
>
> We are dealing with 2 specific configs here.
> **(1).** 'tez.am.resource.memory.mb' for tez-interactive-site and
> **(2).** 'hive.tez.container.size' for hive-interactive-site.
>
> - 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4 onwards and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
> - 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4. Starting from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with a default value of 'SET_ON_FIRST_INVOCATION'.
>
> The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack Advisor (SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive config), so that config values for both can be calculated and set for the 1st time. Subsequesnt calls in the lifecycle, will just read the value set by SA or if later changed by user manually.
>
> **Problem :**
> **---------**
>
> 'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario.
>
> **Fix:**
> **----**
>
> Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of configs during HDP upgrade.
>
> **Possible scenarios:**
> **------------------**
>
> **1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py code would have been updated with new LLAP GA logic.**
> - 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive (HSI) is ON or OFF after Ambari upgrade.
> - If HSI if OFF before the upgrade was done, nothing to do over there.
> - If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic has been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist in hive-interactive-site at this point. It will show up in hive-interative-site aftre the HDP upgrade to 2.6
>
>
> **2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6**
> - 'tez.am.resource.memory.mb' will already be there
> - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got added, having current value picked from hive-site/hive.tez.container.size. SA code carries a calculation logic for config 'hive.tez.container.size', which is bound to yield different value compared to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated (1). when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations).
>
> **3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
> - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
>
>
> **Note** that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb' is caluclated at below conditions:
>
> (1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations, or
> (2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/service_advisor.py e1e03c1
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml 8008bef
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml 1c5117e
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 6f3dfa7
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml da334be
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml d396033
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml c01d11e
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py cf1c0ee
>
> Diff: https://reviews.apache.org/r/57128/diff/
>
>
> Testing
> -------
>
> Manually tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 57128: AMBARI-20225. Take care of
hive-interactive-site's
'hive.tez.container.size' config during HDP upgrade from 2.5 to 2.6 and
update default value for tez-interactive-site's config
'tez.am.resource.memory.mb'.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57128/#review167180
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On Feb. 28, 2017, 11:07 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57128/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2017, 11:07 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-20225
> https://issues.apache.org/jira/browse/AMBARI-20225
>
>
> Repository: ambari
>
>
> Description
> -------
>
> **Background :**
> **----------**
>
> We are dealing with 2 specific configs here.
> **(1).** 'tez.am.resource.memory.mb' for tez-interactive-site and
> **(2).** 'hive.tez.container.size' for hive-interactive-site.
>
> - 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4 onwards and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
> - 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4. Starting from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with a default value of 'SET_ON_FIRST_INVOCATION'.
>
> The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack Advisor (SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive config), so that config values for both can be calculated and set for the 1st time. Subsequesnt calls in the lifecycle, will just read the value set by SA or if later changed by user manually.
>
> **Problem :**
> **---------**
>
> 'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario.
>
> **Fix:**
> **----**
>
> Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of configs during HDP upgrade.
>
> **Possible scenarios:**
> **------------------**
>
> **1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py code would have been updated with new LLAP GA logic.**
> - 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive (HSI) is ON or OFF after Ambari upgrade.
> - If HSI if OFF before the upgrade was done, nothing to do over there.
> - If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic has been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist in hive-interactive-site at this point. It will show up in hive-interative-site aftre the HDP upgrade to 2.6
>
>
> **2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6**
> - 'tez.am.resource.memory.mb' will already be there
> - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got added, having current value picked from hive-site/hive.tez.container.size. SA code carries a calculation logic for config 'hive.tez.container.size', which is bound to yield different value compared to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated (1). when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations).
>
> **3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
> - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
>
>
> **Note** that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb' is caluclated at below conditions:
>
> (1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations, or
> (2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/service_advisor.py e1e03c1
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml 8008bef
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml 1c5117e
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 6f3dfa7
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml da334be
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml d396033
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml c01d11e
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py cf1c0ee
>
> Diff: https://reviews.apache.org/r/57128/diff/
>
>
> Testing
> -------
>
> Manually tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 57128: AMBARI-20225. Take care of
hive-interactive-site's
'hive.tez.container.size' config during HDP upgrade from 2.5 to 2.6 and
update default value for tez-interactive-site's config
'tez.am.resource.memory.mb'.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57128/#review167175
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 1208)
<https://reviews.apache.org/r/57128/#comment239345>
Is this TODO a bug?
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 1312)
<https://reviews.apache.org/r/57128/#comment239346>
Is this TODO a bug?
- Sumit Mohanty
On Feb. 28, 2017, 11:07 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57128/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2017, 11:07 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-20225
> https://issues.apache.org/jira/browse/AMBARI-20225
>
>
> Repository: ambari
>
>
> Description
> -------
>
> **Background :**
> **----------**
>
> We are dealing with 2 specific configs here.
> **(1).** 'tez.am.resource.memory.mb' for tez-interactive-site and
> **(2).** 'hive.tez.container.size' for hive-interactive-site.
>
> - 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4 onwards and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
> - 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4. Starting from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with a default value of 'SET_ON_FIRST_INVOCATION'.
>
> The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack Advisor (SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive config), so that config values for both can be calculated and set for the 1st time. Subsequesnt calls in the lifecycle, will just read the value set by SA or if later changed by user manually.
>
> **Problem :**
> **---------**
>
> 'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario.
>
> **Fix:**
> **----**
>
> Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of configs during HDP upgrade.
>
> **Possible scenarios:**
> **------------------**
>
> **1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py code would have been updated with new LLAP GA logic.**
> - 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive (HSI) is ON or OFF after Ambari upgrade.
> - If HSI if OFF before the upgrade was done, nothing to do over there.
> - If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic has been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist in hive-interactive-site at this point. It will show up in hive-interative-site aftre the HDP upgrade to 2.6
>
>
> **2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6**
> - 'tez.am.resource.memory.mb' will already be there
> - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got added, having current value picked from hive-site/hive.tez.container.size. SA code carries a calculation logic for config 'hive.tez.container.size', which is bound to yield different value compared to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated (1). when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations).
>
> **3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
> - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
>
>
> **Note** that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb' is caluclated at below conditions:
>
> (1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations, or
> (2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/service_advisor.py e1e03c1
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml 8008bef
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml 1c5117e
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 6f3dfa7
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml da334be
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml d396033
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml c01d11e
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py cf1c0ee
>
> Diff: https://reviews.apache.org/r/57128/diff/
>
>
> Testing
> -------
>
> Manually tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>