You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/08/27 02:38:35 UTC
Review Request 61930: Spark2 Shuffle Configuration Property Not
Configure After Upgrade From BigInsights
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61930/
-----------------------------------------------------------
Review request for Ambari and Mahadev Konar.
Bugs: AMBARI-21823
https://issues.apache.org/jira/browse/AMBARI-21823
Repository: ambari
Description
-------
Scenario:
* Upgrade ambari from Ambari 2.2.2.0 to 2.5.2.0
* Upgrade BI 420 from HDP 2.6.2.0
*
In this upgrade scenario, Spark1 was upgraded from BI->HDP
Spark2 was added after upgrade.
In this cluster , it has Spark and Spark2 both.
However, spark2 shuffle is not added to yarn.nodemanager.aux-services.
{code}
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,spark_shuffle</value>
</property>{code}
The migration from IOP to HDP is correct - we're setting the Spark1 properties. Looks like when adding Spark2, the stack advisor is not suggestion adding the spark2_shuffle. This goes back to the default behavior of Ambari which is to always add both {{spark_shuffle}} and {{spark2_shuffle}} properties when adding either Spark1 or Spark2.
This was never an issue previously since we were always adding both aux-services. We can fix this in one of two ways:
- On upgrade, just add both (like we're doing for the 4.2.5 migration)
- Fix stack advisor to correct these properties based on which Spark versions are installed
Diffs
-----
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml bdb9ed3b58
Diff: https://reviews.apache.org/r/61930/diff/1/
Testing
-------
IOP Upgrade With Spark1 to HDP 2.6
Thanks,
Jonathan Hurley
Re: Review Request 61930: Spark2 Shuffle Configuration Property Not
Configure After Upgrade From BigInsights
Posted by Mahadev Konar <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61930/#review183898
-----------------------------------------------------------
Ship it!
Ship It!
- Mahadev Konar
On Aug. 27, 2017, 2:38 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61930/
> -----------------------------------------------------------
>
> (Updated Aug. 27, 2017, 2:38 a.m.)
>
>
> Review request for Ambari and Mahadev Konar.
>
>
> Bugs: AMBARI-21823
> https://issues.apache.org/jira/browse/AMBARI-21823
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Scenario:
> * Upgrade ambari from Ambari 2.2.2.0 to 2.5.2.0
> * Upgrade BI 420 from HDP 2.6.2.0
> *
> In this upgrade scenario, Spark1 was upgraded from BI->HDP
> Spark2 was added after upgrade.
> In this cluster , it has Spark and Spark2 both.
> However, spark2 shuffle is not added to yarn.nodemanager.aux-services.
> {code}
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle,spark_shuffle</value>
> </property>{code}
>
> The migration from IOP to HDP is correct - we're setting the Spark1 properties. Looks like when adding Spark2, the stack advisor is not suggestion adding the spark2_shuffle. This goes back to the default behavior of Ambari which is to always add both {{spark_shuffle}} and {{spark2_shuffle}} properties when adding either Spark1 or Spark2.
>
> This was never an issue previously since we were always adding both aux-services. We can fix this in one of two ways:
> - On upgrade, just add both (like we're doing for the 4.2.5 migration)
> - Fix stack advisor to correct these properties based on which Spark versions are installed
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml bdb9ed3b58
>
>
> Diff: https://reviews.apache.org/r/61930/diff/1/
>
>
> Testing
> -------
>
> IOP Upgrade With Spark1 to HDP 2.6
>
>
> Thanks,
>
> Jonathan Hurley
>
>