[jira] [Commented] (AMBARI-21823) Spark2 Shuffle Configuration Property Not Configure After Upgrade From BigInsights

["Jonathan Hurley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMBARI-21823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142985#comment-16142985 ] 

Jonathan Hurley commented on AMBARI-21823:
------------------------------------------

The migration from IOP to HDP is correct - we're setting the Spark1 properties. Looks like when adding Spark2, the stack advisor is not suggestion adding the spark2_shuffle. This goes back to the default behavior of Ambari which is to always add both {{spark_shuffle}} and {{spark2_shuffle}} properties when adding either Spark1 or Spark2. 

This was never an issue previously since we were always adding both aux-services. We can fix this in one of two ways:
- On upgrade, just add both (like we're doing for the 4.2.5 migration)
- Fix stack advisor to correct these properties based on which Spark versions are installed



> Spark2 Shuffle Configuration Property Not Configure After Upgrade From BigInsights
> ----------------------------------------------------------------------------------
>
>                 Key: AMBARI-21823
>                 URL: https://issues.apache.org/jira/browse/AMBARI-21823
>             Project: Ambari
>          Issue Type: Bug
>    Affects Versions: 2.5.2
>            Reporter: Yesha Vora
>            Assignee: Jonathan Hurley
>
> Scenario:
> * Upgrade ambari from Ambari 2.2.2.0 to 2.5.2.0
> * Upgrade BI 420 from HDP 2.6.2.0
> In this upgrade scenario, Spark1 was upgraded from BI->HDP
> Spark2 was added after upgrade.
> In this cluster , it has Spark and Spark2 both. 
> However, spark2 shuffle is not added to yarn.nodemanager.aux-services.
> {code}
>  <property>
>       <name>yarn.nodemanager.aux-services</name>
>       <value>mapreduce_shuffle,spark_shuffle</value>
>     </property>{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)