[jira] [Resolved] (SPARK-6662) Allow variable substitution in spark.yarn.historyServer.address

["Thomas Graves (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves resolved SPARK-6662.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 1.4.0
         Assignee: Cheolsoo Park

> Allow variable substitution in spark.yarn.historyServer.address
> ---------------------------------------------------------------
>
>                 Key: SPARK-6662
>                 URL: https://issues.apache.org/jira/browse/SPARK-6662
>             Project: Spark
>          Issue Type: Wish
>          Components: YARN
>    Affects Versions: 1.3.0
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>            Priority: Minor
>              Labels: yarn
>             Fix For: 1.4.0
>
>
> In Spark on YARN, explicit hostname and port number need to be set for "spark.yarn.historyServer.address" in SparkConf to make the HISTORY link. If the history server address is known and static, this is usually not a problem.
> But in cloud, that is usually not true. Particularly in EMR, the history server always runs on the same node as with RM. So I could simply set it to {{$\{yarn.resourcemanager.hostname\}:18080}} if variable substitution is allowed.
> In fact, Hadoop configuration already implements variable substitution, so if this property is read via YarnConf, this can be easily achievable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org