You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/07/01 23:34:11 UTC
[jira] [Commented] (SPARK-16272) Allow configs to reference other
configs, env and system properties
[ https://issues.apache.org/jira/browse/SPARK-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359831#comment-15359831 ]
Apache Spark commented on SPARK-16272:
--------------------------------------
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/14022
> Allow configs to reference other configs, env and system properties
> -------------------------------------------------------------------
>
> Key: SPARK-16272
> URL: https://issues.apache.org/jira/browse/SPARK-16272
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Marcelo Vanzin
> Priority: Minor
>
> Currently, Spark's configuration is static; it is whatever is written to the config file, with some rare exceptions (such as some YARN code that does expansion of Hadoop configuration).
> But there are a few use cases that don't work well in that situation. For example, consider {{spark.sql.hive.metastore.jars}}. It references a list of paths containing the classpath for accessing Hive's metastore. If you're launching an application in cluster mode, it means that whatever is in the configuration of the edge node needs to match the configuration of the random node in the cluster where the driver will actually run.
> This would be easily solved if there was a way to reference system properties or env variables; for example, when YARN launches a container, a bunch of env variables are set, which could be used to modify that path to match the correct location on the node.
> So I'm proposing a change where config properties can opt-in to use this variable expansion feature; it's opt-in to avoid breaking existing code (who knows) and to avoid the extra cost of doing the variable expansion of every config read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org