You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mark Grover (JIRA)" <ji...@apache.org> on 2015/07/27 20:16:07 UTC
[jira] [Comment Edited] (SPARK-9384) Easier setting of executor and
driver classpath
[ https://issues.apache.org/jira/browse/SPARK-9384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14643127#comment-14643127 ]
Mark Grover edited comment on SPARK-9384 at 7/27/15 6:15 PM:
-------------------------------------------------------------
Personally, there are two easy ways to solve this problem:
1. To apply the driver's classpath to executor, if executor classpath is not set.
2. To create a new property, something like 'spark.common.classpath' that gets applied to both the driver and executor classpath, in addition to the specific
{code}
spark.{driver|executor}.classpath properties
{code}
In general, I agree with the sentiment that having too many properties can be a mess but in this case, #1 can lead to too drastic of a behavior change on an upgrade, so I am strongly leaning towards #2. That would be backwards compatible in the sense, that if you don't use the new property nothing changes, but for new users and for those who want to make their configuration management easier, they would just need to set things in only one place.
I have taken a quick pass at it [here|https://github.com/markgrover/spark/compare/pmaster...classpath2], I'd appreciate if someone could take a quick look and tell me if I am on the right track. No pull request just yet because I want to convince myself it works by testing it more, before I do so:-)
Your thoughts and suggestions are greatly appreciated!
was (Author: mgrover):
Personally, there are two easy ways to solve this problem:
1. To apply the driver's classpath to executor, if executor classpath is not set.
2. To create a new property, something like 'spark.common.classpath' that gets applied to both the driver and executor classpath, in addition to the specific spark.{driver|executor}.classpath properties.
In general, I agree with the sentiment that having too many properties can be a mess but in this case, #1 can lead to too drastic of a behavior change on an upgrade, so I am strongly leaning towards #2. That would be backwards compatible in the sense, that if you don't use the new property nothing changes, but for new users and for those who want to make their configuration management easier, they would just need to set things in only one place.
I have taken a quick pass at it [here|https://github.com/markgrover/spark/compare/pmaster...classpath2], I'd appreciate if someone could take a quick look and tell me if I am on the right track. No pull request just yet because I want to convince myself it works by testing it more, before I do so:-)
Your thoughts and suggestions are greatly appreciated!
> Easier setting of executor and driver classpath
> -----------------------------------------------
>
> Key: SPARK-9384
> URL: https://issues.apache.org/jira/browse/SPARK-9384
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 1.4.1
> Reporter: Mark Grover
> Priority: Minor
>
> Currently one can set extra jars in the driver classpath using {{spark.driver.extraClassPath}} property. Similarly, the executor's extra classpath can be set using {{spark.executor.extraClassPath}} property.
> However, many users (myself) included have often suffered from the dogma of setting one and missing the other. Causing a lot of wasted time. And, this JIRA is an effort to see if the status quo can be made better.
> More details in comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org