You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by scwf <gi...@git.apache.org> on 2014/05/02 04:57:32 UTC
[GitHub] spark pull request: get sparkhome from sparkconf Preferentially
GitHub user scwf opened a pull request:
https://github.com/apache/spark/pull/615
get sparkhome from sparkconf Preferentially
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/scwf/spark fixsparkhome
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/615.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #615
----
commit 929d784c28de5ea46f3727ba3ffc8eaa524e2a32
Author: wangfei <wa...@126.com>
Date: 2014-05-02T02:55:32Z
get sparkhome from sparkconf Preferentially
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: get sparkhome from sparkconf Preferentially
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/615#discussion_r12228337
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ---
@@ -83,7 +83,8 @@ private[spark] class Worker(
@volatile var registered = false
@volatile var connected = false
val workerId = generateWorkerId()
- val sparkHome = new File(Option(System.getenv("SPARK_HOME")).getOrElse("."))
+ val sparkHome = new File(conf.getOption("spark.home").orElse(Option(
--- End diff --
Right, sparkconf for application-wide configuration is sensible
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: get sparkhome from sparkconf Preferentially
Posted by scwf <gi...@git.apache.org>.
Github user scwf closed the pull request at:
https://github.com/apache/spark/pull/615
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: get sparkhome from sparkconf Preferentially
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/615#discussion_r12223079
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ---
@@ -83,7 +83,8 @@ private[spark] class Worker(
@volatile var registered = false
@volatile var connected = false
val workerId = generateWorkerId()
- val sparkHome = new File(Option(System.getenv("SPARK_HOME")).getOrElse("."))
+ val sparkHome = new File(conf.getOption("spark.home").orElse(Option(
--- End diff --
SparkConf is intended for application-wide configurations. SPARK_HOME is actually a node-dependent configuration, so we'd prefer to keep it outside of Spark conf. If you look we've done this for other node-specific configs like SPARK_LOCAL_DIR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: get sparkhome from sparkconf Preferentially
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/615#issuecomment-41980439
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: get sparkhome from sparkconf Preferentially
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/615#discussion_r12222831
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ---
@@ -83,7 +83,8 @@ private[spark] class Worker(
@volatile var registered = false
@volatile var connected = false
val workerId = generateWorkerId()
- val sparkHome = new File(Option(System.getenv("SPARK_HOME")).getOrElse("."))
+ val sparkHome = new File(conf.getOption("spark.home").orElse(Option(
--- End diff --
Since spark provide sparkconf to manage it's config system, we should first get option from sparkconf, then system env(compatible with old version). And we‘d better avoid using system env.
Actually in sparkcontext it get sparkhome like this
private[spark] def getSparkHome(): Option[String] = {
conf.getOption("spark.home").orElse(Option(System.getenv("SPARK_HOME")))
}
also, we keep consistent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: get sparkhome from sparkconf Preferentially
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/615#discussion_r12218870
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ---
@@ -83,7 +83,8 @@ private[spark] class Worker(
@volatile var registered = false
@volatile var connected = false
val workerId = generateWorkerId()
- val sparkHome = new File(Option(System.getenv("SPARK_HOME")).getOrElse("."))
+ val sparkHome = new File(conf.getOption("spark.home").orElse(Option(
--- End diff --
Just wondering, could you describe the use case where you want to set this via a Spark property and not by setting `SPARK_HOME` on the cluster?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---