You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2014/04/11 02:49:41 UTC
git commit: Set spark.executor.uri from environment variable (needed
by Mesos)
Repository: spark
Updated Branches:
refs/heads/master 2c557837b -> 5cd11d51c
Set spark.executor.uri from environment variable (needed by Mesos)
The Mesos backend uses this property when setting up a slave process. It is similarly set in the Scala repl (org.apache.spark.repl.SparkILoop), but I couldn't find any analogous for pyspark.
Author: Ivan Wick <iv...@gmail.com>
This patch had conflicts when merged, resolved by
Committer: Matei Zaharia <ma...@databricks.com>
Closes #311 from ivanwick/master and squashes the following commits:
da0c3e4 [Ivan Wick] Set spark.executor.uri from environment variable (needed by Mesos)
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5cd11d51
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5cd11d51
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5cd11d51
Branch: refs/heads/master
Commit: 5cd11d51c19321981a6234a7765c7a5be6913433
Parents: 2c55783
Author: Ivan Wick <iv...@gmail.com>
Authored: Thu Apr 10 17:49:30 2014 -0700
Committer: Matei Zaharia <ma...@databricks.com>
Committed: Thu Apr 10 17:49:30 2014 -0700
----------------------------------------------------------------------
python/pyspark/shell.py | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/5cd11d51/python/pyspark/shell.py
----------------------------------------------------------------------
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py
index 35e4827..61613db 100644
--- a/python/pyspark/shell.py
+++ b/python/pyspark/shell.py
@@ -29,6 +29,9 @@ from pyspark.storagelevel import StorageLevel
# this is the equivalent of ADD_JARS
add_files = os.environ.get("ADD_FILES").split(',') if os.environ.get("ADD_FILES") != None else None
+if os.environ.get("SPARK_EXECUTOR_URI"):
+ SparkContext.setSystemProperty("spark.executor.uri", os.environ["SPARK_EXECUTOR_URI"])
+
sc = SparkContext(os.environ.get("MASTER", "local[*]"), "PySparkShell", pyFiles=add_files)
print """Welcome to