You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2014/04/05 03:26:56 UTC
git commit: Update documentation for work around for SPARK-1384
Repository: spark
Updated Branches:
refs/heads/branch-0.9 7f727cf97 -> d4df0765e
Update documentation for work around for SPARK-1384
This is to workaround accessing secure hdfs from spark-shell in yarn-client mode. Note this only applies to branch-0.9 and is intended to be included in the documentation for 0.9.1. The real fix after 0.9.1 is included in https://github.com/apache/spark/pull/287
Author: Thomas Graves <tg...@apache.org>
Closes #314 from tgravescs/docFix09rc3 and squashes the following commits:
222e848 [Thomas Graves] Update documentation for work around for SPARK-1384
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d4df0765
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d4df0765
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d4df0765
Branch: refs/heads/branch-0.9
Commit: d4df0765eabfe682807dde18ce227b7057eda9dd
Parents: 7f727cf
Author: Thomas Graves <tg...@apache.org>
Authored: Fri Apr 4 18:26:51 2014 -0700
Committer: Matei Zaharia <ma...@databricks.com>
Committed: Fri Apr 4 18:26:51 2014 -0700
----------------------------------------------------------------------
docs/running-on-yarn.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/d4df0765/docs/running-on-yarn.md
----------------------------------------------------------------------
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index cd4509e..0981106 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -99,7 +99,7 @@ With this mode, your application is actually run on the remote machine where the
## Launch spark application with yarn-client mode.
-With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR
+With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR. If you are using spark-shell with secure HDFS you also need to export SPARK_YARN_MODE=true.
Configuration in yarn-client mode:
@@ -121,7 +121,7 @@ For example:
SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \
./bin/run-example org.apache.spark.examples.SparkPi yarn-client
-
+ SPARK_YARN_MODE=true \
SPARK_JAR=./assembly/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop2.0.5-alpha.jar \
SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \
MASTER=yarn-client ./bin/spark-shell