You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2013/11/26 00:25:36 UTC
[2/2] git commit: Merge pull request #101 from
colorant/yarn-client-scheduler
Merge pull request #101 from colorant/yarn-client-scheduler
For SPARK-527, Support spark-shell when running on YARN
sync to trunk and resubmit here
In current YARN mode approaching, the application is run in the Application Master as a user program thus the whole spark context is on remote.
This approaching won't support application that involve local interaction and need to be run on where it is launched.
So In this pull request I have a YarnClientClusterScheduler and backend added.
With this scheduler, the user application is launched locally,While the executor will be launched by YARN on remote nodes with a thin AM which only launch the executor and monitor the Driver Actor status, so that when client app is done, it can finish the YARN Application as well.
This enables spark-shell to run upon YARN.
This also enable other Spark applications to have the spark context to run locally with a master-url "yarn-client". Thus e.g. SparkPi could have the result output locally on console instead of output in the log of the remote machine where AM is running on.
Docs also updated to show how to use this yarn-client mode.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/eb4296c8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/eb4296c8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/eb4296c8
Branch: refs/heads/master
Commit: eb4296c8f7561aaf8782479dd5cd7c9320b7fa6b
Parents: 62889c4 ab3cefd
Author: Matei Zaharia <ma...@eecs.berkeley.edu>
Authored: Mon Nov 25 15:25:29 2013 -0800
Committer: Matei Zaharia <ma...@eecs.berkeley.edu>
Committed: Mon Nov 25 15:25:29 2013 -0800
----------------------------------------------------------------------
.../scala/org/apache/spark/SparkContext.scala | 25 ++
docs/running-on-yarn.md | 27 +-
.../org/apache/spark/deploy/yarn/Client.scala | 13 +-
.../spark/deploy/yarn/ClientArguments.scala | 40 +--
.../spark/deploy/yarn/WorkerLauncher.scala | 246 +++++++++++++++++++
.../cluster/YarnClientClusterScheduler.scala | 47 ++++
.../cluster/YarnClientSchedulerBackend.scala | 109 ++++++++
7 files changed, 484 insertions(+), 23 deletions(-)
----------------------------------------------------------------------