spark git commit: [SPARK-6343] Doc driver-worker network reqs

[sr...@apache.org]

Repository: spark
Updated Branches:
  refs/heads/master 2fe0a1aae -> b9c51c049


[SPARK-6343] Doc driver-worker network reqs

Attempt at making the driver-worker networking requirement more explicit and up-front in the documentation (see https://issues.apache.org/jira/browse/SPARK-6343).

Update cluster overview diagram to show connections from workers to driver. Add a bullet below about how driver listens / accepts connections from workers.

Author: Peter Parente <pp...@us.ibm.com>

Closes #5382 from parente/SPARK-6343 and squashes the following commits:

0b2fb9d [Peter Parente] [SPARK-6343] Doc driver-worker network reqs


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b9c51c04
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b9c51c04
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b9c51c04

Branch: refs/heads/master
Commit: b9c51c04932efeeda790752276078314db440634
Parents: 2fe0a1a
Author: Peter Parente <pp...@us.ibm.com>
Authored: Thu Apr 9 06:37:20 2015 -0400
Committer: Sean Owen <so...@cloudera.com>
Committed: Thu Apr 9 06:37:20 2015 -0400

----------------------------------------------------------------------
 docs/cluster-overview.md       |   6 +++++-
 docs/img/cluster-overview.png  | Bin 28011 -> 33565 bytes
 docs/img/cluster-overview.pptx | Bin 51771 -> 28133 bytes
 3 files changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/b9c51c04/docs/cluster-overview.md
----------------------------------------------------------------------
diff --git a/docs/cluster-overview.md b/docs/cluster-overview.md
index 6a75d5c..7079de5 100644
--- a/docs/cluster-overview.md
+++ b/docs/cluster-overview.md
@@ -33,7 +33,11 @@ There are several useful things to note about this architecture:
 2. Spark is agnostic to the underlying cluster manager. As long as it can acquire executor
    processes, and these communicate with each other, it is relatively easy to run it even on a
    cluster manager that also supports other applications (e.g. Mesos/YARN).
-3. Because the driver schedules tasks on the cluster, it should be run close to the worker
+3. The driver program must listen for and accept incoming connections from its executors throughout 
+   its lifetime (e.g., see [spark.driver.port and spark.fileserver.port in the network config 
+   section](configuration.html#networking)). As such, the driver program must be network 
+   addressable from the worker nodes.
+4. Because the driver schedules tasks on the cluster, it should be run close to the worker
    nodes, preferably on the same local area network. If you'd like to send requests to the
    cluster remotely, it's better to open an RPC to the driver and have it submit operations
    from nearby than to run a driver far away from the worker nodes.

http://git-wip-us.apache.org/repos/asf/spark/blob/b9c51c04/docs/img/cluster-overview.png
----------------------------------------------------------------------
diff --git a/docs/img/cluster-overview.png b/docs/img/cluster-overview.png
index 3682740..317554c 100644
Binary files a/docs/img/cluster-overview.png and b/docs/img/cluster-overview.png differ

http://git-wip-us.apache.org/repos/asf/spark/blob/b9c51c04/docs/img/cluster-overview.pptx
----------------------------------------------------------------------
diff --git a/docs/img/cluster-overview.pptx b/docs/img/cluster-overview.pptx
index af3c462..1b90d7e 100644
Binary files a/docs/img/cluster-overview.pptx and b/docs/img/cluster-overview.pptx differ


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org