You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/04/09 12:39:12 UTC
[jira] [Resolved] (SPARK-6343) Make doc more explicit regarding
network connectivity requirements
[ https://issues.apache.org/jira/browse/SPARK-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-6343.
------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
1.3.2
Issue resolved by pull request 5382
[https://github.com/apache/spark/pull/5382]
> Make doc more explicit regarding network connectivity requirements
> ------------------------------------------------------------------
>
> Key: SPARK-6343
> URL: https://issues.apache.org/jira/browse/SPARK-6343
> Project: Spark
> Issue Type: Improvement
> Components: Documentation
> Reporter: Peter Parente
> Priority: Minor
> Fix For: 1.3.2, 1.4.0
>
>
> As a new user of Spark, I read through the official documentation before attempting to stand-up my own cluster and write my own driver application. But only after attempting to run my app remotely against my cluster did I realize that full network connectivity (layer 3) is necessary between my driver program and worker nodes (i.e., my driver was *listening* for connections from my workers).
> I returned to the documentation to see how I had missed this requirement. On a second read-through, I saw that the doc hints at it in a few places (e.g., [driver config|http://spark.apache.org/docs/1.2.0/configuration.html#networking], [submitting applications suggestion|http://spark.apache.org/docs/1.2.0/submitting-applications.html], [cluster overview|http://spark.apache.org/docs/1.2.0/cluster-overview.html]) but never outright says it.
> I think it would help would-be users better understand how Spark works to state the network connectivity requirements right up-front in the overview section of the doc. I suggest revising the diagram and accompanying text found on the [overview page|http://spark.apache.org/docs/1.2.0/cluster-overview.html]:
> !http://spark.apache.org/docs/1.2.0/img/cluster-overview.png!
> so that it depicts at least the directionality of the network connections initiated (perhaps like so):
> !http://i.imgur.com/2dqGbCr.png!
> and states that the driver must listen for and accept connections from other Spark components on a variety of ports.
> Please treat my diagram and text as strawmen: I expect more experienced Spark users and developers will have better ideas on how to convey these requirements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org