[jira] [Commented] (SPARK-15974) Create a socket on YARN AM start-up

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-15974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332637#comment-15332637 ] 

Marcelo Vanzin commented on SPARK-15974:
----------------------------------------

bq.  Spark's ApplicationMaster should optionally create a ServerSocket and pass it to the Spark user program.

That makes a ton of assumptions about how the user code starts to listen for connections. If Spark is to support something like this, there should be some other way of telling Spark (or YARN directly) what the port is.

> Create a socket on YARN AM start-up
> -----------------------------------
>
>                 Key: SPARK-15974
>                 URL: https://issues.apache.org/jira/browse/SPARK-15974
>             Project: Spark
>          Issue Type: New Feature
>          Components: YARN
>            Reporter: Mingyu Kim
>
> YARN provides a way for AppilcationMaster to register a RPC port so that a client outside the YARN cluster can reach the application for any RPCs, but Spark’s YARN AMs simply register a dummy port number of 0. For the Spark programs that starts up a server, this makes it hard for the submitter to discover the server port securely. Spark's ApplicationMaster should optionally create a ServerSocket and pass it to the Spark user program. This socket initialization should be disabled by default.
> Some discussion on dev@spark thread: http://apache-spark-developers-list.1001551.n3.nabble.com/Utilizing-YARN-AM-RPC-port-field-td17892.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org