[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip

["Joe Eloff (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215371#comment-15215371 ] 

Joe Eloff commented on SPARK-4563:
----------------------------------

I have the same issue developing against Spark running on AWS while at home on a private network.
Obvious work around is to package the source each time and deploy it but this is such a waste of time vs just doing the submit from your dev environment.

Also the way this is implemented seems like an over kill. I can't see why the driver must be listening on its own listener again. It seems like the implementation is trying to make TCP behave like UDP. I might not understand the whole problem but in my opinion once the driver(client) has made the connection to the master(server) the NAT tunnel should now be open. The master should just talk back on that end point that was created by making the TCP connection from the client to the server.  
In this case you don't need to have another random listener that needs to be registered that is now stuck behind a NATed network, just use the one you got(which is also somehow random and network controlled) when the client makes the initial connection.

The only case where this won't work and that is when the connection drops, but there is a good to great chance that the registered driver listener at this point won't work either as the network has dropped. It is TCP after all so it should stay connected for as long as there is network. But even so you could always build in a client side keep alive to make it even more robust with very little extra overhead as to just keep alive the NAT tunnel. But tis is only needed if there is going to be longer periods of no comms, which is not really the case here as te whole point is to run a job with a constant feed back loop.

> Allow spark driver to bind to different ip then advertise ip
> ------------------------------------------------------------
>
>                 Key: SPARK-4563
>                 URL: https://issues.apache.org/jira/browse/SPARK-4563
>             Project: Spark
>          Issue Type: Improvement
>          Components: Deploy
>            Reporter: Long Nguyen
>            Priority: Minor
>
> Spark driver bind ip and advertise is not configurable. spark.driver.host is only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option to set advertised ip/hostname



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org