[jira] [Created] (SPARK-4799) Spark should not rely on local host being resolvable on every node

["Santiago M. Mola (JIRA)" <ji...@apache.org>]

Santiago M. Mola created SPARK-4799:
---------------------------------------

             Summary: Spark should not rely on local host being resolvable on every node
                 Key: SPARK-4799
                 URL: https://issues.apache.org/jira/browse/SPARK-4799
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.1.0
         Environment: Tested a Spark+Mesos cluster on top of Docker to reproduce the issue.
            Reporter: Santiago M. Mola


Spark fails when a node hostname is not resolvable by other nodes.

See an example trace:

{code}
14/12/09 17:02:41 ERROR SendingConnection: Error connecting to 27e434cf36ac:35093
java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:127)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
	at org.apache.spark.network.SendingConnection.connect(Connection.scala:299)
	at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:278)
	at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
{code}

The relevant code is here:
https://github.com/apache/spark/blob/bcb5cdad614d4fce43725dfec3ce88172d2f8c11/core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala#L170

{code}
val id = new ConnectionManagerId(Utils.localHostName, serverChannel.socket.getLocalPort)
{code}

This piece of code should use the host IP with Utils.localIpAddress or a method that acknowleges user settings (e.g. SPARK_LOCAL_IP). Since I cannot think about a use case for using hostname here, I'm creating a PR with the former solution, but if you think the later is better, I'm willing to create a new PR with a more elaborate fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org