You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/03/06 22:06:40 UTC
[jira] [Reopened] (SPARK-13620) Avoid reverse DNS lookup for
0.0.0.0 on startup
[ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen reopened SPARK-13620:
-------------------------------
> Avoid reverse DNS lookup for 0.0.0.0 on startup
> -----------------------------------------------
>
> Key: SPARK-13620
> URL: https://issues.apache.org/jira/browse/SPARK-13620
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 1.6.0
> Reporter: Daniel Darabos
> Priority: Minor
>
> I noticed we spend 5+ seconds during application startup with the following stack trace:
> {code}
> at java.net.Inet6AddressImpl.getHostByAddr(Native Method)
> at java.net.InetAddress$1.getHostByAddr(InetAddress.java:926)
> at java.net.InetAddress.getHostFromNameService(InetAddress.java:611)
> at java.net.InetAddress.getHostName(InetAddress.java:553)
> at java.net.InetAddress.getHostName(InetAddress.java:525)
> at java.net.InetSocketAddress$InetSocketAddressHolder.getHostName(InetSocketAddress.java:82)
> at java.net.InetSocketAddress$InetSocketAddressHolder.access$600(InetSocketAddress.java:56)
> at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:345)
> at org.spark-project.jetty.server.Server.<init>(Server.java:115)
> at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:243)
> at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
> at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
> at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1964)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1955)
> at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:262)
> at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
> at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
> at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
> at scala.Option.foreach(Option.scala:236)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:481)
> {code}
> Spark wants to start a server on localhost. So it [creates an {{InetSocketAddress}}|https://github.com/apache/spark/blob/v1.6.0/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L243] [with hostname {{"0.0.0.0"}}|https://github.com/apache/spark/blob/v1.6.0/core/src/main/scala/org/apache/spark/ui/WebUI.scala#L136]. Spark passes in a hostname string, but Java [recognizes that it's actually an address|https://github.com/openjdk-mirror/jdk/blob/adea42765ae4e7117c3f0e2d618d5e6aed44ced2/src/share/classes/java/net/InetSocketAddress.java#L220] and so sets the hostname to {{null}}. So when Jetty [calls {{getHostName}}|https://github.com/eclipse/jetty.project/blob/jetty-8.1.14.v20131031/jetty-server/src/main/java/org/eclipse/jetty/server/Server.java#L115] Java has to do a reverse DNS lookup for {{0.0.0.0}}. That takes 5+ seconds on my machine. Maybe it's just me? It's a very vanilla Ubuntu 14.04.
> There is a simple fix. Instead of passing in {{"0.0.0.0"}} we should not set a hostname. In this case [{{InetAddress.anyLocalAddress()}} is used|https://github.com/openjdk-mirror/jdk/blob/adea42765ae4e7117c3f0e2d618d5e6aed44ced2/src/share/classes/java/net/InetSocketAddress.java#L166], which is the same, but does not need resolving.
> {code}
> scala> { val t0 = System.currentTimeMillis; new java.net.InetSocketAddress("0.0.0.0", 8000).getHostName; System.currentTimeMillis - t0 }
> res0: Long = 5432
> scala> { val t0 = System.currentTimeMillis; new java.net.InetSocketAddress(8000).getHostName; System.currentTimeMillis - t0 }
> res1: Long = 0
> {code}
> I'll send a pull request for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org