[jira] [Commented] (SPARK-8109) TestSQLContext's static initialization is run during MiMa tests, causing SparkContexts to be created

["Josh Rosen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-8109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573554#comment-14573554 ] 

Josh Rosen commented on SPARK-8109:
-----------------------------------

/cc [~rxin]

> TestSQLContext's static initialization is run during MiMa tests, causing SparkContexts to be created
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-8109
>                 URL: https://issues.apache.org/jira/browse/SPARK-8109
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL, Tests
>            Reporter: Josh Rosen
>
> Check out this stacktrace which occurred during MiMa tests in the pull request builder:
> {code}
> java.net.BindException: Address already in use
> 	at sun.nio.ch.Net.bind0(Native Method)
> 	at sun.nio.ch.Net.bind(Net.java:444)
> 	at sun.nio.ch.Net.bind(Net.java:436)
> 	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> 	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> 	at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
> 	at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
> 	at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
> 	at org.eclipse.jetty.server.Server.doStart(Server.java:293)
> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
> 	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:228)
> 	at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238)
> 	at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238)
> 	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991)
> 	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> 	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982)
> 	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238)
> 	at org.apache.spark.ui.WebUI.bind(WebUI.scala:117)
> 	at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:448)
> 	at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:448)
> 	at scala.Option.foreach(Option.scala:236)
> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:448)
> 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:135)
> 	at org.apache.spark.sql.test.LocalSQLContext.<init>(TestSQLContext.scala:29)
> 	at org.apache.spark.sql.test.TestSQLContext$.<init>(TestSQLContext.scala:55)
> 	at org.apache.spark.sql.test.TestSQLContext$.<clinit>(TestSQLContext.scala)
> 	at java.lang.Class.forName0(Native Method)
> 	at java.lang.Class.forName(Class.java:274)
> 	at scala.reflect.runtime.JavaMirrors$JavaMirror.javaClass(JavaMirrors.scala:500)
> 	at scala.reflect.runtime.JavaMirrors$JavaMirror.tryJavaClass(JavaMirrors.scala:505)
> 	at scala.reflect.runtime.SymbolLoaders$PackageScope.lookupEntry(SymbolLoaders.scala:109)
> 	at scala.reflect.internal.Types$Type.findMember(Types.scala:1185)
> 	at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:722)
> 	at scala.reflect.internal.Types$Type.member(Types.scala:680)
> 	at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:43)
> 	at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
> 	at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:72)
> 	at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:161)
> 	at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:21)
> 	at org.apache.spark.tools.GenerateMIMAIgnore$$anonfun$privateWithin$1.apply(GenerateMIMAIgnore.scala:72)
> 	at org.apache.spark.tools.GenerateMIMAIgnore$$anonfun$privateWithin$1.apply(GenerateMIMAIgnore.scala:69)
> 	at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
> 	at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
> 	at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
> 	at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
> 	at org.apache.spark.tools.GenerateMIMAIgnore$.privateWithin(GenerateMIMAIgnore.scala:69)
> 	at org.apache.spark.tools.GenerateMIMAIgnore$.main(GenerateMIMAIgnore.scala:126)
> 	at org.apache.spark.tools.GenerateMIMAIgnore.main(GenerateMIMAIgnore.scala)
> {code}
> Here, TestSQLContext's static initialization code is being run during MiMa checks and that initialization creates a SparkContext.  Because MiMa doesn't run with our test system properties, the UI tries to bind to a contended port.  This may lead to flakiness.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org