[jira] [Created] (SPARK-26414) Race between SparkContext and YARN AM can cause NPE in UI setup code

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

Marcelo Vanzin created SPARK-26414:
--------------------------------------

             Summary: Race between SparkContext and YARN AM can cause NPE in UI setup code
                 Key: SPARK-26414
                 URL: https://issues.apache.org/jira/browse/SPARK-26414
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 2.4.0
            Reporter: Marcelo Vanzin


There's a super narrow race between the SparkContext and the AM startup code:

- SC starts the AM and waits for it to go into running state
- AM goes into running state, unblocking SC
- AM sends AmIpFilter config to SC, adds the filter to the list and then the filter configs
- unblocked SC is in the middle of setting up the UI and sees only the filter, but not the configs

Then you get this:

{noformat}
ERROR org.apache.spark.SparkContext  - Error initializing SparkContext.
java.lang.NullPointerException
	at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.init(AmIpFilter.java:81)
	at org.spark_project.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
	at org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:881)
	at org.spark_project.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349)
	at org.spark_project.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
	at org.spark_project.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
	at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:520)
	at org.apache.spark.ui.WebUI$$anonfun$attachHandler$1.apply(WebUI.scala:96)
	at org.apache.spark.ui.WebUI$$anonfun$attachHandler$1.apply(WebUI.scala:96)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.ui.WebUI.attachHandler(WebUI.scala:96)
	at org.apache.spark.SparkContext$$anonfun$22$$anonfun$apply$8.apply(SparkContext.scala:522)
	at org.apache.spark.SparkContext$$anonfun$22$$anonfun$apply$8.apply(SparkContext.scala:522)
	at scala.Option.foreach(Option.scala:257)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org