[jira] [Resolved] (SPARK-9479) ReceiverTrackerSuite fails for maven build

["Tathagata Das (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tathagata Das resolved SPARK-9479.
----------------------------------
       Resolution: Fixed
         Assignee: Shixiong Zhu
    Fix Version/s: 1.5.0

> ReceiverTrackerSuite fails for maven build
> ------------------------------------------
>
>                 Key: SPARK-9479
>                 URL: https://issues.apache.org/jira/browse/SPARK-9479
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming, Tests
>            Reporter: Shixiong Zhu
>            Assignee: Shixiong Zhu
>             Fix For: 1.5.0
>
>
> The test failure is here: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/3109/
> I saw the following exception in the log:
> {code}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.NullPointerException
> org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80)
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> org.apache.spark.SparkContext.broadcast(SparkContext.scala:1297)
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:834)
> {code}
> This exception is because SparkEnv.get returns null.
> I found the maven build is different from the sbt build. The maven build will create all Suite classes at the beginning. `ReceiverTrackerSuite` creates StreamingContext (SparkContext) in the constructor. That means SparkContext is created very early. And the global SparkEnv will be set to null in the previous test. Therefore we saw the above exception when running `Receiver tracker - propagates rate limit` in `ReceiverTrackerSuite`. This test was added recently.
> Note: the previous tests in `ReceiverTrackerSuite` didn't use SparkContext actually, that's why we didn't see such failure before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org