[spark1.4] sparkContext.stop causes exception on Mesos

[Ayoub <be...@gmail.com>]

Hello Spark developers, 

After upgrading to spark 1.4 on Mesos 0.22.1 existing code started to throw
this exception when calling sparkContext.stop :

(SparkListenerBus) [ERROR -
org.apache.spark.Logging$class.logError(Logging.scala:96)] Listener
EventLoggingListener threw an exception 
java.lang.reflect.InvocationTargetException 
        at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) 
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:606) 
        at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:146) 
        at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:146) 
        at scala.Option.foreach(Option.scala:236) 
        at
org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:146) 
        at
org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:190) 
        at
org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:54) 
        at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) 
        at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) 
        at
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) 
        at
org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) 
        at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) 
        at
org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1215) 
        at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) 
Caused by: java.io.IOException: Filesystem closed 
        at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:730) 
        at
org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1855) 
        at
org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1816) 
        at
org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130) 
        ... 16 more 
I0701 15:03:46.101809  1612 sched.cpp:1589] Asked to stop the driver 
I0701 15:03:46.101971  1355 sched.cpp:831] Stopping framework
'20150629-132734-1224736778-5050-6126-0028'


This problems happens only when spark.eventLog.enabled flag is set to true,
it happens also if sparkContext.stop is omitted in the code, I think because
Spark shut down indirectly the spark context. 

Does anyone know what could cause this problem ?

Thanks,
Ayoub.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark1-4-sparkContext-stop-causes-exception-on-Mesos-tp23605.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org