You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2014/04/03 16:07:15 UTC
[jira] [Created] (SPARK-1407) EventLogging to HDFS doesn't work
properly on yarn
Thomas Graves created SPARK-1407:
------------------------------------
Summary: EventLogging to HDFS doesn't work properly on yarn
Key: SPARK-1407
URL: https://issues.apache.org/jira/browse/SPARK-1407
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0
Reporter: Thomas Graves
Priority: Blocker
When running on spark on yarn and accessing an HDFS file (like in the SparkHdfsLR example) while using the event logging configured to write logs to HDFS, it throws an exception at the end of the application.
SPARK_JAVA_OPTS=-Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs:///history/spark/
14/04/03 13:41:31 INFO yarn.ApplicationMaster$$anon$1: Invoking sc stop from shutdown hook
Exception in thread "Thread-41" java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:398)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1465)
at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1450)
at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:116)
at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.util.FileLogger.flush(FileLogger.scala:137)
at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:69)
at org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:101)
at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.SparkListenerBus$class.postToAll(SparkListenerBus.scala:67)
at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:31)
at org.apache.spark.scheduler.LiveListenerBus.post(LiveListenerBus.scala:78)
at org.apache.spark.SparkContext.postApplicationEnd(SparkContext.scala:1081)
at org.apache.spark.SparkContext.stop(SparkContext.scala:828)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:460)
--
This message was sent by Atlassian JIRA
(v6.2#6252)