You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "sagar (JIRA)" <ji...@apache.org> on 2015/05/10 19:36:00 UTC
[jira] [Commented] (SPARK-1132) Persisting Web UI through
refactoring the SparkListener interface
[ https://issues.apache.org/jira/browse/SPARK-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14537292#comment-14537292 ]
sagar commented on SPARK-1132:
------------------------------
Hi Team,
I see the issue is resolved and Fix Version/s: is -1.0.0.
Is 1.0.0 is spark version ?
Where i can get the spark version 1.0.0.
Currently i am getting below error -
15/05/10 08:42:20 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
15/05/10 08:42:20 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
15/05/10 08:42:20 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
15/05/10 08:42:20 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
15/05/10 08:42:20 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
15/05/10 08:42:20 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1750 bytes result sent to driver
15/05/10 08:42:20 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 207 ms on localhost (1/1)
15/05/10 08:42:20 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/05/10 08:42:20 INFO DAGScheduler: Stage 0 (count at SparkFilter.java:22) finished in 0.225 s
15/05/10 08:42:20 INFO DAGScheduler: Job 0 finished: count at SparkFilter.java:22, took 0.314437 s
0
15/05/10 08:42:20 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
at org.apache.spark.scheduler.EventLoggingListener.onStageCompleted(EventLoggingListener.scala:165)
at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:32)
at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:792)
at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1998)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1959)
at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
... 19 more
> Persisting Web UI through refactoring the SparkListener interface
> -----------------------------------------------------------------
>
> Key: SPARK-1132
> URL: https://issues.apache.org/jira/browse/SPARK-1132
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Web UI
> Affects Versions: 0.9.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Fix For: 1.0.0
>
>
> This issue is a spin-off from another issue - https://spark-project.atlassian.net/browse/SPARK-969
> The main issue with the existing Spark Web UI is that its information is lost as soon as the application terminates. This is the direct result of the SparkUI being coupled with SparkContext, which is stopped when the application is finished.
> The attached document proposes to tackle this by logging SparkListenerEvents to persist information displayed on the Web UI. We take this opportunity to replace the existing format for storing this information, HTML, with one that is more flexible, JSON. This allows further post-hoc analysis of a particular Spark application beyond simply reviving the Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org