You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2014/10/26 20:04:33 UTC

[jira] [Commented] (SPARK-2105) SparkUI doesn't remove active stages that failed

    [ https://issues.apache.org/jira/browse/SPARK-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184613#comment-14184613 ] 

Josh Rosen commented on SPARK-2105:
-----------------------------------

I tried and failed to reproduce this: https://github.com/apache/spark/commit/bf589fc717c842d1998e3c3a523bc8775cb30269#diff-f346ada4cd59416756b6dd36b6c2605aR97

That doesn't mean that we've fixed the issue, though.  In my tests, the stage never becomes active because the ClosureCleaner detects that the task isn't serializable.  Maybe there's some UDF that manages to slip through the closure cleaning step and fails once the stage is submitted to the scheduler, so it's still possible that we could hit this bug.

> SparkUI doesn't remove active stages that failed
> ------------------------------------------------
>
>                 Key: SPARK-2105
>                 URL: https://issues.apache.org/jira/browse/SPARK-2105
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, Web UI
>    Affects Versions: 1.0.0
>            Reporter: Andrew Or
>
> If a stage fails because its tasks cannot be serialized, for instance, the failed stage remains in the Active Stages section forever. This is because the StageCompleted event is never posted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org