[jira] [Resolved] (SPARK-1235) DAGScheduler ignores exceptions thrown in handleTaskCompletion

["Matei Zaharia (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matei Zaharia resolved SPARK-1235.
----------------------------------

       Resolution: Fixed
    Fix Version/s: 1.0.0

Resolved in https://github.com/apache/spark/pull/186.

> DAGScheduler ignores exceptions thrown in handleTaskCompletion
> --------------------------------------------------------------
>
>                 Key: SPARK-1235
>                 URL: https://issues.apache.org/jira/browse/SPARK-1235
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 0.9.0, 0.9.1
>            Reporter: Kay Ousterhout
>            Assignee: Nan Zhu
>            Priority: Blocker
>             Fix For: 1.0.0
>
>
> If an exception gets thrown in the handleTaskCompletion method, the method exits, but the exception is caught somewhere (not clear where) and the DAGScheduler keeps running.  Jobs hang as a result -- because not all of the task completion code gets run.
> This was first reported by Brad Miller on the mailing list: http://apache-spark-user-list.1001560.n3.nabble.com/Fwd-pyspark-crash-on-mesos-td2256.html and this behavior seems to have changed since 0.8 (when, based on Brad's description, it sounds like an exception in handleTaskCompletion would cause the DAGScheduler to crash), suggesting that this may be related to the Scala 2.10.3.
> To reproduce this problem, add "throw new Exception("foo")" anywhere in handleTaskCompletion and run any job locally.  The job will hang and you can see the exception get printed in the logs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)