You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2014/12/04 20:45:12 UTC

[jira] [Commented] (SPARK-4737) Prevent serialization errors from ever crashing the DAG scheduler

    [ https://issues.apache.org/jira/browse/SPARK-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234538#comment-14234538 ] 

Michael Armbrust commented on SPARK-4737:
-----------------------------------------

I think another big problem here is that the DAGScheduler restarts (somewhat silently) and comes back in a bad state.  Perhaps if the DAGScheduler crashes we should kill the whole process if we aren't actually resilient to restarts.

> Prevent serialization errors from ever crashing the DAG scheduler
> -----------------------------------------------------------------
>
>                 Key: SPARK-4737
>                 URL: https://issues.apache.org/jira/browse/SPARK-4737
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Patrick Wendell
>            Assignee: Matthew Cheah
>            Priority: Blocker
>
> Currently in Spark we assume that when tasks are serialized in the TaskSetManager that the serialization cannot fail. We assume this because upstream in the DAGScheduler we attempt to catch any serialization errors by serializing a single partition. However, in some cases this upstream test is not accurate - i.e. an RDD can have one partition that can serialize cleanly but not others.
> Do do this in the proper way we need to catch and propagate the exception at the time of serialization. The tricky bit is making sure it gets propagated in the right way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org