You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ad...@apache.org on 2014/07/08 19:43:54 UTC
git commit: [SPARK-2403] Catch all errors during serialization in
DAGScheduler
Repository: spark
Updated Branches:
refs/heads/master cc3e0a14d -> c8a2313cd
[SPARK-2403] Catch all errors during serialization in DAGScheduler
https://issues.apache.org/jira/browse/SPARK-2403
Spark hangs for us whenever we forget to register a class with Kryo. This should be a simple fix for that. But let me know if you have a better suggestion.
I did not write a new test for this. It would be pretty complicated and I'm not sure it's worthwhile for such a simple change. Let me know if you disagree.
Author: Daniel Darabos <da...@gmail.com>
Closes #1329 from darabos/spark-2403 and squashes the following commits:
3aceaad [Daniel Darabos] Print full stack trace for miscellaneous exceptions during serialization.
52c22ba [Daniel Darabos] Only catch NonFatal exceptions.
361e962 [Daniel Darabos] Catch all errors during serialization in DAGScheduler.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c8a2313c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c8a2313c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c8a2313c
Branch: refs/heads/master
Commit: c8a2313cdf825e0191680a423d17619b5504ff89
Parents: cc3e0a1
Author: Daniel Darabos <da...@gmail.com>
Authored: Tue Jul 8 10:43:46 2014 -0700
Committer: Aaron Davidson <aa...@databricks.com>
Committed: Tue Jul 8 10:43:46 2014 -0700
----------------------------------------------------------------------
.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/c8a2313c/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index 81c136d..f72bfde 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -26,6 +26,7 @@ import scala.concurrent.Await
import scala.concurrent.duration._
import scala.language.postfixOps
import scala.reflect.ClassTag
+import scala.util.control.NonFatal
import akka.actor._
import akka.actor.OneForOneStrategy
@@ -768,6 +769,10 @@ class DAGScheduler(
abortStage(stage, "Task not serializable: " + e.toString)
runningStages -= stage
return
+ case NonFatal(e) => // Other exceptions, such as IllegalArgumentException from Kryo.
+ abortStage(stage, s"Task serialization failed: $e\n${e.getStackTraceString}")
+ runningStages -= stage
+ return
}
logInfo("Submitting " + tasks.size + " missing tasks from " + stage + " (" + stage.rdd + ")")