You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kalyana Chakravarthy Kadiyala (JIRA)" <ji...@apache.org> on 2014/11/10 03:07:33 UTC
[jira] [Commented] (SPARK-2403) Spark stuck when class is not
registered with Kryo
[ https://issues.apache.org/jira/browse/SPARK-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204237#comment-14204237 ]
Kalyana Chakravarthy Kadiyala commented on SPARK-2403:
------------------------------------------------------
I still get this error (deploying on YARN cluster). Built from source (date git repo cloned - 11/9/2014).
mvn -Pyarn -Pdeb -Phadoop-2.4 -Dhadoop.version=2.4.1 -DskipTests clean package install
Here is the error trace:
2014-11-08 14:58:21,134 INFO [sparkDriver-akka.actor.default-dispatcher-3] spark.SparkContext (Logging.scala:logInfo(59)) - Created broadcast 0 from broadcast at DAGScheduler.scala:838
2014-11-08 14:58:21,158 INFO [sparkDriver-akka.actor.default-dispatcher-3] cluster.YarnClusterScheduler (Logging.scala:logInfo(59)) - Cancelling stage 0
2014-11-08 14:58:21,163 INFO [Driver] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Job 0 failed: reduce at SparkPi.scala:35, took 0.425176 s
2014-11-08 14:58:21,170 INFO [Driver] yarn.ApplicationMaster (Logging.scala:logInfo(59)) - Final app status: FAILED, exitCode: 15, (reason: User class threw exception: Job aborted due to stage failure: Task serialization failed: java.io.IOException: java.lang.IllegalArgumentException: Class is not registered: scala.collection.immutable.Range
Note: To register this class use: kryo.register(scala.collection.immutable.Range.class);
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:988)
org.apache.spark.rdd.ParallelCollectionPartition.writeObject(ParallelCollectionRDD.scala:51)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:876)
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:762)
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1381)
akka.actor.Actor$class.aroundReceive(Actor.scala:465)
org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1367)
akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
akka.actor.ActorCell.invoke(ActorCell.scala:487)
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
akka.dispatch.Mailbox.run(Mailbox.scala:220)
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
)
2014-11-08 14:58:31,722 INFO [main] ipc.Client (Client.java:handleConnectionFailure(841)) - Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-11-08 14:58:32,724 INFO [main] ipc.Client (Client.java:handleConnectionFailure(841)) - Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-11-08 14:58:33,726 INFO [main] ipc.Client (Client.java:handleConnectionFailure(841)) - Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
> Spark stuck when class is not registered with Kryo
> --------------------------------------------------
>
> Key: SPARK-2403
> URL: https://issues.apache.org/jira/browse/SPARK-2403
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Daniel Darabos
> Fix For: 1.0.2, 1.1.0
>
>
> We are using Kryo and require registering classes. When trying to serialize something containing an unregistered class, Kryo will raise an exception.
> DAGScheduler.submitMissingTasks runs in the scheduler thread and checks if the contents of the task can be serialized by trying to serialize it:
> https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L767
> It catches NotSerializableException and aborts the task with an error when this happens.
> The problem is, Kryo does not raise NotSerializableException for unregistered classes. It raises IllegalArgumentException instead. This exception is not caught and kills the scheduler thread. The application then hangs, waiting indefinitely for the job to finish.
> Catching IllegalArgumentException also is a quick fix. I'll send a pull request for it if you agree. Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org