You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/09/11 00:49:46 UTC

[jira] [Comment Edited] (SPARK-10251) Some internal spark classes are not registered with kryo

    [ https://issues.apache.org/jira/browse/SPARK-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739781#comment-14739781 ] 

Josh Rosen edited comment on SPARK-10251 at 9/10/15 10:49 PM:
--------------------------------------------------------------

Just to be clear, my suggestion was that we change the defaults (via system properties) for all of our tests, such that our existing tests exercise the registration-required codepath. I was *not* proposing to run those tests twice, once with registration required and once without.


was (Author: joshrosen):
Just to be clear, my suggestion was that we change the defaults (via system properties) for all of our tests, such that our existing tests exercise the registration-required codepath. I was not proposing to run those tests twice, once with registration required and once without.

> Some internal spark classes are not registered with kryo
> --------------------------------------------------------
>
>                 Key: SPARK-10251
>                 URL: https://issues.apache.org/jira/browse/SPARK-10251
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.4.1
>            Reporter: Soren Macbeth
>            Assignee: Ram Sriharsha
>             Fix For: 1.6.0
>
>
> When running a job using kryo serialization and setting `spark.kryo.registrationRequired=true` some internal classes are not registered, causing the job to die. This is still a problem when this setting is false (which is the default) because it makes the space required to store serialized objects in memory or disk much much more expensive in terms of runtime and storage space.
> {code}
> 15/08/25 20:28:21 WARN spark.scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, a.b.c.d): java.lang.IllegalArgumentException: Class is not registered: scala.Tuple2[]
> Note: To register this class use: kryo.register(scala.Tuple2[].class);
>         at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
>         at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
>         at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
>         at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565)
>         at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:250)
>         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:236)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org