You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Igor Kamyshnikov (Jira)" <ji...@apache.org> on 2020/09/22 17:56:00 UTC
[jira] [Comment Edited] (SPARK-19938) java.lang.ClassCastException:
cannot assign instance of
scala.collection.immutable.List$SerializationProxy to field
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200272#comment-17200272 ]
Igor Kamyshnikov edited comment on SPARK-19938 at 9/22/20, 5:55 PM:
--------------------------------------------------------------------
[~rdblue], my analysis shows the different root cause of the problem:
https://bugs.openjdk.java.net/browse/JDK-8024931 (never fixed)
https://github.com/scala/bug/issues/9777 (asking scala to solve on their side)
It's about circular references among the objects being serialized:
RDD1.dependencies_ = Seq1[RDD2]
RDD2.dependences_ = Seq2[RDD3]
RDD3 with some Dataset/catalyst magic can refer back to the Seq1[RDD2]
Seq are instances of scala.collection.immutable.List which uses writeReplace, giving an instance of 'SerializationProxy'. The serialization of RDD3 puts a reference to the Seq1's SerializationProxy. When the deserialization works, it reads that reference to SerializationProxy earlier than the 'readResolve' method is called (see the JDK bug reported).
was (Author: kamyshnikov):
[~rdblue], my analysis shows the different root cause of the problem:
https://bugs.openjdk.java.net/browse/JDK-8024931
https://github.com/scala/bug/issues/9777
It's about circular references among the objects being serialized:
RDD1.dependencies_ = Seq1[RDD2]
RDD2.dependences_ = Seq2[RDD3]
RDD3 with some Dataset/catalyst magic can refer back to the Seq1[RDD2]
Seq are instances of scala.collection.immutable.List which uses writeReplace, giving an instance of 'SerializationProxy'. The serialization of RDD3 puts a reference to the Seq1's SerializationProxy. When the deserialization works, it reads that reference to SerializationProxy earlier than the 'readResolve' method is called (see the JDK bug reported).
> java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field
> -------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19938
> URL: https://issues.apache.org/jira/browse/SPARK-19938
> Project: Spark
> Issue Type: Bug
> Components: Tests
> Affects Versions: 2.0.2
> Reporter: srinivas thallam
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org