You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/12/06 06:45:00 UTC
[jira] [Commented] (SPARK-37556) Deser void class fail with Java serialization
[ https://issues.apache.org/jira/browse/SPARK-37556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453804#comment-17453804 ]
Apache Spark commented on SPARK-37556:
--------------------------------------
User 'daijyc' has created a pull request for this issue:
https://github.com/apache/spark/pull/34816
> Deser void class fail with Java serialization
> ---------------------------------------------
>
> Key: SPARK-37556
> URL: https://issues.apache.org/jira/browse/SPARK-37556
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Daniel Dai
> Priority: Major
>
> Spark code contains Java void type cannot be serialized with JavaSerializer. For example:
> {code:java}
> class Foo extends Serializable {
> Class k = Void.TYPE;
> }
> {code}
> Spark will throw error:
> {code:java}
> java.lang.Void; local class name incompatible with stream class name "void"
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:703)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1940)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
> at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1771)
> ......
> {code}
> All other primitive type works. The reason is when SPARK-8730 try to fix the primitive type deserialization, it introduces the [following code|https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala#L80]:
> {code:java}
> private object JavaDeserializationStream {
> val primitiveMappings = Map[String, Class[_]](
> "boolean" -> classOf[Boolean],
> "byte" -> classOf[Byte],
> "char" -> classOf[Char],
> "short" -> classOf[Short],
> "int" -> classOf[Int],
> "long" -> classOf[Long],
> "float" -> classOf[Float],
> "double" -> classOf[Double],
> "void" -> classOf[Void]
> )
> }
> {code}
> However, classOf[Void] is not the equivalence of other types. It's point to Void.class not Void.Type. The equivalence for void should be classOf[Unit]:
> {code:java}
> scala> classOf[Long]
> val res0: Class[Long] = long
> scala> classOf[Double]
> val res1: Class[Double] = double
> scala> classOf[Byte]
> val res2: Class[Byte] = byte
> scala> classOf[Void]
> val res3: Class[Void] = class java.lang.Void <--- this is wrong
> scala> classOf[Unit]
> val res4: Class[Unit] = void <---- this is right
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org