[jira] [Created] (SPARK-1845) Use AllScalaRegistrar for SparkSqlSerializer to register serializers of Scala collections.

["Takuya Ueshin (JIRA)" <ji...@apache.org>]

Takuya Ueshin created SPARK-1845:
------------------------------------

             Summary: Use AllScalaRegistrar for SparkSqlSerializer to register serializers of Scala collections.
                 Key: SPARK-1845
                 URL: https://issues.apache.org/jira/browse/SPARK-1845
             Project: Spark
          Issue Type: Bug
          Components: SQL
            Reporter: Takuya Ueshin


When I execute {{orderBy}} or {{limit}} for {{SchemaRDD}} including {{ArrayType}} or {{MapType}}, {{SparkSqlSerializer}} throws the following exception:

{quote}
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): scala.collection.immutable.$colon$colon
{quote}

or

{quote}
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): scala.collection.immutable.Vector
{quote}

or

{quote}
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): scala.collection.immutable.HashMap$HashTrieMap
{quote}

and so on.

This is because registrations of serializers for each concrete collections are missing in {{SparkSqlSerializer}}.
I believe it should use {{AllScalaRegistrar}}.
{{AllScalaRegistrar}} covers a lot of serializers for concrete classes of {{Seq}}, {{Map}} for {{ArrayType}}, {{MapType}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)