You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Guillermo Ortiz <ko...@gmail.com> on 2018/03/25 22:11:15 UTC
Deserialize Map[Int, UDT] to a case class from Spark Connector
Hello,
I'm working with UDT's and spark connector with these dependencies:
<scala.version>2.11.12</scala.version>
<spark.version>2.0.2</spark.version>
<cassandra-conector.version>2.0.7</cassandra-conector.version>
<cassandra-driver.version>3.4.0</cassandra-driver.version>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>${cassandra-conector.version}</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>${cassandra-driver.version}</version>
</dependency>
So, with these dependencies I'm using scala 2.11, but I get this error
*the GettableToMappedTypeConverter which can't deserialize TypeTags
due
to Scala 2.10 TypeTag limitation. They come back as nulls and therefore
you see this NPE.*
Why do I get this error if I'm using SCALA 2.11? I want to read a
MAP[Int,<frozen>MyUDT>]
from Spark with the connector. The problem it's that if theree are any
field which it's not setted it's not possible.
If all fields are setted it works.
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure:
Lost task 0.0 in stage 0.0 (TID 0, localhost):
java.lang.NullPointerException: Requested a TypeTag of the
GettableToMappedTypeConverter which can't deserialize TypeTags due to Scala
2.10 TypeTag limitation. They come back as nulls and therefore you see this
NPE.
at
com.datastax.spark.connector.rdd.reader.GettableDataToMappedTypeConverter.targetTypeTag(GettableDataToMappedTypeConverter.scala:34)
at
com.datastax.spark.connector.types.TypeConverter$AbstractMapConverter.valueTypeTag(TypeConverter.scala:707)
at
com.datastax.spark.connector.types.TypeConverter$MapConverter$$typecreator45$1.apply(TypeConverter.scala:791)
at
scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
at
com.datastax.spark.connector.types.TypeConverter$class.targetTypeName(TypeConverter.scala:36)
at
com.datastax.spark.connector.types.TypeConverter$CollectionConverter.targetTypeName(TypeConverter.scala:682)
at
com.datastax.spark.connector.rdd.reader.GettableDataToMappedTypeConverter.tryConvert(GettableDataToMappedTypeConverter.scala:156)
Re: Deserialize Map[Int, UDT] to a case class from Spark Connector
Posted by Christophe Schmitz <ch...@instaclustr.com>.
Hi Guillermo
Which version of Spark are you using? Starting with Version 2.0, Spark is
built with Scala 2.11 by default. If you are using a prior version (which
looks like it's the case since your error message mention scala 2.10), you
might need to build it yourself from sources with Scala 2.11 support or to
upgrade your Spark cluster to 2.x
Cheers,
Christophe
On 26 March 2018 at 09:11, Guillermo Ortiz <ko...@gmail.com> wrote:
> Hello,
>
> I'm working with UDT's and spark connector with these dependencies:
>
> <scala.version>2.11.12</scala.version>
> <spark.version>2.0.2</spark.version>
> <cassandra-conector.version>2.0.7</cassandra-conector.version>
> <cassandra-driver.version>3.4.0</cassandra-driver.version>
>
>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.11</artifactId>
> <version>${spark.version}</version>
> </dependency>
>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming_2.11</artifactId>
> <version>${spark.version}</version>
> </dependency>
>
>
> <dependency>
> <groupId>com.datastax.spark</groupId>
> <artifactId>spark-cassandra-connector_2.11</artifactId>
> <version>${cassandra-conector.version}</version>
> </dependency>
>
> <dependency>
> <groupId>com.datastax.cassandra</groupId>
> <artifactId>cassandra-driver-core</artifactId>
> <version>${cassandra-driver.version}</version>
> </dependency>
>
> So, with these dependencies I'm using scala 2.11, but I get this error *the GettableToMappedTypeConverter which can't deserialize TypeTags due
> to Scala 2.10 TypeTag limitation. They come back as nulls and therefore
> you see this NPE.*
>
> Why do I get this error if I'm using SCALA 2.11? I want to read a MAP[Int,<frozen>MyUDT>]
> from Spark with the connector. The problem it's that if theree are any field which it's not setted it's not possible.
>
> If all fields are setted it works.
>
>
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted
> due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent
> failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NullPointerException:
> Requested a TypeTag of the GettableToMappedTypeConverter which can't
> deserialize TypeTags due to Scala 2.10 TypeTag limitation. They come back
> as nulls and therefore you see this NPE.
> at com.datastax.spark.connector.rdd.reader.
> GettableDataToMappedTypeConverter.targetTypeTag(
> GettableDataToMappedTypeConverter.scala:34)
> at com.datastax.spark.connector.types.TypeConverter$
> AbstractMapConverter.valueTypeTag(TypeConverter.scala:707)
> at com.datastax.spark.connector.types.TypeConverter$
> MapConverter$$typecreator45$1.apply(TypeConverter.scala:791)
> at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$
> lzycompute(TypeTags.scala:232)
> at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
> at com.datastax.spark.connector.types.TypeConverter$class.
> targetTypeName(TypeConverter.scala:36)
> at com.datastax.spark.connector.types.TypeConverter$
> CollectionConverter.targetTypeName(TypeConverter.scala:682)
> at com.datastax.spark.connector.rdd.reader.
> GettableDataToMappedTypeConverter.tryConvert(
> GettableDataToMappedTypeConverter.scala:156)
>
--
*Christophe Schmitz - **VP Consulting*
AU: +61 4 03751980 / FR: +33 7 82022899
<https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
<https://www.linkedin.com/company/instaclustr>
Read our latest technical blog posts here
<https://www.instaclustr.com/blog/>. This email has been sent on behalf
of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.