You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Revanth Reddy <re...@gmail.com> on 2018/04/15 02:29:14 UTC
How to register cassandra custom codecs in spark? (https://github.com/datastax/spark-cassandra-connector/issues/1173)
Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my
spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another
while reading from cassandra. for example i have a timestamp column in
cassandra table ,but i want to read it as long and save as parquet file to
HDFS , then this cutom codecs will be usefull instead of doing an
additional transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
I had tried with solution provided in the below link but it does not work
, any ideas around this will be helpfull.
https://stackoverflow.com/questions/40363611/adding-custom-codec-to-cassandraconnector/49750791?noredirect=1#comment86515847_49750791
*Thanks & Regards*
*S.Revanth kumar reddy*
https://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login
Re: How to register cassandra custom codecs in spark?
(https://github.com/datastax/spark-cassandra-connector/issues/1173)
Posted by Dinesh Joshi <di...@yahoo.com.INVALID>.
Hi Revanth,
I took a quick look and don't think you can override existing mappings. You should see a warning like this -
18/04/15 19:50:27 WARN CodecRegistry: Ignoring codec CustomTSTypeCodec [date <-> java.lang.Long] because it collides with previously registered codec CustomTSTypeCodec [date <-> java.lang.Long]
The issue is that CodecRegistry does not allow you to override existing type mappings[1][2]. You can however register a codec that converts between a CqlType and your custom class.
Dinesh
[1] https://docs.datastax.com/en/developer/java-driver/3.1/manual/custom_codecs/[2] https://github.com/datastax/java-driver/blob/3.x/driver-core/src/main/java/com/datastax/driver/core/CodecRegistry.java#L357
On Sunday, April 15, 2018, 1:06:53 AM PDT, Revanth Reddy <re...@gmail.com> wrote:
Hi Dinesh,
Thanks a lot for your response . I had tried with the existing codec (SimpleTimestampCodec) ,but it says the codec has been registered but it is not converting the timestamp column to long . also i don't observe any error.
Below are the steps that i had followed and also attached the code that i had used:
I had used this property to set the codec in spark config("spark.cassandra.connection.factory", "com.rasa.devops.cassandra.export.CustomConnectionFactory")
Since there is no direct way to refer the custom codec in spark , i had followed the above approach.
I had used the following dependencies :
<dependency> <groupId>com.datastax.spark</groupId> <artifactId>spark-cassandra-connector_2.11</artifactId> <version>2.0.1</version> </dependency> <dependency> <groupId>com.datastax.cassandra</groupId> <artifactId>cassandra-driver-extras</artifactId> <version>3.1.4</version> </dependency> <dependency> <groupId>joda-time</groupId> <artifactId>joda-time</artifactId> <version>2.9.1</version> </dependency>
Cassandra table used for testing :
CREATE TABLE dev.my_table (
partition_key text PRIMARY KEY,
some_timestamp timestamp
);
cqlsh:dev> select * from my_table;
partition_key | some_timestamp
---------------+--------------------------
key | 2015-12-25 18:30:00+0000
foo | 2018-04-11 06:22:29+0000
Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login
On Sun, Apr 15, 2018 at 11:36 AM, Dinesh Joshi <di...@yahoo.com> wrote:
Hi Revanth,
How do you register the custom codec? Do you get any errors? Have you tried using a pre-existing codec? It would be helpful if you can give more information.
Dinesh On Saturday, April 14, 2018, 7:29:30 PM PDT, Revanth Reddy <re...@gmail.com> wrote:
Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another while reading from cassandra. for example i have a timestamp column in cassandra table ,but i want to read it as long and save as parquet file to HDFS , then this cutom codecs will be usefull instead of doing an additional transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
I had tried with solution provided in the below link but it does not work , any ideas around this will be helpfull.
https://stackoverflow.com/ questions/40363611/adding- custom-codec-to- cassandraconnector/49750791? noredirect=1#comment86515847_ 49750791
Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/ crazyusers/login
Re: How to register cassandra custom codecs in spark?
(https://github.com/datastax/spark-cassandra-connector/issues/1173)
Posted by Dinesh Joshi <di...@yahoo.com.INVALID>.
Hi Revanth,
How do you register the custom codec? Do you get any errors? Have you tried using a pre-existing codec? It would be helpful if you can give more information.
Dinesh On Saturday, April 14, 2018, 7:29:30 PM PDT, Revanth Reddy <re...@gmail.com> wrote:
Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another while reading from cassandra. for example i have a timestamp column in cassandra table ,but i want to read it as long and save as parquet file to HDFS , then this cutom codecs will be usefull instead of doing an additional transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
I had tried with solution provided in the below link but it does not work , any ideas around this will be helpfull.
https://stackoverflow.com/questions/40363611/adding-custom-codec-to-cassandraconnector/49750791?noredirect=1#comment86515847_49750791
Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login