You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by vi...@wipro.com on 2015/12/29 13:40:33 UTC

Spark 1.5.2 compatible spark-cassandra-connector

All,
What is the compatible spark-cassandra-connector for spark 1.5.2? I can only find the latest connector version spark-cassandra-connector_2.10-1.5.0-M3 which has dependency with 1.5.1 spark. Can we use the same for 1.5.2? Any classpath issues needs to be handled or any jars needs to be excluded while packaging the application jar?

http://central.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.10/1.5.0-M3/spark-cassandra-connector_2.10-1.5.0-M3.pom

Regards,
Vivek M
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com

RE: Spark 1.5.2 compatible spark-cassandra-connector

Posted by vi...@wipro.com.
Thank you mwy and Sun for your response. Yes basic things are working for me using this connector(guava issue was encountered earlier but with proper exclusion of old version we have resolved it).

The current issue is strange one �C we have a kafka-spark-cassandra streaming job in spark. The all jobs are working fine in local cluster (local lab) using 1.3.0 spark. We are trying the same setup in Google cloud platform in 1.3.0 the jobs are up but it does not seems to be processing the kafka messages. Using 1.5.2 spark with 1.5.0-M3 connector + 0.8.2.2 kafka I am able to make most of the jobs work but one of them is processing only 1 out of 50 messages from kafka.
We have wrote a test program(scala we use) to parse data from kafka stream �C it is working fine and always receives the messages. It connects to Cassandra gets some data works fine and prints the input data till that time,  but when I enable the join,map,reduce login , it starts to miss the message(not printing above worked lines also). Please find the lines added below, once I add the commented line to the job , it does not print any incoming data while I am able to print till then. Any ideas how to debug?

val cqlOfferSummaryRdd = ssc.sc.cql3Cassandra[OfferSummary](casOfferSummary)
              .map(summary => (summary.listingId, (summary.offerId, summary.redeem, summary.viewed, summary.reserved)))


/**
              //RDD -> ((listingId, searchId), Iterable(offerId, redeemCount, viewCount))
              val directListingOfferSummary = directViews.transform(Rdd => { Rdd.join(cqlOfferSummaryRdd,10)}) //RDD -> ((listingId), (Direct, (offerId, redeemCount, viewCount)))
              .map(rdd => ((rdd._1, rdd._2._1.searchId, rdd._2._2._1), (rdd._2._2._2, rdd._2._2._3, rdd._2._2._4))) //RDD -> ((listingId, searchId, offerId), (redeemCount, viewCount))
              .reduceByKey((a, b) => (a._1 + b._1, a._2 + b._2, a._3 + b._3))
              .map(rdd => ((rdd._1._1, rdd._1._2), (rdd._1._3, rdd._2._1, rdd._2._2, rdd._2._3))).groupByKey(10) //RDD -> ((listingId, searchId), Iterable(offerId, redeemCount, viewCount))
              directListingOfferSummary.print()

**/

Regards,
Vivek

From: mwy [mailto:wenyao.ma@dewmobile.net]
Sent: 30 December 2015 08:27
To: fightfate@163.com; Vivek Meghanathan (WT01 - NEP) <vi...@wipro.com>; user <us...@spark.apache.org>
Subject: Re: Spark 1.5.2 compatible spark-cassandra-connector

2.10-1.5.0-M3 & spark 1.5.2 work for me. The jar is built by sbt-assembly.

Just for reference.

������: "fightfate@163.com<ma...@163.com>" <fi...@163.com>>
����: Wednesday, December 30, 2015 at 10:22
��: "vivek.meghanathan@wipro.com<ma...@wipro.com>" <vi...@wipro.com>>, user <us...@spark.apache.org>>
����: Re: Spark 1.5.2 compatible spark-cassandra-connector

Hi, Vivek M
I had ever tried 1.5.x spark-cassandra connector and indeed encounter some classpath issues, mainly for the guaua dependency.
I believe that can be solved by some maven config, but have not tried that yet.

Best,
Sun.

________________________________
fightfate@163.com<ma...@163.com>

From: vivek.meghanathan@wipro.com<ma...@wipro.com>
Date: 2015-12-29 20:40
To: user@spark.apache.org<ma...@spark.apache.org>
Subject: Spark 1.5.2 compatible spark-cassandra-connector
All,
What is the compatible spark-cassandra-connector for spark 1.5.2? I can only find the latest connector version spark-cassandra-connector_2.10-1.5.0-M3 which has dependency with 1.5.1 spark. Can we use the same for 1.5.2? Any classpath issues needs to be handled or any jars needs to be excluded while packaging the application jar?

http://central.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.10/1.5.0-M3/spark-cassandra-connector_2.10-1.5.0-M3.pom

Regards,
Vivek M
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com<http://www.wipro.com>
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com

Re: Spark 1.5.2 compatible spark-cassandra-connector

Posted by mwy <we...@dewmobile.net>.
2.10-1.5.0-M3 & spark 1.5.2 work for me. The jar is built by sbt-assembly.

Just for reference.

发件人:  "fightfate@163.com" <fi...@163.com>
日期:  Wednesday, December 30, 2015 at 10:22
至:  "vivek.meghanathan@wipro.com" <vi...@wipro.com>, user
<us...@spark.apache.org>
主题:  Re: Spark 1.5.2 compatible spark-cassandra-connector

Hi, Vivek M
I had ever tried 1.5.x spark-cassandra connector and indeed encounter some
classpath issues, mainly for the guaua dependency.
I believe that can be solved by some maven config, but have not tried that
yet. 

Best,
Sun.


fightfate@163.com
>  
> From: vivek.meghanathan@wipro.com
> Date: 2015-12-29 20:40
> To: user@spark.apache.org
> Subject: Spark 1.5.2 compatible spark-cassandra-connector
> All,
> What is the compatible spark-cassandra-connector for spark 1.5.2? I can only
> find the latest connector version spark-cassandra-connector_2.10-1.5.0-M3
> which has dependency with 1.5.1 spark. Can we use the same for 1.5.2? Any
> classpath issues needs to be handled or any jars needs to be excluded while
> packaging the application jar?
>  
> http://central.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2
> .10/1.5.0-M3/spark-cassandra-connector_2.10-1.5.0-M3.pom
>  
> Regards,
> Vivek M
> The information contained in this electronic message and any attachments to
> this message are intended for the exclusive use of the addressee(s) and may
> contain proprietary, confidential or privileged information. If you are not
> the intended recipient, you should not disseminate, distribute or copy this
> e-mail. Please notify the sender immediately and destroy all copies of this
> message and any attachments. WARNING: Computer viruses can be transmitted via
> email. The recipient should check this email and any attachments for the
> presence of viruses. The company accepts no liability for any damage caused by
> any virus transmitted by this email. www.wipro.com

Re: Spark 1.5.2 compatible spark-cassandra-connector

Posted by "fightfate@163.com" <fi...@163.com>.
Hi, Vivek M
I had ever tried 1.5.x spark-cassandra connector and indeed encounter some classpath issues, mainly for the guaua dependency. 
I believe that can be solved by some maven config, but have not tried that yet. 

Best,
Sun.



fightfate@163.com
 
From: vivek.meghanathan@wipro.com
Date: 2015-12-29 20:40
To: user@spark.apache.org
Subject: Spark 1.5.2 compatible spark-cassandra-connector
All,
What is the compatible spark-cassandra-connector for spark 1.5.2? I can only find the latest connector version spark-cassandra-connector_2.10-1.5.0-M3 which has dependency with 1.5.1 spark. Can we use the same for 1.5.2? Any classpath issues needs to be handled or any jars needs to be excluded while packaging the application jar?
 
http://central.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.10/1.5.0-M3/spark-cassandra-connector_2.10-1.5.0-M3.pom
 
Regards,
Vivek M
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com