You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by satish chandra j <js...@gmail.com> on 2015/08/10 14:44:40 UTC
Spark Cassandra Connector issue
HI All,
Please help me to fix Spark Cassandra Connector issue, find the details
below
*Command:*
dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
--jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
*Error:*
WARN 2015-08-10 06:33:35 org.apache.spark.util.Utils: Service 'SparkUI'
could not bind on port 4040. Attempting port 4041.
Exception in thread "main" java.lang.NoSuchMethodError:
com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions;
at HelloWorld$.main(HelloWorld.scala:29)
at HelloWorld.main(HelloWorld.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
*Code:*
*import* *org.apache*.spark.SparkContext
*import* *org.apache*.spark.SparkContext._
*import* *org.apache*.spark.SparkConf
*import* *org.apache*.spark.rdd.JdbcRDD
*import* *com.datastax*.spark.connector._
*import* com.datastax.spark.connector.cql.CassandraConnector
*import* com.datastax.bdp.spark.DseSparkConfHelper._
*import* java.sql.{Connection, DriverManager, ResultSet, PreparedStatement,
SQLException, Statement}
*object* HelloWorld {
*def* main(args: Array[String]) {
*def* createSparkContext() = {
*val** myJar =
*getClass.getProtectionDomain.getCodeSource.getLocation.getPath
*val* conf = *new* SparkConf().set("spark.cassandra.connection.host",
"10.246.43.15")
.setAppName("First Spark App")
.setMaster("local")
* .s*etJars(Array(myJar))
.set("cassandra.username", "username")
.set("cassandra.password", "password")
.forDse
*new* SparkContext(conf)
}
*val* sc = createSparkContext()
*val* user="hkonak0"
*val** pass=*"Winter18"
Class.forName("org.postgresql.Driver").newInstance
*val* url = "jdbc:postgresql://gptester:5432/db_test"
*val* myRDD27 = *new* JdbcRDD( sc, ()=>
DriverManager.getConnection(url,user,pass),"select * from
wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) => {(r.getInt(
"alarm_type_code"),r.getString("language_code"),r.getString(
"alrm_type_cd_desc"))})
myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns(
"alarm_type_code","language_code","alrm_type_cd_desc"))
println(myRDD27.count())
println(myRDD27.first)
sc.stop()
sys.exit()
}
}
*POM XML:*
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>*hadoop*-client</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.scala-*lang*</groupId>
<artifactId>*scala*-library</artifactId>
<version>2.10.5</version>
</dependency>
<dependency>
<groupId>*junit*</groupId>
<artifactId>*junit*</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>*dse*</artifactId>
<version>4.7.2</version>
<scope>system</scope>
<systemPath>C:\workspace\*etl*\*lib*\dse.jar</
systemPath>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-*cassandra*-connector-java_2.10</artifactId>
<version>1.1.1</version>
</dependency>
</dependencies>
Please let me know if any further details required to analyze the issue
Regards,
Satish Chandra
Re: Spark Cassandra Connector issue
Posted by satish chandra j <js...@gmail.com>.
HI,
Can we apply *saveToCassandra method to a JdbcRDD *
Code:
*Code:*
*import* *org.apache*.spark.SparkContext
*import* *org.apache*.spark.SparkContext._
*import* *org.apache*.spark.SparkConf
*import* *org.apache*.spark.rdd.JdbcRDD
*import* *com.datastax*.spark.connector._
*import* com.datastax.spark.connector.cql.CassandraConnector
*import* com.datastax.bdp.spark.DseSparkConfHelper._
*import* java.sql.{Connection, DriverManager, ResultSet, PreparedStatement,
SQLException, Statement}
*object* HelloWorld {
*def* main(args: Array[String]) {
*def* createSparkContext() = {
*val* conf = *new* SparkConf().set("spark.cassandra.connection.host"
, "10.246.43.15")
.setAppName("First Spark App")
.setMaster("local")
.set("cassandra.username", "username")
.set("cassandra.password", "password")
.forDse
*new* SparkContext(conf)
}
*val* sc = createSparkContext()
*val* user="hkonak0"
*val** pass=*"Winter18"
Class.forName("org.postgresql.Driver").newInstance
*val* url = "jdbc:postgresql://gptester:5432/db_test"
*val* myRDD27 = *new* JdbcRDD( sc, ()=>
DriverManager.getConnection(url,user,pass),"select * from
wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) => {(r.getInt(
"alarm_type_code"),r.getString("language_code"),r.getString(
"alrm_type_cd_desc"))})
* myRDD27.saveToCassandra(*
*"keyspace","arm_typ_txt",SomeColumns("alarm_type_code","language_code","alrm_type_cd_desc"))*
println(myRDD27.count())
println(myRDD27.first)
sc.stop()
sys.exit()
}
}
I am getting the error as below:
*Exception in thread "main" java.lang.NoSuchMethodError:
com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions*
Regards,
Satish Chandra
On Tue, Aug 11, 2015 at 9:23 AM, satish chandra j <js...@gmail.com>
wrote:
> HI All,
> I have tried Commands as mentioned below but still it is errors
>
> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
> --jars /home/missingmerch/
> postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
> cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0.
> 1-SNAPSHOT.jar
>
> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
> --jars /home/missingmerch/
> postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
> cassandra-connector-java_2.10-1.1.1.jar,/home/missingmerch/etl-0.0.
> 1-SNAPSHOT.jar
>
> I understand only problem with the way I provide list of jar file in the
> command, if anybody using Datastax Enterprise could please provide thier
> inputs to get this issue resolved
>
> Thanks for your support
>
> Satish Chandra
>
>
> On Mon, Aug 10, 2015 at 7:16 PM, Dean Wampler <de...@gmail.com>
> wrote:
>
>> I don't know if DSE changed spark-submit, but you have to use a
>> comma-separated list of jars to --jars. It probably looked for HelloWorld
>> in the second one, the dse.jar file. Do this:
>>
>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>> --jars /home/missingmerch/
>> postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
>> cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0.
>> 1-SNAPSHOT.jar
>>
>> I also removed the extra "//". Or put "file:" in front of them so they
>> are proper URLs. Note the "snapshot" jar isn't in the --jars list. I assume
>> that's where HelloWorld is found. Confusing, yes it is...
>>
>> dean
>>
>> Dean Wampler, Ph.D.
>> Author: Programming Scala, 2nd Edition
>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>> Typesafe <http://typesafe.com>
>> @deanwampler <http://twitter.com/deanwampler>
>> http://polyglotprogramming.com
>>
>> On Mon, Aug 10, 2015 at 8:23 AM, satish chandra j <
>> jsatishchandra@gmail.com> wrote:
>>
>>> Hi,
>>> Thanks for quick input, now I am getting class not found error
>>>
>>> *Command:*
>>>
>>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>>> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>> ///home/missingmerch/dse.jar
>>> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>
>>>
>>> *Error:*
>>>
>>> java.lang.ClassNotFoundException: HelloWorld
>>>
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>
>>> at java.lang.Class.forName0(Native Method)
>>>
>>> at java.lang.Class.forName(Class.java:270)
>>>
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
>>>
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>>
>>> Previously I could fix the issue by changing the order of arguments
>>> passing in DSE command line interface but now I am not sure why the issue
>>> again
>>>
>>> Please let me know if still I am missing anything in my Command as
>>> mentioned above(as insisted I have added dse.jar and
>>> spark-cassandra-connector-java_2.10.1.1.1.jar)
>>>
>>>
>>> Thanks for support
>>>
>>>
>>> Satish Chandra
>>>
>>> On Mon, Aug 10, 2015 at 6:19 PM, Dean Wampler <de...@gmail.com>
>>> wrote:
>>>
>>>> Add the other Cassandra dependencies (dse.jar,
>>>> spark-cassandra-connect-java_2.10) to your --jars argument on the command
>>>> line.
>>>>
>>>> Dean Wampler, Ph.D.
>>>> Author: Programming Scala, 2nd Edition
>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>>> Typesafe <http://typesafe.com>
>>>> @deanwampler <http://twitter.com/deanwampler>
>>>> http://polyglotprogramming.com
>>>>
>>>> On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j <
>>>> jsatishchandra@gmail.com> wrote:
>>>>
>>>>> HI All,
>>>>> Please help me to fix Spark Cassandra Connector issue, find the
>>>>> details below
>>>>>
>>>>> *Command:*
>>>>>
>>>>> dse spark-submit --master spark://10.246.43.15:7077 --class
>>>>> HelloWorld --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>>>
>>>>>
>>>>> *Error:*
>>>>>
>>>>>
>>>>> WARN 2015-08-10 06:33:35 org.apache.spark.util.Utils: Service
>>>>> 'SparkUI' could not bind on port 4040. Attempting port 4041.
>>>>>
>>>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>>> com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions;
>>>>>
>>>>> at HelloWorld$.main(HelloWorld.scala:29)
>>>>>
>>>>> at HelloWorld.main(HelloWorld.scala)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>>>>>
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>>>
>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>
>>>>>
>>>>> *Code:*
>>>>>
>>>>> *import* *org.apache*.spark.SparkContext
>>>>>
>>>>> *import* *org.apache*.spark.SparkContext._
>>>>>
>>>>> *import* *org.apache*.spark.SparkConf
>>>>>
>>>>> *import* *org.apache*.spark.rdd.JdbcRDD
>>>>>
>>>>> *import* *com.datastax*.spark.connector._
>>>>>
>>>>> *import* com.datastax.spark.connector.cql.CassandraConnector
>>>>>
>>>>> *import* com.datastax.bdp.spark.DseSparkConfHelper._
>>>>>
>>>>> *import* java.sql.{Connection, DriverManager, ResultSet,
>>>>> PreparedStatement, SQLException, Statement}
>>>>>
>>>>> *object* HelloWorld {
>>>>>
>>>>> *def* main(args: Array[String]) {
>>>>>
>>>>> *def* createSparkContext() = {
>>>>>
>>>>> *val** myJar = *getClass.getProtectionDomain.getCodeSource.getLocation.getPath
>>>>>
>>>>>
>>>>> *val* conf = *new* SparkConf().set(
>>>>> "spark.cassandra.connection.host", "10.246.43.15")
>>>>>
>>>>> .setAppName("First Spark App")
>>>>>
>>>>> .setMaster("local")
>>>>>
>>>>> * .s*etJars(Array(myJar))
>>>>>
>>>>> .set("cassandra.username", "username")
>>>>>
>>>>> .set("cassandra.password", "password")
>>>>>
>>>>> .forDse
>>>>>
>>>>> *new* SparkContext(conf)
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> *val* sc = createSparkContext()
>>>>>
>>>>> *val* user="hkonak0"
>>>>>
>>>>> *val** pass=*"Winter18"
>>>>>
>>>>> Class.forName("org.postgresql.Driver").newInstance
>>>>>
>>>>> *val* url = "jdbc:postgresql://gptester:5432/db_test"
>>>>>
>>>>> *val* myRDD27 = *new* JdbcRDD( sc, ()=>
>>>>> DriverManager.getConnection(url,user,pass),"select * from
>>>>> wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) =>
>>>>> {(r.getInt("alarm_type_code"),r.getString("language_code"
>>>>> ),r.getString("alrm_type_cd_desc"))})
>>>>>
>>>>> myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns(
>>>>> "alarm_type_code","language_code","alrm_type_cd_desc"))
>>>>>
>>>>> println(myRDD27.count())
>>>>>
>>>>> println(myRDD27.first)
>>>>>
>>>>> sc.stop()
>>>>>
>>>>> sys.exit()
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> *POM XML:*
>>>>>
>>>>>
>>>>> <dependencies>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>org.apache.spark</groupId>
>>>>>
>>>>> <artifactId>spark-core_2.10</artifactId>
>>>>>
>>>>> <version>1.2.2</version>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>org.apache.hadoop</groupId>
>>>>>
>>>>> <artifactId>*hadoop*-client</artifactId>
>>>>>
>>>>> <version>1.2.1</version>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>org.scala-*lang*</groupId>
>>>>>
>>>>> <artifactId>*scala*-library</artifactId>
>>>>>
>>>>> <version>2.10.5</version>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>*junit*</groupId>
>>>>>
>>>>> <artifactId>*junit*</artifactId>
>>>>>
>>>>> <version>3.8.1</version>
>>>>>
>>>>> <scope>test</scope>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>com.datastax.dse</groupId>
>>>>>
>>>>> <artifactId>*dse*</artifactId>
>>>>>
>>>>> <version>4.7.2</version>
>>>>>
>>>>> <scope>system</scope>
>>>>>
>>>>> <systemPath>C:\workspace\*etl*\*lib*\dse.jar</
>>>>> systemPath>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>com.datastax.spark</groupId>
>>>>>
>>>>> <artifactId>spark-*cassandra*-connector-java_2.10</
>>>>> artifactId>
>>>>>
>>>>> <version>1.1.1</version>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> </dependencies>
>>>>>
>>>>>
>>>>> Please let me know if any further details required to analyze the issue
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Satish Chandra
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Spark Cassandra Connector issue
Posted by satish chandra j <js...@gmail.com>.
HI All,
I have tried Commands as mentioned below but still it is errors
dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
--jars /home/missingmerch/
postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0.
1-SNAPSHOT.jar
dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
--jars /home/missingmerch/
postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
cassandra-connector-java_2.10-1.1.1.jar,/home/missingmerch/etl-0.0.
1-SNAPSHOT.jar
I understand only problem with the way I provide list of jar file in the
command, if anybody using Datastax Enterprise could please provide thier
inputs to get this issue resolved
Thanks for your support
Satish Chandra
On Mon, Aug 10, 2015 at 7:16 PM, Dean Wampler <de...@gmail.com> wrote:
> I don't know if DSE changed spark-submit, but you have to use a
> comma-separated list of jars to --jars. It probably looked for HelloWorld
> in the second one, the dse.jar file. Do this:
>
> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
> --jars /home/missingmerch/
> postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
> cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0.
> 1-SNAPSHOT.jar
>
> I also removed the extra "//". Or put "file:" in front of them so they are
> proper URLs. Note the "snapshot" jar isn't in the --jars list. I assume
> that's where HelloWorld is found. Confusing, yes it is...
>
> dean
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Mon, Aug 10, 2015 at 8:23 AM, satish chandra j <
> jsatishchandra@gmail.com> wrote:
>
>> Hi,
>> Thanks for quick input, now I am getting class not found error
>>
>> *Command:*
>>
>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>> ///home/missingmerch/dse.jar
>> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>
>>
>> *Error:*
>>
>> java.lang.ClassNotFoundException: HelloWorld
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>
>> at java.lang.Class.forName0(Native Method)
>>
>> at java.lang.Class.forName(Class.java:270)
>>
>> at
>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
>>
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
>> Previously I could fix the issue by changing the order of arguments
>> passing in DSE command line interface but now I am not sure why the issue
>> again
>>
>> Please let me know if still I am missing anything in my Command as
>> mentioned above(as insisted I have added dse.jar and
>> spark-cassandra-connector-java_2.10.1.1.1.jar)
>>
>>
>> Thanks for support
>>
>>
>> Satish Chandra
>>
>> On Mon, Aug 10, 2015 at 6:19 PM, Dean Wampler <de...@gmail.com>
>> wrote:
>>
>>> Add the other Cassandra dependencies (dse.jar,
>>> spark-cassandra-connect-java_2.10) to your --jars argument on the command
>>> line.
>>>
>>> Dean Wampler, Ph.D.
>>> Author: Programming Scala, 2nd Edition
>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>> Typesafe <http://typesafe.com>
>>> @deanwampler <http://twitter.com/deanwampler>
>>> http://polyglotprogramming.com
>>>
>>> On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j <
>>> jsatishchandra@gmail.com> wrote:
>>>
>>>> HI All,
>>>> Please help me to fix Spark Cassandra Connector issue, find the details
>>>> below
>>>>
>>>> *Command:*
>>>>
>>>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>>>> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>>
>>>>
>>>> *Error:*
>>>>
>>>>
>>>> WARN 2015-08-10 06:33:35 org.apache.spark.util.Utils: Service
>>>> 'SparkUI' could not bind on port 4040. Attempting port 4041.
>>>>
>>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>> com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions;
>>>>
>>>> at HelloWorld$.main(HelloWorld.scala:29)
>>>>
>>>> at HelloWorld.main(HelloWorld.scala)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>>>>
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>>
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>
>>>>
>>>> *Code:*
>>>>
>>>> *import* *org.apache*.spark.SparkContext
>>>>
>>>> *import* *org.apache*.spark.SparkContext._
>>>>
>>>> *import* *org.apache*.spark.SparkConf
>>>>
>>>> *import* *org.apache*.spark.rdd.JdbcRDD
>>>>
>>>> *import* *com.datastax*.spark.connector._
>>>>
>>>> *import* com.datastax.spark.connector.cql.CassandraConnector
>>>>
>>>> *import* com.datastax.bdp.spark.DseSparkConfHelper._
>>>>
>>>> *import* java.sql.{Connection, DriverManager, ResultSet,
>>>> PreparedStatement, SQLException, Statement}
>>>>
>>>> *object* HelloWorld {
>>>>
>>>> *def* main(args: Array[String]) {
>>>>
>>>> *def* createSparkContext() = {
>>>>
>>>> *val** myJar = *getClass.getProtectionDomain.getCodeSource.getLocation.getPath
>>>>
>>>>
>>>> *val* conf = *new* SparkConf().set(
>>>> "spark.cassandra.connection.host", "10.246.43.15")
>>>>
>>>> .setAppName("First Spark App")
>>>>
>>>> .setMaster("local")
>>>>
>>>> * .s*etJars(Array(myJar))
>>>>
>>>> .set("cassandra.username", "username")
>>>>
>>>> .set("cassandra.password", "password")
>>>>
>>>> .forDse
>>>>
>>>> *new* SparkContext(conf)
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> *val* sc = createSparkContext()
>>>>
>>>> *val* user="hkonak0"
>>>>
>>>> *val** pass=*"Winter18"
>>>>
>>>> Class.forName("org.postgresql.Driver").newInstance
>>>>
>>>> *val* url = "jdbc:postgresql://gptester:5432/db_test"
>>>>
>>>> *val* myRDD27 = *new* JdbcRDD( sc, ()=>
>>>> DriverManager.getConnection(url,user,pass),"select * from
>>>> wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) =>
>>>> {(r.getInt("alarm_type_code"),r.getString("language_code"),r.getString(
>>>> "alrm_type_cd_desc"))})
>>>>
>>>> myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns(
>>>> "alarm_type_code","language_code","alrm_type_cd_desc"))
>>>>
>>>> println(myRDD27.count())
>>>>
>>>> println(myRDD27.first)
>>>>
>>>> sc.stop()
>>>>
>>>> sys.exit()
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> *POM XML:*
>>>>
>>>>
>>>> <dependencies>
>>>>
>>>> <dependency>
>>>>
>>>> <groupId>org.apache.spark</groupId>
>>>>
>>>> <artifactId>spark-core_2.10</artifactId>
>>>>
>>>> <version>1.2.2</version>
>>>>
>>>> </dependency>
>>>>
>>>> <dependency>
>>>>
>>>> <groupId>org.apache.hadoop</groupId>
>>>>
>>>> <artifactId>*hadoop*-client</artifactId>
>>>>
>>>> <version>1.2.1</version>
>>>>
>>>> </dependency>
>>>>
>>>> <dependency>
>>>>
>>>> <groupId>org.scala-*lang*</groupId>
>>>>
>>>> <artifactId>*scala*-library</artifactId>
>>>>
>>>> <version>2.10.5</version>
>>>>
>>>> </dependency>
>>>>
>>>> <dependency>
>>>>
>>>> <groupId>*junit*</groupId>
>>>>
>>>> <artifactId>*junit*</artifactId>
>>>>
>>>> <version>3.8.1</version>
>>>>
>>>> <scope>test</scope>
>>>>
>>>> </dependency>
>>>>
>>>> <dependency>
>>>>
>>>> <groupId>com.datastax.dse</groupId>
>>>>
>>>> <artifactId>*dse*</artifactId>
>>>>
>>>> <version>4.7.2</version>
>>>>
>>>> <scope>system</scope>
>>>>
>>>> <systemPath>C:\workspace\*etl*\*lib*\dse.jar</
>>>> systemPath>
>>>>
>>>> </dependency>
>>>>
>>>> <dependency>
>>>>
>>>> <groupId>com.datastax.spark</groupId>
>>>>
>>>> <artifactId>spark-*cassandra*-connector-java_2.10</
>>>> artifactId>
>>>>
>>>> <version>1.1.1</version>
>>>>
>>>> </dependency>
>>>>
>>>> </dependencies>
>>>>
>>>>
>>>> Please let me know if any further details required to analyze the issue
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Satish Chandra
>>>>
>>>
>>>
>>
>
Re: Spark Cassandra Connector issue
Posted by Dean Wampler <de...@gmail.com>.
I don't know if DSE changed spark-submit, but you have to use a
comma-separated list of jars to --jars. It probably looked for HelloWorld
in the second one, the dse.jar file. Do this:
dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
--jars /home/missingmerch/
postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0.
1-SNAPSHOT.jar
I also removed the extra "//". Or put "file:" in front of them so they are
proper URLs. Note the "snapshot" jar isn't in the --jars list. I assume
that's where HelloWorld is found. Confusing, yes it is...
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Mon, Aug 10, 2015 at 8:23 AM, satish chandra j <js...@gmail.com>
wrote:
> Hi,
> Thanks for quick input, now I am getting class not found error
>
> *Command:*
>
> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
> ///home/missingmerch/dse.jar
> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>
>
> *Error:*
>
> java.lang.ClassNotFoundException: HelloWorld
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> at java.lang.Class.forName0(Native Method)
>
> at java.lang.Class.forName(Class.java:270)
>
> at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> Previously I could fix the issue by changing the order of arguments
> passing in DSE command line interface but now I am not sure why the issue
> again
>
> Please let me know if still I am missing anything in my Command as
> mentioned above(as insisted I have added dse.jar and
> spark-cassandra-connector-java_2.10.1.1.1.jar)
>
>
> Thanks for support
>
>
> Satish Chandra
>
> On Mon, Aug 10, 2015 at 6:19 PM, Dean Wampler <de...@gmail.com>
> wrote:
>
>> Add the other Cassandra dependencies (dse.jar,
>> spark-cassandra-connect-java_2.10) to your --jars argument on the command
>> line.
>>
>> Dean Wampler, Ph.D.
>> Author: Programming Scala, 2nd Edition
>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>> Typesafe <http://typesafe.com>
>> @deanwampler <http://twitter.com/deanwampler>
>> http://polyglotprogramming.com
>>
>> On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j <
>> jsatishchandra@gmail.com> wrote:
>>
>>> HI All,
>>> Please help me to fix Spark Cassandra Connector issue, find the details
>>> below
>>>
>>> *Command:*
>>>
>>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>>> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>
>>>
>>> *Error:*
>>>
>>>
>>> WARN 2015-08-10 06:33:35 org.apache.spark.util.Utils: Service 'SparkUI'
>>> could not bind on port 4040. Attempting port 4041.
>>>
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions;
>>>
>>> at HelloWorld$.main(HelloWorld.scala:29)
>>>
>>> at HelloWorld.main(HelloWorld.scala)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>>>
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>>
>>> *Code:*
>>>
>>> *import* *org.apache*.spark.SparkContext
>>>
>>> *import* *org.apache*.spark.SparkContext._
>>>
>>> *import* *org.apache*.spark.SparkConf
>>>
>>> *import* *org.apache*.spark.rdd.JdbcRDD
>>>
>>> *import* *com.datastax*.spark.connector._
>>>
>>> *import* com.datastax.spark.connector.cql.CassandraConnector
>>>
>>> *import* com.datastax.bdp.spark.DseSparkConfHelper._
>>>
>>> *import* java.sql.{Connection, DriverManager, ResultSet,
>>> PreparedStatement, SQLException, Statement}
>>>
>>> *object* HelloWorld {
>>>
>>> *def* main(args: Array[String]) {
>>>
>>> *def* createSparkContext() = {
>>>
>>> *val** myJar = *getClass.getProtectionDomain.getCodeSource.getLocation.getPath
>>>
>>>
>>> *val* conf = *new* SparkConf().set(
>>> "spark.cassandra.connection.host", "10.246.43.15")
>>>
>>> .setAppName("First Spark App")
>>>
>>> .setMaster("local")
>>>
>>> * .s*etJars(Array(myJar))
>>>
>>> .set("cassandra.username", "username")
>>>
>>> .set("cassandra.password", "password")
>>>
>>> .forDse
>>>
>>> *new* SparkContext(conf)
>>>
>>> }
>>>
>>>
>>>
>>> *val* sc = createSparkContext()
>>>
>>> *val* user="hkonak0"
>>>
>>> *val** pass=*"Winter18"
>>>
>>> Class.forName("org.postgresql.Driver").newInstance
>>>
>>> *val* url = "jdbc:postgresql://gptester:5432/db_test"
>>>
>>> *val* myRDD27 = *new* JdbcRDD( sc, ()=>
>>> DriverManager.getConnection(url,user,pass),"select * from
>>> wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) =>
>>> {(r.getInt("alarm_type_code"),r.getString("language_code"),r.getString(
>>> "alrm_type_cd_desc"))})
>>>
>>> myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns(
>>> "alarm_type_code","language_code","alrm_type_cd_desc"))
>>>
>>> println(myRDD27.count())
>>>
>>> println(myRDD27.first)
>>>
>>> sc.stop()
>>>
>>> sys.exit()
>>>
>>>
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> *POM XML:*
>>>
>>>
>>> <dependencies>
>>>
>>> <dependency>
>>>
>>> <groupId>org.apache.spark</groupId>
>>>
>>> <artifactId>spark-core_2.10</artifactId>
>>>
>>> <version>1.2.2</version>
>>>
>>> </dependency>
>>>
>>> <dependency>
>>>
>>> <groupId>org.apache.hadoop</groupId>
>>>
>>> <artifactId>*hadoop*-client</artifactId>
>>>
>>> <version>1.2.1</version>
>>>
>>> </dependency>
>>>
>>> <dependency>
>>>
>>> <groupId>org.scala-*lang*</groupId>
>>>
>>> <artifactId>*scala*-library</artifactId>
>>>
>>> <version>2.10.5</version>
>>>
>>> </dependency>
>>>
>>> <dependency>
>>>
>>> <groupId>*junit*</groupId>
>>>
>>> <artifactId>*junit*</artifactId>
>>>
>>> <version>3.8.1</version>
>>>
>>> <scope>test</scope>
>>>
>>> </dependency>
>>>
>>> <dependency>
>>>
>>> <groupId>com.datastax.dse</groupId>
>>>
>>> <artifactId>*dse*</artifactId>
>>>
>>> <version>4.7.2</version>
>>>
>>> <scope>system</scope>
>>>
>>> <systemPath>C:\workspace\*etl*\*lib*\dse.jar</
>>> systemPath>
>>>
>>> </dependency>
>>>
>>> <dependency>
>>>
>>> <groupId>com.datastax.spark</groupId>
>>>
>>> <artifactId>spark-*cassandra*-connector-java_2.10</
>>> artifactId>
>>>
>>> <version>1.1.1</version>
>>>
>>> </dependency>
>>>
>>> </dependencies>
>>>
>>>
>>> Please let me know if any further details required to analyze the issue
>>>
>>>
>>> Regards,
>>>
>>> Satish Chandra
>>>
>>
>>
>
Re: Spark Cassandra Connector issue
Posted by satish chandra j <js...@gmail.com>.
Hi,
Thanks for quick input, now I am getting class not found error
*Command:*
dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
--jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
///home/missingmerch/dse.jar
///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
*Error:*
java.lang.ClassNotFoundException: HelloWorld
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Previously I could fix the issue by changing the order of arguments passing
in DSE command line interface but now I am not sure why the issue again
Please let me know if still I am missing anything in my Command as
mentioned above(as insisted I have added dse.jar and
spark-cassandra-connector-java_2.10.1.1.1.jar)
Thanks for support
Satish Chandra
On Mon, Aug 10, 2015 at 6:19 PM, Dean Wampler <de...@gmail.com> wrote:
> Add the other Cassandra dependencies (dse.jar,
> spark-cassandra-connect-java_2.10) to your --jars argument on the command
> line.
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j <
> jsatishchandra@gmail.com> wrote:
>
>> HI All,
>> Please help me to fix Spark Cassandra Connector issue, find the details
>> below
>>
>> *Command:*
>>
>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>
>>
>> *Error:*
>>
>>
>> WARN 2015-08-10 06:33:35 org.apache.spark.util.Utils: Service 'SparkUI'
>> could not bind on port 4040. Attempting port 4041.
>>
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions;
>>
>> at HelloWorld$.main(HelloWorld.scala:29)
>>
>> at HelloWorld.main(HelloWorld.scala)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:606)
>>
>> at
>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>>
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
>> *Code:*
>>
>> *import* *org.apache*.spark.SparkContext
>>
>> *import* *org.apache*.spark.SparkContext._
>>
>> *import* *org.apache*.spark.SparkConf
>>
>> *import* *org.apache*.spark.rdd.JdbcRDD
>>
>> *import* *com.datastax*.spark.connector._
>>
>> *import* com.datastax.spark.connector.cql.CassandraConnector
>>
>> *import* com.datastax.bdp.spark.DseSparkConfHelper._
>>
>> *import* java.sql.{Connection, DriverManager, ResultSet,
>> PreparedStatement, SQLException, Statement}
>>
>> *object* HelloWorld {
>>
>> *def* main(args: Array[String]) {
>>
>> *def* createSparkContext() = {
>>
>> *val** myJar = *getClass.getProtectionDomain.getCodeSource.getLocation.getPath
>>
>>
>> *val* conf = *new* SparkConf().set(
>> "spark.cassandra.connection.host", "10.246.43.15")
>>
>> .setAppName("First Spark App")
>>
>> .setMaster("local")
>>
>> * .s*etJars(Array(myJar))
>>
>> .set("cassandra.username", "username")
>>
>> .set("cassandra.password", "password")
>>
>> .forDse
>>
>> *new* SparkContext(conf)
>>
>> }
>>
>>
>>
>> *val* sc = createSparkContext()
>>
>> *val* user="hkonak0"
>>
>> *val** pass=*"Winter18"
>>
>> Class.forName("org.postgresql.Driver").newInstance
>>
>> *val* url = "jdbc:postgresql://gptester:5432/db_test"
>>
>> *val* myRDD27 = *new* JdbcRDD( sc, ()=>
>> DriverManager.getConnection(url,user,pass),"select * from
>> wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) =>
>> {(r.getInt("alarm_type_code"),r.getString("language_code"),r.getString(
>> "alrm_type_cd_desc"))})
>>
>> myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns(
>> "alarm_type_code","language_code","alrm_type_cd_desc"))
>>
>> println(myRDD27.count())
>>
>> println(myRDD27.first)
>>
>> sc.stop()
>>
>> sys.exit()
>>
>>
>>
>> }
>>
>> }
>>
>>
>>
>> *POM XML:*
>>
>>
>> <dependencies>
>>
>> <dependency>
>>
>> <groupId>org.apache.spark</groupId>
>>
>> <artifactId>spark-core_2.10</artifactId>
>>
>> <version>1.2.2</version>
>>
>> </dependency>
>>
>> <dependency>
>>
>> <groupId>org.apache.hadoop</groupId>
>>
>> <artifactId>*hadoop*-client</artifactId>
>>
>> <version>1.2.1</version>
>>
>> </dependency>
>>
>> <dependency>
>>
>> <groupId>org.scala-*lang*</groupId>
>>
>> <artifactId>*scala*-library</artifactId>
>>
>> <version>2.10.5</version>
>>
>> </dependency>
>>
>> <dependency>
>>
>> <groupId>*junit*</groupId>
>>
>> <artifactId>*junit*</artifactId>
>>
>> <version>3.8.1</version>
>>
>> <scope>test</scope>
>>
>> </dependency>
>>
>> <dependency>
>>
>> <groupId>com.datastax.dse</groupId>
>>
>> <artifactId>*dse*</artifactId>
>>
>> <version>4.7.2</version>
>>
>> <scope>system</scope>
>>
>> <systemPath>C:\workspace\*etl*\*lib*\dse.jar</
>> systemPath>
>>
>> </dependency>
>>
>> <dependency>
>>
>> <groupId>com.datastax.spark</groupId>
>>
>> <artifactId>spark-*cassandra*-connector-java_2.10</
>> artifactId>
>>
>> <version>1.1.1</version>
>>
>> </dependency>
>>
>> </dependencies>
>>
>>
>> Please let me know if any further details required to analyze the issue
>>
>>
>> Regards,
>>
>> Satish Chandra
>>
>
>
Re: Spark Cassandra Connector issue
Posted by Dean Wampler <de...@gmail.com>.
Add the other Cassandra dependencies (dse.jar,
spark-cassandra-connect-java_2.10) to your --jars argument on the command
line.
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j <js...@gmail.com>
wrote:
> HI All,
> Please help me to fix Spark Cassandra Connector issue, find the details
> below
>
> *Command:*
>
> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>
>
> *Error:*
>
>
> WARN 2015-08-10 06:33:35 org.apache.spark.util.Utils: Service 'SparkUI'
> could not bind on port 4040. Attempting port 4041.
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions;
>
> at HelloWorld$.main(HelloWorld.scala:29)
>
> at HelloWorld.main(HelloWorld.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> *Code:*
>
> *import* *org.apache*.spark.SparkContext
>
> *import* *org.apache*.spark.SparkContext._
>
> *import* *org.apache*.spark.SparkConf
>
> *import* *org.apache*.spark.rdd.JdbcRDD
>
> *import* *com.datastax*.spark.connector._
>
> *import* com.datastax.spark.connector.cql.CassandraConnector
>
> *import* com.datastax.bdp.spark.DseSparkConfHelper._
>
> *import* java.sql.{Connection, DriverManager, ResultSet,
> PreparedStatement, SQLException, Statement}
>
> *object* HelloWorld {
>
> *def* main(args: Array[String]) {
>
> *def* createSparkContext() = {
>
> *val** myJar = *getClass.getProtectionDomain.getCodeSource.getLocation.getPath
>
>
> *val* conf = *new* SparkConf().set(
> "spark.cassandra.connection.host", "10.246.43.15")
>
> .setAppName("First Spark App")
>
> .setMaster("local")
>
> * .s*etJars(Array(myJar))
>
> .set("cassandra.username", "username")
>
> .set("cassandra.password", "password")
>
> .forDse
>
> *new* SparkContext(conf)
>
> }
>
>
>
> *val* sc = createSparkContext()
>
> *val* user="hkonak0"
>
> *val** pass=*"Winter18"
>
> Class.forName("org.postgresql.Driver").newInstance
>
> *val* url = "jdbc:postgresql://gptester:5432/db_test"
>
> *val* myRDD27 = *new* JdbcRDD( sc, ()=>
> DriverManager.getConnection(url,user,pass),"select * from
> wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) =>
> {(r.getInt("alarm_type_code"),r.getString("language_code"),r.getString(
> "alrm_type_cd_desc"))})
>
> myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns(
> "alarm_type_code","language_code","alrm_type_cd_desc"))
>
> println(myRDD27.count())
>
> println(myRDD27.first)
>
> sc.stop()
>
> sys.exit()
>
>
>
> }
>
> }
>
>
>
> *POM XML:*
>
>
> <dependencies>
>
> <dependency>
>
> <groupId>org.apache.spark</groupId>
>
> <artifactId>spark-core_2.10</artifactId>
>
> <version>1.2.2</version>
>
> </dependency>
>
> <dependency>
>
> <groupId>org.apache.hadoop</groupId>
>
> <artifactId>*hadoop*-client</artifactId>
>
> <version>1.2.1</version>
>
> </dependency>
>
> <dependency>
>
> <groupId>org.scala-*lang*</groupId>
>
> <artifactId>*scala*-library</artifactId>
>
> <version>2.10.5</version>
>
> </dependency>
>
> <dependency>
>
> <groupId>*junit*</groupId>
>
> <artifactId>*junit*</artifactId>
>
> <version>3.8.1</version>
>
> <scope>test</scope>
>
> </dependency>
>
> <dependency>
>
> <groupId>com.datastax.dse</groupId>
>
> <artifactId>*dse*</artifactId>
>
> <version>4.7.2</version>
>
> <scope>system</scope>
>
> <systemPath>C:\workspace\*etl*\*lib*\dse.jar</
> systemPath>
>
> </dependency>
>
> <dependency>
>
> <groupId>com.datastax.spark</groupId>
>
> <artifactId>spark-*cassandra*-connector-java_2.10</
> artifactId>
>
> <version>1.1.1</version>
>
> </dependency>
>
> </dependencies>
>
>
> Please let me know if any further details required to analyze the issue
>
>
> Regards,
>
> Satish Chandra
>