You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Madabhattula Rajesh Kumar <mr...@gmail.com> on 2016/02/03 13:37:21 UTC

spark-cassandra

Hi,

I am using Spark Jobserver to submit the jobs. I am using spark-cassandra
connector to connect to Cassandra. I am getting below exception through
spak jobserver.

If I submit the job through *Spark-Submit *command it is working fine,.

Please let me know how to solve this issue


Exception in thread "pool-1-thread-1" java.lang.NoSuchMethodError:
com.datastax.driver.core.TableMetadata.getIndexes()Ljava/util/List;
    at
com.datastax.spark.connector.cql.Schema$.getIndexMap(Schema.scala:193)
    at
com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchPartitionKey(Schema.scala:197)
    at
com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchTables$1$2.apply(Schema.scala:239)
    at
com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchTables$1$2.apply(Schema.scala:238)
    at
scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
    at
scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
    at
scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
    at
scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
    at
com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchTables$1(Schema.scala:238)
    at
com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1$2.apply(Schema.scala:247)
    at
com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1$2.apply(Schema.scala:246)
    at
scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
    at
scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
    at
scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
    at
scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
    at
com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1(Schema.scala:246)
    at
com.datastax.spark.connector.cql.Schema$$anonfun$fromCassandra$1.apply(Schema.scala:252)
    at
com.datastax.spark.connector.cql.Schema$$anonfun$fromCassandra$1.apply(Schema.scala:249)
    at
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withClusterDo$1.apply(CassandraConnector.scala:121)
    at
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withClusterDo$1.apply(CassandraConnector.scala:120)
    at
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:110)
    at
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:109)
    at
com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:139)
    at
com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
    at
com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120)
    at
com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:249)
    at
com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:263)
    at
com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
    at
com.cisco.ss.etl.utils.ETLHelper$class.persistBackupConfigDevicesData(ETLHelper.scala:79)
    at com.cisco.ss.etl.Main$.persistBackupConfigDevicesData(Main.scala:13)
    at
com.cisco.ss.etl.utils.ETLHelper$class.persistByBacthes(ETLHelper.scala:43)
    at com.cisco.ss.etl.Main$.persistByBacthes(Main.scala:13)
    at com.cisco.ss.etl.Main$$anonfun$runJob$3.apply(Main.scala:48)
    at com.cisco.ss.etl.Main$$anonfun$runJob$3.apply(Main.scala:45)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at com.cisco.ss.etl.Main$.runJob(Main.scala:45)
    at com.cisco.ss.etl.Main$.runJob(Main.scala:13)
    at
spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:274)
    at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Regards,
Rajesh

RE: spark-cassandra

Posted by Mohammed Guller <mo...@glassbeam.com>.
Another thing to check is what version of the Spark-Cassandra-Connector the Spark Job server passing to the workers. It looks like when you use Spark-submit, you are sending the correct SCC jar, but the Spark Job server may be using a different one.

Mohammed
Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>

From: Gerard Maas [mailto:gerard.maas@gmail.com]
Sent: Wednesday, February 3, 2016 4:56 AM
To: Madabhattula Rajesh Kumar
Cc: user@spark.apache.org
Subject: Re: spark-cassandra

NoSuchMethodError usually refers to a version conflict. Probably your job was built against a higher version of the cassandra connector than what's available on the run time.
Check that the versions are aligned.

-kr, Gerard.

On Wed, Feb 3, 2016 at 1:37 PM, Madabhattula Rajesh Kumar <mr...@gmail.com>> wrote:
Hi,
I am using Spark Jobserver to submit the jobs. I am using spark-cassandra connector to connect to Cassandra. I am getting below exception through spak jobserver.
If I submit the job through Spark-Submit command it is working fine,.
Please let me know how to solve this issue


Exception in thread "pool-1-thread-1" java.lang.NoSuchMethodError: com.datastax.driver.core.TableMetadata.getIndexes()Ljava/util/List;
    at com.datastax.spark.connector.cql.Schema$.getIndexMap(Schema.scala:193)
    at com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchPartitionKey(Schema.scala:197)
    at com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchTables$1$2.apply(Schema.scala:239)
    at com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchTables$1$2.apply(Schema.scala:238)
    at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
    at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
    at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
    at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
    at com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchTables$1(Schema.scala:238)
    at com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1$2.apply(Schema.scala:247)
    at com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1$2.apply(Schema.scala:246)
    at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
    at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
    at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
    at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
    at com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1(Schema.scala:246)
    at com.datastax.spark.connector.cql.Schema$$anonfun$fromCassandra$1.apply(Schema.scala:252)
    at com.datastax.spark.connector.cql.Schema$$anonfun$fromCassandra$1.apply(Schema.scala:249)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withClusterDo$1.apply(CassandraConnector.scala:121)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withClusterDo$1.apply(CassandraConnector.scala:120)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:110)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:109)
    at com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:139)
    at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
    at com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120)
    at com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:249)
    at com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:263)
    at com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
    at com.cisco.ss.etl.utils.ETLHelper$class.persistBackupConfigDevicesData(ETLHelper.scala:79)
    at com.cisco.ss.etl.Main$.persistBackupConfigDevicesData(Main.scala:13)
    at com.cisco.ss.etl.utils.ETLHelper$class.persistByBacthes(ETLHelper.scala:43)
    at com.cisco.ss.etl.Main$.persistByBacthes(Main.scala:13)
    at com.cisco.ss.etl.Main$$anonfun$runJob$3.apply(Main.scala:48)
    at com.cisco.ss.etl.Main$$anonfun$runJob$3.apply(Main.scala:45)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at com.cisco.ss.etl.Main$.runJob(Main.scala:45)
    at com.cisco.ss.etl.Main$.runJob(Main.scala:13)
    at spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:274)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Regards,
Rajesh

Re: spark-cassandra

Posted by Gerard Maas <ge...@gmail.com>.
NoSuchMethodError usually refers to a version conflict. Probably your job
was built against a higher version of the cassandra connector than what's
available on the run time.
Check that the versions are aligned.

-kr, Gerard.

On Wed, Feb 3, 2016 at 1:37 PM, Madabhattula Rajesh Kumar <
mrajaforu@gmail.com> wrote:

> Hi,
>
> I am using Spark Jobserver to submit the jobs. I am using spark-cassandra
> connector to connect to Cassandra. I am getting below exception through
> spak jobserver.
>
> If I submit the job through *Spark-Submit *command it is working fine,.
>
> Please let me know how to solve this issue
>
>
> Exception in thread "pool-1-thread-1" java.lang.NoSuchMethodError:
> com.datastax.driver.core.TableMetadata.getIndexes()Ljava/util/List;
>     at
> com.datastax.spark.connector.cql.Schema$.getIndexMap(Schema.scala:193)
>     at
> com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchPartitionKey(Schema.scala:197)
>     at
> com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchTables$1$2.apply(Schema.scala:239)
>     at
> com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchTables$1$2.apply(Schema.scala:238)
>     at
> scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
>     at
> scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
>     at
> scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
>     at
> scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
>     at
> com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchTables$1(Schema.scala:238)
>     at
> com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1$2.apply(Schema.scala:247)
>     at
> com.datastax.spark.connector.cql.Schema$$anonfun$com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1$2.apply(Schema.scala:246)
>     at
> scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
>     at
> scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
>     at
> scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
>     at
> scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
>     at
> com.datastax.spark.connector.cql.Schema$.com$datastax$spark$connector$cql$Schema$$fetchKeyspaces$1(Schema.scala:246)
>     at
> com.datastax.spark.connector.cql.Schema$$anonfun$fromCassandra$1.apply(Schema.scala:252)
>     at
> com.datastax.spark.connector.cql.Schema$$anonfun$fromCassandra$1.apply(Schema.scala:249)
>     at
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withClusterDo$1.apply(CassandraConnector.scala:121)
>     at
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withClusterDo$1.apply(CassandraConnector.scala:120)
>     at
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:110)
>     at
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:109)
>     at
> com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:139)
>     at
> com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
>     at
> com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120)
>     at
> com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:249)
>     at
> com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:263)
>     at
> com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
>     at
> com.cisco.ss.etl.utils.ETLHelper$class.persistBackupConfigDevicesData(ETLHelper.scala:79)
>     at com.cisco.ss.etl.Main$.persistBackupConfigDevicesData(Main.scala:13)
>     at
> com.cisco.ss.etl.utils.ETLHelper$class.persistByBacthes(ETLHelper.scala:43)
>     at com.cisco.ss.etl.Main$.persistByBacthes(Main.scala:13)
>     at com.cisco.ss.etl.Main$$anonfun$runJob$3.apply(Main.scala:48)
>     at com.cisco.ss.etl.Main$$anonfun$runJob$3.apply(Main.scala:45)
>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>     at com.cisco.ss.etl.Main$.runJob(Main.scala:45)
>     at com.cisco.ss.etl.Main$.runJob(Main.scala:13)
>     at
> spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:274)
>     at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>     at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
>
> Regards,
> Rajesh
>