You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by أنس الليثي <de...@gmail.com> on 2016/05/04 13:17:03 UTC
Re: Spark and Kafka direct approach problem
NoSuchMethodError usually appears because of a difference in the library
versions.
Check the version of the libraries you downloaded, the version of spark,
the version of Kafka.
On 4 May 2016 at 16:18, Luca Ferrari <fe...@live.it> wrote:
> Hi,
>
> I’m new on Apache Spark and I’m trying to run the Spark Streaming + Kafka
> Integration Direct Approach example (JavaDirectKafkaWordCount.java).
>
> I’ve downloaded all the libraries but when I try to run I get this error
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
>
> at kafka.api.RequestKeys$.<init>(RequestKeys.scala:48)
>
> at kafka.api.RequestKeys$.<clinit>(RequestKeys.scala)
>
> at kafka.api.TopicMetadataRequest.<init>(TopicMetadataRequest.scala:55)
>
> at
> org.apache.spark.streaming.kafka.KafkaCluster.getPartitionMetadata(KafkaCluster.scala:122)
>
> at
> org.apache.spark.streaming.kafka.KafkaCluster.getPartitions(KafkaCluster.scala:112)
>
> at
> org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:211)
>
> at
> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:484)
>
> at
> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:607)
>
> at
> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
> at it.unimi.di.luca.SimpleApp.main(SimpleApp.java:53)
>
> Any suggestions?
>
> Cheers
> Luca
>
>
--
Anas Rabei
Senior Software Developer
Mubasher.info
anas.rabei@mubasher.info
Re: Spark and Kafka direct approach problem
Posted by Mich Talebzadeh <mi...@gmail.com>.
This works
spark 1.61, using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM,
Java 1.8.0_77)
Kafka version 0.9.0.1 using scala-library-2.11.7.jar
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 4 May 2016 at 19:52, Shixiong(Ryan) Zhu <sh...@databricks.com> wrote:
> It's because the Scala version of Spark and the Scala version of Kafka
> don't match. Please check them.
>
> On Wed, May 4, 2016 at 6:17 AM, أنس الليثي <de...@gmail.com> wrote:
>
>> NoSuchMethodError usually appears because of a difference in the library
>> versions.
>>
>> Check the version of the libraries you downloaded, the version of spark,
>> the version of Kafka.
>>
>> On 4 May 2016 at 16:18, Luca Ferrari <fe...@live.it> wrote:
>>
>>> Hi,
>>>
>>> I’m new on Apache Spark and I’m trying to run the Spark Streaming +
>>> Kafka Integration Direct Approach example (JavaDirectKafkaWordCount.java
>>> ).
>>>
>>> I’ve downloaded all the libraries but when I try to run I get this error
>>>
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
>>>
>>> at kafka.api.RequestKeys$.<init>(RequestKeys.scala:48)
>>>
>>> at kafka.api.RequestKeys$.<clinit>(RequestKeys.scala)
>>>
>>> at kafka.api.TopicMetadataRequest.<init>(TopicMetadataRequest.scala:55)
>>>
>>> at
>>> org.apache.spark.streaming.kafka.KafkaCluster.getPartitionMetadata(KafkaCluster.scala:122)
>>>
>>> at
>>> org.apache.spark.streaming.kafka.KafkaCluster.getPartitions(KafkaCluster.scala:112)
>>>
>>> at
>>> org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:211)
>>>
>>> at
>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:484)
>>>
>>> at
>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:607)
>>>
>>> at
>>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
>>> at it.unimi.di.luca.SimpleApp.main(SimpleApp.java:53)
>>>
>>> Any suggestions?
>>>
>>> Cheers
>>> Luca
>>>
>>>
>>
>>
>>
>> --
>> Anas Rabei
>> Senior Software Developer
>> Mubasher.info
>> anas.rabei@mubasher.info
>>
>
>
Re: Spark and Kafka direct approach problem
Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
It's because the Scala version of Spark and the Scala version of Kafka
don't match. Please check them.
On Wed, May 4, 2016 at 6:17 AM, أنس الليثي <de...@gmail.com> wrote:
> NoSuchMethodError usually appears because of a difference in the library
> versions.
>
> Check the version of the libraries you downloaded, the version of spark,
> the version of Kafka.
>
> On 4 May 2016 at 16:18, Luca Ferrari <fe...@live.it> wrote:
>
>> Hi,
>>
>> I’m new on Apache Spark and I’m trying to run the Spark Streaming +
>> Kafka Integration Direct Approach example (JavaDirectKafkaWordCount.java
>> ).
>>
>> I’ve downloaded all the libraries but when I try to run I get this error
>>
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
>>
>> at kafka.api.RequestKeys$.<init>(RequestKeys.scala:48)
>>
>> at kafka.api.RequestKeys$.<clinit>(RequestKeys.scala)
>>
>> at kafka.api.TopicMetadataRequest.<init>(TopicMetadataRequest.scala:55)
>>
>> at
>> org.apache.spark.streaming.kafka.KafkaCluster.getPartitionMetadata(KafkaCluster.scala:122)
>>
>> at
>> org.apache.spark.streaming.kafka.KafkaCluster.getPartitions(KafkaCluster.scala:112)
>>
>> at
>> org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:211)
>>
>> at
>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:484)
>>
>> at
>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:607)
>>
>> at
>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
>> at it.unimi.di.luca.SimpleApp.main(SimpleApp.java:53)
>>
>> Any suggestions?
>>
>> Cheers
>> Luca
>>
>>
>
>
>
> --
> Anas Rabei
> Senior Software Developer
> Mubasher.info
> anas.rabei@mubasher.info
>