You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pulkit Manchanda <pu...@gmail.com> on 2018/07/09 21:23:59 UTC
Spark structured streaming + kafka
Hi,
I am trying to do structured streaming with kafka as source.
I am unable to get pass of this code.
val df = spark
.readStream
.format("org.apache.spark.sql.kafka010.KafkaSourceProvider")
.option("kafka.bootstrap.servers", "localhost:8082")
.option("subscribe", "jsontest")
.load()
The error i am getting is
Exception in thread "main" java.lang.ClassNotFoundException: Failed to find
data source: org.apache.spark.sql.kafka010.KafkaSourceProvider. Please find
packages at http://spark.apache.org/third-party-projects.html
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
I have the dependency in SBT
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10
libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" %
"2.3.0" % "provided"
I have tried on spark-shell too, but same error
scala> :require /Users/pulkit/Downloads/spark-sql-kafka-0-10_2.11-2.3.0.jar
The path '/Users/pulkit/Downloads/spark-sql-kafka-0-10_2.11-2.3.0.jar'
cannot be loaded, because existing classpath entries conflict.
scala> val df =
spark.readStream.format("kafka").option("kafka.bootstrap.servers",
"localhost:9092").option("subscribe", "topicName").load()
java.lang.ClassNotFoundException: Failed to find data source: kafka. Please
find packages at http://spark.apache.org/third-party-projects.html
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
at
org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:159)
... 49 elided
I looked on google but didnt get any useful info.
Does anyone faced same issues? and have a solution?
Thanks
Pulkit
Re: Spark structured streaming + kafka
Posted by Pulkit Manchanda <pu...@gmail.com>.
You were right.
It worked.
Thanks
Pulkit
On Mon, Jul 9, 2018 at 5:29 PM, Liam Clarke <li...@adscale.co.nz>
wrote:
> In your SBT script you've specified that your Kafka data source jar is
> provided.
>
> Please read the documentation of dependency scopes.
> https://maven.apache.org/guides/introduction/introduction-to-dependency-
> mechanism.html#Dependency_Scope
>
> Kind regards,
>
> Liam Clarke
>
> On Tue, 10 Jul. 2018, 9:24 am Pulkit Manchanda, <pu...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am trying to do structured streaming with kafka as source.
> > I am unable to get pass of this code.
> >
> > val df = spark
> > .readStream
> > .format("org.apache.spark.sql.kafka010.KafkaSourceProvider")
> > .option("kafka.bootstrap.servers", "localhost:8082")
> > .option("subscribe", "jsontest")
> > .load()
> >
> > The error i am getting is
> >
> > Exception in thread "main" java.lang.ClassNotFoundException: Failed to
> find
> > data source: org.apache.spark.sql.kafka010.KafkaSourceProvider. Please
> find
> > packages at http://spark.apache.org/third-party-projects.html
> > at
> >
> > org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(
> DataSource.scala:635)
> >
> > I have the dependency in SBT
> >
> > libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
> > //
> > https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10
> > libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" %
> > "2.3.0" % "provided"
> >
> >
> > I have tried on spark-shell too, but same error
> >
> >
> > scala> :require /Users/pulkit/Downloads/spark-
> sql-kafka-0-10_2.11-2.3.0.jar
> >
> > The path '/Users/pulkit/Downloads/spark-sql-kafka-0-10_2.11-2.3.0.jar'
> > cannot be loaded, because existing classpath entries conflict.
> >
> >
> > scala> val df =
> > spark.readStream.format("kafka").option("kafka.bootstrap.servers",
> > "localhost:9092").option("subscribe", "topicName").load()
> >
> > java.lang.ClassNotFoundException: Failed to find data source: kafka.
> Please
> > find packages at http://spark.apache.org/third-party-projects.html
> >
> > at
> >
> > org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(
> DataSource.scala:635)
> >
> > at
> >
> > org.apache.spark.sql.streaming.DataStreamReader.
> load(DataStreamReader.scala:159)
> >
> > ... 49 elided
> >
> >
> > I looked on google but didnt get any useful info.
> > Does anyone faced same issues? and have a solution?
> >
> > Thanks
> > Pulkit
> >
>
Re: Spark structured streaming + kafka
Posted by Liam Clarke <li...@adscale.co.nz>.
In your SBT script you've specified that your Kafka data source jar is
provided.
Please read the documentation of dependency scopes.
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
Kind regards,
Liam Clarke
On Tue, 10 Jul. 2018, 9:24 am Pulkit Manchanda, <pu...@gmail.com>
wrote:
> Hi,
>
> I am trying to do structured streaming with kafka as source.
> I am unable to get pass of this code.
>
> val df = spark
> .readStream
> .format("org.apache.spark.sql.kafka010.KafkaSourceProvider")
> .option("kafka.bootstrap.servers", "localhost:8082")
> .option("subscribe", "jsontest")
> .load()
>
> The error i am getting is
>
> Exception in thread "main" java.lang.ClassNotFoundException: Failed to find
> data source: org.apache.spark.sql.kafka010.KafkaSourceProvider. Please find
> packages at http://spark.apache.org/third-party-projects.html
> at
>
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
>
> I have the dependency in SBT
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
> //
> https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10
> libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" %
> "2.3.0" % "provided"
>
>
> I have tried on spark-shell too, but same error
>
>
> scala> :require /Users/pulkit/Downloads/spark-sql-kafka-0-10_2.11-2.3.0.jar
>
> The path '/Users/pulkit/Downloads/spark-sql-kafka-0-10_2.11-2.3.0.jar'
> cannot be loaded, because existing classpath entries conflict.
>
>
> scala> val df =
> spark.readStream.format("kafka").option("kafka.bootstrap.servers",
> "localhost:9092").option("subscribe", "topicName").load()
>
> java.lang.ClassNotFoundException: Failed to find data source: kafka. Please
> find packages at http://spark.apache.org/third-party-projects.html
>
> at
>
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
>
> at
>
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:159)
>
> ... 49 elided
>
>
> I looked on google but didnt get any useful info.
> Does anyone faced same issues? and have a solution?
>
> Thanks
> Pulkit
>