You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ryan Weald <ry...@weald.com> on 2013/10/12 00:14:14 UTC
Kafka dependency issues
When pulling the latest (0.8.0-incubating) dependencies from maven I get a
compile error when I try and write a basic streaming job. The error
produces a large stack trace which you can find here(
https://gist.github.com/rweald/6942840) which seems to point to a lack of a
Kafka dependency.
Has anyone else had this problem and is there a documented solution?
Cheers
-Ryan Weald
Re: Kafka dependency issues
Posted by Matei Zaharia <ma...@gmail.com>.
Yeah, good point. I think that in the future we'll move to having smaller "plugin" projects for various input sources (e.g. spark-kafka, spark-rabbitmq).
Matei
On Oct 15, 2013, at 9:50 AM, Ryan Weald <ry...@weald.com> wrote:
> Thanks for the response. It is interesting that you must manually specify the Kafka dependency given it is required for even the most basic streaming job to compile, regardless of whether you have any intention of using the Kafka input source. Might be worth considering a way to exclude the entire kafka input source from the default Spark Streaming dependency if the goal is to keep size down and you don't want to confuse new adopters who aren't using Kafka as part of their tech stack.
>
> -Ryan
>
>
> On Sat, Oct 12, 2013 at 10:52 AM, Matei Zaharia <ma...@gmail.com> wrote:
> Hi Ryan,
>
> Spark Streaming ships with a special version of the Kafka 0.7.2 client that we ported to Scala 2.9, and you need to add that as a JAR explicitly in your project. The JAR is in streaming/lib/org/apache/kafka/kafka/0.7.2-spark/kafka-0.7.2-spark.jar under Spark. The streaming/lib directory is also designed to act as a Maven repo -- see streaming/pom.xml for how to reference it from Maven.
>
> Matei
>
> On Oct 11, 2013, at 3:14 PM, Ryan Weald <ry...@weald.com> wrote:
>
>> When pulling the latest (0.8.0-incubating) dependencies from maven I get a compile error when I try and write a basic streaming job. The error produces a large stack trace which you can find here(https://gist.github.com/rweald/6942840) which seems to point to a lack of a Kafka dependency.
>>
>> Has anyone else had this problem and is there a documented solution?
>>
>> Cheers
>>
>> -Ryan Weald
>
>
Re: Kafka dependency issues
Posted by Ryan Weald <ry...@weald.com>.
Thanks for the response. It is interesting that you must manually specify
the Kafka dependency given it is required for even the most basic streaming
job to compile, regardless of whether you have any intention of using the
Kafka input source. Might be worth considering a way to exclude the entire
kafka input source from the default Spark Streaming dependency if the goal
is to keep size down and you don't want to confuse new adopters who aren't
using Kafka as part of their tech stack.
-Ryan
On Sat, Oct 12, 2013 at 10:52 AM, Matei Zaharia <ma...@gmail.com>wrote:
> Hi Ryan,
>
> Spark Streaming ships with a special version of the Kafka 0.7.2 client
> that we ported to Scala 2.9, and you need to add that as a JAR explicitly
> in your project. The JAR is
> in streaming/lib/org/apache/kafka/kafka/0.7.2-spark/kafka-0.7.2-spark.jar
> under Spark. The streaming/lib directory is also designed to act as a Maven
> repo -- see streaming/pom.xml for how to reference it from Maven.
>
> Matei
>
> On Oct 11, 2013, at 3:14 PM, Ryan Weald <ry...@weald.com> wrote:
>
> When pulling the latest (0.8.0-incubating) dependencies from maven I get a
> compile error when I try and write a basic streaming job. The error
> produces a large stack trace which you can find here(
> https://gist.github.com/rweald/6942840) which seems to point to a lack of
> a Kafka dependency.
>
> Has anyone else had this problem and is there a documented solution?
>
> Cheers
>
> -Ryan Weald
>
>
>
Re: Kafka dependency issues
Posted by Matei Zaharia <ma...@gmail.com>.
Hi Ryan,
Spark Streaming ships with a special version of the Kafka 0.7.2 client that we ported to Scala 2.9, and you need to add that as a JAR explicitly in your project. The JAR is in streaming/lib/org/apache/kafka/kafka/0.7.2-spark/kafka-0.7.2-spark.jar under Spark. The streaming/lib directory is also designed to act as a Maven repo -- see streaming/pom.xml for how to reference it from Maven.
Matei
On Oct 11, 2013, at 3:14 PM, Ryan Weald <ry...@weald.com> wrote:
> When pulling the latest (0.8.0-incubating) dependencies from maven I get a compile error when I try and write a basic streaming job. The error produces a large stack trace which you can find here(https://gist.github.com/rweald/6942840) which seems to point to a lack of a Kafka dependency.
>
> Has anyone else had this problem and is there a documented solution?
>
> Cheers
>
> -Ryan Weald
Re: Kafka dependency issues
Posted by Paul Snively <ps...@icloud.com>.
The Kafka dependency is not included in Spark Streaming to keep size down. Your project needs to express the dependency itself.
Hope this helps,
Paul
On Oct 11, 2013, at 3:14 PM, Ryan Weald wrote:
> When pulling the latest (0.8.0-incubating) dependencies from maven I get a compile error when I try and write a basic streaming job. The error produces a large stack trace which you can find here(https://gist.github.com/rweald/6942840) which seems to point to a lack of a Kafka dependency.
>
> Has anyone else had this problem and is there a documented solution?
>
> Cheers
>
> -Ryan Weald