You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Serkan TAS <Se...@enerjisa.com> on 2017/12/25 08:16:57 UTC
Which kafka client to use with spark streaming
Hi,
Working on spark 2.2.0 cluster and 1.0 kafka brokers.
I was using the library
"org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
and had lots of problems during streaming process then downgraded to
"org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.2.0"
And i know there is also another path which is using kafka-clients jars which has the latest version of 1.0.0
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.0.0</version>
</dependency>
I am confused which path is the right one
Thanks…
________________________________
Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.
This communication may contain information that is legally privileged, confidential or exempt from disclosure. If you are not the intended recipient, please note that any dissemination, distribution, or copying of this communication is strictly prohibited. Anyone who receives this message in error should notify the sender immediately by telephone or by return communication and delete it from his or her computer. Only the person who has sent this message is responsible for its content.
RE: Which kafka client to use with spark streaming
Posted by Serkan TAS <Se...@enerjisa.com>.
Kafka Clients are blocking spark streaming jobs and after a time streaming job queue increases.
-----Original Message-----
From: Cody Koeninger [mailto:cody@koeninger.org]
Sent: Tuesday, December 26, 2017 6:47 PM
To: Diogo Munaro Vieira <di...@corp.globo.com>
Cc: Serkan TAS <Se...@enerjisa.com>; user <us...@spark.apache.org>
Subject: Re: Which kafka client to use with spark streaming
Do not add a dependency on kafka-clients, the spark-streaming-kafka library has appropriate transitive dependencies.
Either version of the spark-streaming-kafka library should work with
1.0 brokers; what problems were you having?
On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira <di...@corp.globo.com> wrote:
> Hey Serkan, it depends of your Kafka version... Is it 0.8.2?
>
> Em 25 de dez de 2017 06:17, "Serkan TAS" <Se...@enerjisa.com> escreveu:
>>
>> Hi,
>>
>>
>>
>> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>>
>>
>>
>> I was using the library
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>>
>>
>>
>> and had lots of problems during streaming process then downgraded to
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-8_2.11"
>> % "2.2.0"
>>
>>
>>
>> And i know there is also another path which is using kafka-clients
>> jars which has the latest version of 1.0.0
>>
>>
>>
>> <!--
>> https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
>>
>> <dependency>
>>
>> <groupId>org.apache.kafka</groupId>
>>
>> <artifactId>kafka-clients</artifactId>
>>
>> <version>1.0.0</version>
>>
>> </dependency>
>>
>>
>>
>> I am confused which path is the right one
>>
>>
>>
>> Thanks…
>>
>>
>>
>>
>>
>>
>> ________________________________
>>
>> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken
>> bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu
>> iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan
>> kişi, bu durumu derhal gönderene telefonla ya da e-posta ile
>> bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden
>> yalnızca iletiyi gönderen kişi sorumludur.
>>
>> This communication may contain information that is legally
>> privileged, confidential or exempt from disclosure. If you are not
>> the intended recipient, please note that any dissemination,
>> distribution, or copying of this communication is strictly
>> prohibited. Anyone who receives this message in error should notify
>> the sender immediately by telephone or by return communication and
>> delete it from his or her computer. Only the person who has sent this message is responsible for its content.
________________________________
Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur.
This communication may contain information that is legally privileged, confidential or exempt from disclosure. If you are not the intended recipient, please note that any dissemination, distribution, or copying of this communication is strictly prohibited. Anyone who receives this message in error should notify the sender immediately by telephone or by return communication and delete it from his or her computer. Only the person who has sent this message is responsible for its content.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Which kafka client to use with spark streaming
Posted by Cody Koeninger <co...@koeninger.org>.
Do not add a dependency on kafka-clients, the spark-streaming-kafka
library has appropriate transitive dependencies.
Either version of the spark-streaming-kafka library should work with
1.0 brokers; what problems were you having?
On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira
<di...@corp.globo.com> wrote:
> Hey Serkan, it depends of your Kafka version... Is it 0.8.2?
>
> Em 25 de dez de 2017 06:17, "Serkan TAS" <Se...@enerjisa.com> escreveu:
>>
>> Hi,
>>
>>
>>
>> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>>
>>
>>
>> I was using the library
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>>
>>
>>
>> and had lots of problems during streaming process then downgraded to
>>
>> "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" %
>> "2.2.0"
>>
>>
>>
>> And i know there is also another path which is using kafka-clients jars
>> which has the latest version of 1.0.0
>>
>>
>>
>> <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
>>
>> <dependency>
>>
>> <groupId>org.apache.kafka</groupId>
>>
>> <artifactId>kafka-clients</artifactId>
>>
>> <version>1.0.0</version>
>>
>> </dependency>
>>
>>
>>
>> I am confused which path is the right one
>>
>>
>>
>> Thanks…
>>
>>
>>
>>
>>
>>
>> ________________________________
>>
>> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler
>> içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak
>> ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal
>> gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından
>> silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi
>> sorumludur.
>>
>> This communication may contain information that is legally privileged,
>> confidential or exempt from disclosure. If you are not the intended
>> recipient, please note that any dissemination, distribution, or copying of
>> this communication is strictly prohibited. Anyone who receives this message
>> in error should notify the sender immediately by telephone or by return
>> communication and delete it from his or her computer. Only the person who
>> has sent this message is responsible for its content.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Which kafka client to use with spark streaming
Posted by Diogo Munaro Vieira <di...@corp.globo.com>.
Hey Serkan, it depends of your Kafka version... Is it 0.8.2?
Em 25 de dez de 2017 06:17, "Serkan TAS" <Se...@enerjisa.com> escreveu:
> Hi,
>
>
>
> Working on spark 2.2.0 cluster and 1.0 kafka brokers.
>
>
>
> I was using the library
>
> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
>
>
>
> and had lots of problems during streaming process then downgraded to
>
> "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" %
> "2.2.0"
>
>
>
> And i know there is also another path which is using kafka-clients jars
> which has the latest version of 1.0.0
>
>
>
> <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
>
> <dependency>
>
> <groupId>org.apache.kafka</groupId>
>
> <artifactId>kafka-clients</artifactId>
>
> <version>1.0.0</version>
>
> </dependency>
>
>
>
> I am confused which path is the right one
>
>
>
> Thanks…
>
>
>
>
>
> ------------------------------
>
> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler
> içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi
> çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu
> derhal gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından
> silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi
> sorumludur.
>
> This communication may contain information that is legally privileged,
> confidential or exempt from disclosure. If you are not the intended
> recipient, please note that any dissemination, distribution, or copying of
> this communication is strictly prohibited. Anyone who receives this message
> in error should notify the sender immediately by telephone or by return
> communication and delete it from his or her computer. Only the person who
> has sent this message is responsible for its content.
>