Spark 1.5 Streaming + Kafka 0.9.0

[Pavel Sýkora <pa...@seznam.cz>]

Hi,



According to the Spark docs, Spark Streaming 1.5 (and 1.6) is compatible 
with Kafka 0.8.2.1 (Direct Kafka API). Nevertheless, I need to use Kafka 
0.9.0 with Spark 1.5.x streaming.




I tried to use Kafka 0.9.0 as both source and output of Spark 1.5 Streaming,
but it seems it works well. 




Does anybody have the same experience? Or does anybody have some problems 
integrating Spark Streaming 1.5 or 1.6 with Kafka 0.9.0?




Thanks in advance,




Pavel  

Re: Spark 1.5 Streaming + Kafka 0.9.0

[Cody Koeninger <co...@koeninger.org>]

0.9 brokers should be backwards compatible with 0.8 simple consumer, which
is what the direct stream uses.  If you're finding any problems, please
report them.

There's work underway to add 0.9 consumer support to spark, at
https://issues.apache.org/jira/browse/SPARK-12177


On Wed, Feb 3, 2016 at 9:56 AM, Pavel Sýkora <pa...@seznam.cz> wrote:

> Hi,
>
> According to the Spark docs, Spark Streaming 1.5 (and 1.6) is compatible
> with Kafka 0.8.2.1 (Direct Kafka API). Nevertheless, I need to use Kafka
> 0.9.0 with Spark 1.5.x streaming.
>
> I tried to use Kafka 0.9.0 as both source and output of Spark 1.5
> Streaming, but it seems it works well.
>
> Does anybody have the same experience? Or does anybody have some problems
> integrating Spark Streaming 1.5 or 1.6 with Kafka 0.9.0?
>
> Thanks in advance,
>
> Pavel
>