You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Koert Kuipers <ko...@tresata.com> on 2017/01/28 05:34:18 UTC
possible bug or inconsistency in kafka-clients
hello all,
i just wanted to point out a potential issue in kafka-clients 0.10.1.1
i was using spark-sql-kafka-0-10, which is spark structured streaming
integration for kafka. it depends on kafka-clients 0.10.0.1 but since my
kafka servers are 0.10.1.1 i decided to upgrade kafka-clients to 0.10.1.1
also. i was not able to read from kafka in spark reliably. the issue seemed
to be that the kafka consumer got the latest offsets wrong. after
downgrading kafka-clients back to 0.10.0.1 it all worked correctly again.
did the behavior of KafkaConsumer.seekToEnd change between 0.10.0.1 and
0.10.1.1?
for the original discussion see here:
https://www.mail-archive.com/user@spark.apache.org/msg61290.html
i think the relevant code in spark is here:
https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L399
best,
koert
Re: possible bug or inconsistency in kafka-clients
Posted by Koert Kuipers <ko...@tresata.com>.
that sounds like it. thanks!
On Sat, Jan 28, 2017 at 12:49 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Could this be the same issue as the one reported here?
> https://issues.apache.org/jira/browse/KAFKA-4547
>
> --Vahid
>
>
>
>
>
> From: Koert Kuipers <ko...@tresata.com>
> To: users@kafka.apache.org
> Date: 01/27/2017 09:34 PM
> Subject: possible bug or inconsistency in kafka-clients
>
>
>
> hello all,
>
> i just wanted to point out a potential issue in kafka-clients 0.10.1.1
>
> i was using spark-sql-kafka-0-10, which is spark structured streaming
> integration for kafka. it depends on kafka-clients 0.10.0.1 but since my
> kafka servers are 0.10.1.1 i decided to upgrade kafka-clients to 0.10.1.1
> also. i was not able to read from kafka in spark reliably. the issue
> seemed
> to be that the kafka consumer got the latest offsets wrong. after
> downgrading kafka-clients back to 0.10.0.1 it all worked correctly again.
>
> did the behavior of KafkaConsumer.seekToEnd change between 0.10.0.1 and
> 0.10.1.1?
>
> for the original discussion see here:
> https://www.mail-archive.com/user@spark.apache.org/msg61290.html
>
>
> i think the relevant code in spark is here:
> https://github.com/apache/spark/blob/master/external/
> kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/
> KafkaSource.scala#L399
>
>
> best,
> koert
>
>
>
>
>
Re: possible bug or inconsistency in kafka-clients
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Could this be the same issue as the one reported here?
https://issues.apache.org/jira/browse/KAFKA-4547
--Vahid
From: Koert Kuipers <ko...@tresata.com>
To: users@kafka.apache.org
Date: 01/27/2017 09:34 PM
Subject: possible bug or inconsistency in kafka-clients
hello all,
i just wanted to point out a potential issue in kafka-clients 0.10.1.1
i was using spark-sql-kafka-0-10, which is spark structured streaming
integration for kafka. it depends on kafka-clients 0.10.0.1 but since my
kafka servers are 0.10.1.1 i decided to upgrade kafka-clients to 0.10.1.1
also. i was not able to read from kafka in spark reliably. the issue
seemed
to be that the kafka consumer got the latest offsets wrong. after
downgrading kafka-clients back to 0.10.0.1 it all worked correctly again.
did the behavior of KafkaConsumer.seekToEnd change between 0.10.0.1 and
0.10.1.1?
for the original discussion see here:
https://www.mail-archive.com/user@spark.apache.org/msg61290.html
i think the relevant code in spark is here:
https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L399
best,
koert