KafkaSpout v1.1.1 : FetchOffset vs CommitOffset

[chandan singh <ck...@gmail.com>]

Hi

I wanted to get my understanding about CommitOffset/FetchOffset in
KafkaSout (v1.1.1) confirmed.

For Kafka consumer, commitOffset and fetchOffset are same. In other words,
Kafka consumer reads from last committedOffset and hence their official
documentation (
https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html)
has the following note in bold "*Note: The committed offset should always
be the offset of the next message that your application will read.*"

KafkaSpout does not follow that guideline. Obviously, what KafkaSpout does,
by treating commitOffset as one less than the next fetchOffset, sounds much
better but this makes some kafka tools misbehave with offsets committed by
KafkaSpout. Also, a migration from custom spout to KafkaSpout becomes less
trivial.

Please confirm if my understanding is correct and leave your comments.

Regards
Chandan

Re: KafkaSpout v1.1.1 : FetchOffset vs CommitOffset

[Stig Rohde Døssing <sr...@apache.org>]

You are right that there's currently a bug, there's a pending fix here
https://github.com/apache/storm/pull/2181. Feedback on it is appreciated :)

2017-09-18 7:34 GMT+02:00 chandan singh <ck...@gmail.com>:

> Hi
>
> I wanted to get my understanding about CommitOffset/FetchOffset in
> KafkaSout (v1.1.1) confirmed.
>
> For Kafka consumer, commitOffset and fetchOffset are same. In other words,
> Kafka consumer reads from last committedOffset and hence their official
> documentation (
> https://kafka.apache.org/0110/javadoc/index.html?org/apache/
> kafka/clients/consumer/KafkaConsumer.html)
> has the following note in bold "*Note: The committed offset should always
> be the offset of the next message that your application will read.*"
>
> KafkaSpout does not follow that guideline. Obviously, what KafkaSpout does,
> by treating commitOffset as one less than the next fetchOffset, sounds much
> better but this makes some kafka tools misbehave with offsets committed by
> KafkaSpout. Also, a migration from custom spout to KafkaSpout becomes less
> trivial.
>
> Please confirm if my understanding is correct and leave your comments.
>
> Regards
> Chandan
>