Incorrect auto-commited offsets

[Tom van den Berge <to...@gmail.com>]

I'm using a (java) consumer with default configuration, so auto-commit is
enabled. The consumer is reading from 5 partitions of a single topic. The
consumer processes one message at a time (synchronously). Sometimes, large
numbers of messages are posted to the topic, and the consumer will have to
work for one or more hours to process all messages. This is not a problem.

Sometimes, if there is a significant lag, I notice that the committed
offsets that Kafka reports for the consumer, are very strange, or even
impossible. It goes up and down. I've created a graph for the
current-offset and log-end-offset for one partition: https://ibb.co/frJ3eb.
The dotted line is the log-end-offset (seems fine), and the solid line is
the current-offset, as reported by the kafka-consumer-groups.sh tool.

As you can clearly see, the current offset suddenly drops by a few
thousand, then gradually increases, but shows a number of spikes, until it
reaches the original value again, then suddenly increases twice by a few
thousand. Obviously the current offset should never decrease, but also the
sudden increases are also impossible; the consumer is technically not able
to process the messages this fast.

The same behaviour is shown for the other partitions of this consumer, and
also for other consumers of the same topic.
I'm using Kafka 0.10.0.1 and kafka-client 0.10.0.1.

Does anyone have an idea on what is happening here?

Regards,
Tom

Re: Incorrect auto-commited offsets

[Guozhang Wang <wa...@gmail.com>]

Hello Tom,

It is indeed a bit weird, I remember someone else have reported similar
issues before. Do you still have the client-side DEBUG logs when this
happens? I'd like to exclude or narrow down the root cause of whether a
reset indeed happens on the consumer, and if yes, it should contains the
following entries:

"

Fetch offset {} is out of range for partition {}, resetting offset ...

Resetting offset for partition {} to offset {}

"


Guozhang



On Thu, Nov 30, 2017 at 6:11 AM, Tom van den Berge <
tom.vandenberge@gmail.com> wrote:

> I'm using a (java) consumer with default configuration, so auto-commit is
> enabled. The consumer is reading from 5 partitions of a single topic. The
> consumer processes one message at a time (synchronously). Sometimes, large
> numbers of messages are posted to the topic, and the consumer will have to
> work for one or more hours to process all messages. This is not a problem.
>
> Sometimes, if there is a significant lag, I notice that the committed
> offsets that Kafka reports for the consumer, are very strange, or even
> impossible. It goes up and down. I've created a graph for the
> current-offset and log-end-offset for one partition: https://ibb.co/frJ3eb
> .
> The dotted line is the log-end-offset (seems fine), and the solid line is
> the current-offset, as reported by the kafka-consumer-groups.sh tool.
>
> As you can clearly see, the current offset suddenly drops by a few
> thousand, then gradually increases, but shows a number of spikes, until it
> reaches the original value again, then suddenly increases twice by a few
> thousand. Obviously the current offset should never decrease, but also the
> sudden increases are also impossible; the consumer is technically not able
> to process the messages this fast.
>
> The same behaviour is shown for the other partitions of this consumer, and
> also for other consumers of the same topic.
> I'm using Kafka 0.10.0.1 and kafka-client 0.10.0.1.
>
> Does anyone have an idea on what is happening here?
>
> Regards,
> Tom
>



-- 
-- Guozhang