Blocked consumer - next offset same as current offset

[Stevo Slavić <ss...@gmail.com>]

Hello Apache Kafka community,


I'm using simple consumer with Kafka 0.8.2.2 and noticed that under some
conditions fetch response message set for a partition can contain at least
one (if not all) MessageAndOffset with nextOffset being equal to current
(committed) offset, offset used in fetch request. Not sure how it's related
but I could notice this behavior especially often when I was using new
async producer, and when fetch request was able to fetch several messages
all the way to the end of the partition.

Is this a feature or a bug?


Kind regards,

Stevo Slavic.

Re: Blocked consumer - next offset same as current offset

[Stevo Slavić <ss...@gmail.com>]

Not sure if related, but this line in AbstractFetcherThread (Kafka 0.8.2
branch) looks suspicious
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L122

This logic seems to have been changed significantly via
https://github.com/apache/kafka/commit/79f7cca85e9ed6511ad50cb9412bfa5c8e5b9ddb#diff-2d03a5d4349a23e56c09b4e773055249
for https://issues.apache.org/jira/browse/KAFKA-1461
so in Kafka 0.9 branch it looks like this
https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L132

Have dirty workaround for the moment, not trusting nextOffset info from
MessageAndOffset, transforming response to a new one with custom logic
calculating nextOffset (= offset used in fetch request + index of message
in message set). Will recheck at latest once 0.9 is fully released.

Unwanted behavior seems would get triggered only when consumer lag is of
appropriate size - I guess async producer made this issue more likely,
because of batch sending multiple messages to same partition.

On Wed, Nov 11, 2015 at 9:40 AM, Stevo Slavić <ss...@gmail.com> wrote:

> Hello Apache Kafka community,
>
>
> I'm using simple consumer with Kafka 0.8.2.2 and noticed that under some
> conditions fetch response message set for a partition can contain at least
> one (if not all) MessageAndOffset with nextOffset being equal to current
> (committed) offset, offset used in fetch request. Not sure how it's related
> but I could notice this behavior especially often when I was using new
> async producer, and when fetch request was able to fetch several messages
> all the way to the end of the partition.
>
> Is this a feature or a bug?
>
>
> Kind regards,
>
> Stevo Slavic.
>

Re: Blocked consumer - next offset same as current offset

[Stevo Slavić <ss...@gmail.com>]

Thanks Jun for heads up!

Looked it up in wiki page:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example#id-0.8.0SimpleConsumerExample-ReadingtheData

"Also note that we are explicitly checking that the offset being read is
not less than the offset that we requested. This is needed since if Kafka
is compressing the messages, the fetch request will return an entire
compressed block even if the requested offset isn't the beginning of the
compressed block. Thus a message we saw previously may be returned again."

Thanks once more!

Kind regards,
Stevo Slavic.

On Wed, Nov 11, 2015 at 6:12 PM, Jun Rao <ju...@confluent.io> wrote:

> Are you using compressed messages? If so, when using SimpleConsumer, it's
> possible for you to see messages whose offset is smaller than the offset in
> the fetch request, if those messages are in the same compressed batch. It's
> the responsibility of the client to skip over those messages. Note that the
> high level consumer handles that logic already.
>
> Thanks,
>
> Jun
>
> On Wed, Nov 11, 2015 at 12:40 AM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > Hello Apache Kafka community,
> >
> >
> > I'm using simple consumer with Kafka 0.8.2.2 and noticed that under some
> > conditions fetch response message set for a partition can contain at
> least
> > one (if not all) MessageAndOffset with nextOffset being equal to current
> > (committed) offset, offset used in fetch request. Not sure how it's
> related
> > but I could notice this behavior especially often when I was using new
> > async producer, and when fetch request was able to fetch several messages
> > all the way to the end of the partition.
> >
> > Is this a feature or a bug?
> >
> >
> > Kind regards,
> >
> > Stevo Slavic.
> >
>

Re: Blocked consumer - next offset same as current offset

[Jun Rao <ju...@confluent.io>]

Are you using compressed messages? If so, when using SimpleConsumer, it's
possible for you to see messages whose offset is smaller than the offset in
the fetch request, if those messages are in the same compressed batch. It's
the responsibility of the client to skip over those messages. Note that the
high level consumer handles that logic already.

Thanks,

Jun

On Wed, Nov 11, 2015 at 12:40 AM, Stevo Slavić <ss...@gmail.com> wrote:

> Hello Apache Kafka community,
>
>
> I'm using simple consumer with Kafka 0.8.2.2 and noticed that under some
> conditions fetch response message set for a partition can contain at least
> one (if not all) MessageAndOffset with nextOffset being equal to current
> (committed) offset, offset used in fetch request. Not sure how it's related
> but I could notice this behavior especially often when I was using new
> async producer, and when fetch request was able to fetch several messages
> all the way to the end of the partition.
>
> Is this a feature or a bug?
>
>
> Kind regards,
>
> Stevo Slavic.
>