New Consumer & committed offsets

[Damian Guy <da...@gmail.com>]

Hi,

I've been trying out the new consumer and have noticed that i get duplicate
messages when i stop the consumer and then restart (different processes,
same consumer group).

I consume all of the messages on the topic and commit the offsets for each
partition and stop the consumer. On the next run i expect to get 0
messages, however i get a batch of records from each partition - in this
case works out 1020 messages. Run it again and i get the same batch of
records.

My logging shows that i've received messages with offsets lower than were
previously committed.

committed:{damian_test_one-2=137669}
committed:{damian_test_one-0=139161}
committed:{damian_test_one-1=137663}

min offsets received: {damian_test_one-0=138824, damian_test_one-1=137321,
damian_test_one-2=137331}

I've debugged the initial fetch requests for offsets and the offsets match
up with what has been committed. Is this expected behaviour? Something to
do with batching of compression of message sets?

TIA,
Damian

Re: New Consumer & committed offsets

[Stevo Slavić <ss...@gmail.com>]

Hello Damian,

Yes, there's a +1 difference. See related discussion
http://mail-archives.apache.org/mod_mbox/kafka-users/201507.mbox/%3CCAOeJiJh2SMzVn23JsoWiNk3sfsw82Jr_-kRLcNRd-oZ7pR1yWg%40mail.gmail.com%3E

Kind regards,
Stevo Slavic.

On Tue, Sep 15, 2015 at 3:56 PM, Damian Guy <da...@gmail.com> wrote:

> I turned off compression and still get duplicates, but only 1 from each
> topic.
> Should the initial fetch offset for a partition be committed offset +1 ?
>
> Thanks,
> Damian
>
> On 15 September 2015 at 14:07, Damian Guy <da...@gmail.com> wrote:
>
> > Hi,
> >
> > I've been trying out the new consumer and have noticed that i get
> > duplicate messages when i stop the consumer and then restart (different
> > processes, same consumer group).
> >
> > I consume all of the messages on the topic and commit the offsets for
> each
> > partition and stop the consumer. On the next run i expect to get 0
> > messages, however i get a batch of records from each partition - in this
> > case works out 1020 messages. Run it again and i get the same batch of
> > records.
> >
> > My logging shows that i've received messages with offsets lower than were
> > previously committed.
> >
> > committed:{damian_test_one-2=137669}
> > committed:{damian_test_one-0=139161}
> > committed:{damian_test_one-1=137663}
> >
> > min offsets received: {damian_test_one-0=138824,
> damian_test_one-1=137321,
> > damian_test_one-2=137331}
> >
> > I've debugged the initial fetch requests for offsets and the offsets
> match
> > up with what has been committed. Is this expected behaviour? Something to
> > do with batching of compression of message sets?
> >
> > TIA,
> > Damian
> >
>

Re: New Consumer & committed offsets

[Damian Guy <da...@gmail.com>]

I turned off compression and still get duplicates, but only 1 from each
topic.
Should the initial fetch offset for a partition be committed offset +1 ?

Thanks,
Damian

On 15 September 2015 at 14:07, Damian Guy <da...@gmail.com> wrote:

> Hi,
>
> I've been trying out the new consumer and have noticed that i get
> duplicate messages when i stop the consumer and then restart (different
> processes, same consumer group).
>
> I consume all of the messages on the topic and commit the offsets for each
> partition and stop the consumer. On the next run i expect to get 0
> messages, however i get a batch of records from each partition - in this
> case works out 1020 messages. Run it again and i get the same batch of
> records.
>
> My logging shows that i've received messages with offsets lower than were
> previously committed.
>
> committed:{damian_test_one-2=137669}
> committed:{damian_test_one-0=139161}
> committed:{damian_test_one-1=137663}
>
> min offsets received: {damian_test_one-0=138824, damian_test_one-1=137321,
> damian_test_one-2=137331}
>
> I've debugged the initial fetch requests for offsets and the offsets match
> up with what has been committed. Is this expected behaviour? Something to
> do with batching of compression of message sets?
>
> TIA,
> Damian
>