kafka message expiry question

[Weide Zhang <we...@gmail.com>]

hi,

as far as i understand, log retention time in kafka will delete message
that's older than the retention time. i'm wonder what it implies for
consumer since i'm using simple consumer to manage offsets in predefined
consumer group.

say i have a list of messages for a partition of topic:

1,2,3,4,5 are the message (offsets) associated with the partition in
current time.

if message 1,2,3 expired earlier and only 4,5 are left, does that mean
consumer can only consume 4,5 and need a way to detect 1,2,3 has expired
and make sure it never reads before the earliest offset for the partition ?

if all 1,2,3,4,5 are expired, it seems the offset will become 0. i assume
in this case consumer need to reset its consumed offsets to 0 for the
consumer group.

Thanks,

Weide

Re: kafka message expiry question

[Manikumar Reddy <ku...@nmsworks.co.in>]

HI,


> as far as i understand, log retention time in kafka will delete message
> that's older than the retention time.


  log retention is applicable to log segment files. In kafka, each topic
can have multiple
  partitions and each partition data stored in multiple log segment files.


> say i have a list of messages for a partition of topic:
>
> 1,2,3,4,5 are the message (offsets) associated with the partition in
> current time.
>
> if message 1,2,3 expired earlier and only 4,5 are left, does that mean
> consumer can only consume 4,5 and need a way to detect 1,2,3 has expired
> and make sure it never reads before the earliest offset for the partition
>

  You can consume only the currently available/retained messages.
SimpleConsumerAPI will
  throw OffsetOutOfRange error code for non-existent offsets.


https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example


> if all 1,2,3,4,5 are expired, it seems the offset will become 0. i assume
> in this case consumer need to reset its consumed offsets to 0 for the
> consumer group.
>

   offset is a sequential id number and will not be reset to 0. Consumer
needs to
   be reset to latest offset available on the partition. Pl check
SImpleConsumer
   example for details.

Regards,
Kumar