Log retention policy - offset ?

[Ranjith Venkatesan <ra...@zohocorp.com>]

Hi,

We are about to use kafka-0.8 in our production. We have our config to delete messages less than a day i.e 24 hrs. We have our simple consumer which will have its offset in memory and will handle if offsetoutofrange exception occurs.


Suppose if old log messages are getting deleted what will happens to offset ? 




Let me give an example,


Suppose we have 10 messages in which 3 messages are older than 24 hrs. My consumer consumed upto message 5 (offset 4). still 5 more msgs yet to consume


After log deletion what will be the offset ? 
1. If offset gonna be 0 for message 4, then our consumer will consume only one message. (4 msgs lost ?)
2. If offset gonna be 3 for message 4, then its fine. But what happens if offset keeps on increasing and reaches its maximum ? (just curious)




Correct me if i am wrong.




Thanks in advance




Ranjith Venkatesan

Re: Log retention policy - offset ?

[Neha Narkhede <ne...@gmail.com>]

Suppose if old log messages are getting deleted what will happens to offset
?

Offset of a message will never change.

Suppose we have 10 messages in which 3 messages are older than 24 hrs. My
consumer consumed upto message 5 (offset 4). still 5 more msgs yet to
consume

In this case, your consumption is unaffected since only offsets 0 through 2
are garbage collected.

Thanks,
Neha



On Thu, Mar 27, 2014 at 3:44 AM, Ranjith Venkatesan
<ra...@zohocorp.com>wrote:

> Hi,
>
> We are about to use kafka-0.8 in our production. We have our config to
> delete messages less than a day i.e 24 hrs. We have our simple consumer
> which will have its offset in memory and will handle if offsetoutofrange
> exception occurs.
>
> Suppose if old log messages are getting deleted what will happens to
> offset ?
>
>
> Let me give an example,
>
> Suppose we have 10 messages in which 3 messages are older than 24 hrs. My
> consumer consumed upto message 5 (offset 4). still 5 more msgs yet to
> consume
>
> After log deletion what will be the offset ?
> 1. If offset gonna be 0 for message 4, then our consumer will consume only
> one message. (4 msgs lost ?)
> 2. If offset gonna be 3 for message 4, then its fine. But what happens if
> offset keeps on increasing and reaches its maximum ? (just curious)
>
>
> Correct me if i am wrong.
>
>
> Thanks in advance
>
>
> Ranjith Venkatesan
>
>