Offset management implementation

[Clement Dussieux | AT Internet <Cl...@atinternet.com>]

Hi,


I am using Kafka_2.9.2-0.8.2 and play a bit with offsets in my code.
I would like to know how is implemented the offset system for message posting.

The main question here is: for every message posted, it gets an offset greater that the previous one, meaning that message1 gets offset x and message2 gets offset x+1 but how far can offset go ?
Offset does loop and comeback to zero at a point, right ?
Is it deterministic ? Is it just a huge constant and the offset is incremented modulo(%) this constant ?

The fact is that I store the offset in my code and like to update it regarding to the treatments i do, but I imagine that i cannot just increase it every time i read a message (i read in the middle of topic).
I tried to find out how it was in your source code but it did not really turned to be  success.

Thank you for your possible answer and for kafka !

Clément Dussieux

Re: Offset management implementation

[Gwen Shapira <gs...@cloudera.com>]

We store offsets in INT64, so you can go as high as:
9,223,372,036,854,775,807
messages per topic-partition before looping around :)

Gwen

On Fri, Feb 20, 2015 at 12:21 AM, Clement Dussieux | AT Internet <
Clement.Dussieux@atinternet.com> wrote:

> Hi,
>
>
> I am using Kafka_2.9.2-0.8.2 and play a bit with offsets in my code.
> I would like to know how is implemented the offset system for message
> posting.
>
> The main question here is: for every message posted, it gets an offset
> greater that the previous one, meaning that message1 gets offset x and
> message2 gets offset x+1 but how far can offset go ?
> Offset does loop and comeback to zero at a point, right ?
> Is it deterministic ? Is it just a huge constant and the offset is
> incremented modulo(%) this constant ?
>
> The fact is that I store the offset in my code and like to update it
> regarding to the treatments i do, but I imagine that i cannot just increase
> it every time i read a message (i read in the middle of topic).
> I tried to find out how it was in your source code but it did not really
> turned to be  success.
>
> Thank you for your possible answer and for kafka !
>
> Clément Dussieux
>
>