You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Francis Dallaire <fr...@ubisoft.com> on 2013/10/07 21:45:01 UTC
Offset question
Hi,
I have question regarding the offset in kafka (0.8). I've gone through the documentation and did some tests, but I want to make sure I'm on the right track.
* Are the offsets guaranteed to be sequential in a partition ?
o Can it contain holes ?
* How offsets are distributed amongst partitions ?
o Unique inside a partition, not inside a topic ?
o What determines where it starts for each partition ?
If you have a link to some documentation, that could be useful.
Thanks,
Francis
Re: Offset question
Posted by Jay Kreps <ja...@gmail.com>.
Offsets always begin at 0 for each partition and increase sequentially from
there. Offsets aren't unique within a topic. As old data is discarded the
first retained offset will not remain 0. The behavior of what is retained
is controlled by your retention settings.
In trunk there is a feature that allows removing messages from the log (
https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction). This
would leave holes between offsets for topics where you enabled compaction.
-Jay
On Mon, Oct 7, 2013 at 12:45 PM, Francis Dallaire <
francis.dallaire@ubisoft.com> wrote:
> Hi,
>
> I have question regarding the offset in kafka (0.8). I've gone through the
> documentation and did some tests, but I want to make sure I'm on the right
> track.
>
>
> * Are the offsets guaranteed to be sequential in a partition ?
>
> o Can it contain holes ?
>
> * How offsets are distributed amongst partitions ?
>
> o Unique inside a partition, not inside a topic ?
>
> o What determines where it starts for each partition ?
>
> If you have a link to some documentation, that could be useful.
>
> Thanks,
>
> Francis
>