You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Krishna Raj <re...@gmail.com> on 2014/03/25 06:43:27 UTC

Question about manual tracking of Offset

Hi experts & Kafka Dev team,

Have a very quick question and need your help in designing a consumer. I am
trying to keep the email short and simple.

Scenario, Lets say:

1) Have a Kafka with 20 messages(from Offset 0 - 19)

2) Pulling the 1st 10 message from 0th to 9th Offset(assuming the desired
size pulls 1st 10 message)

3) Manually commit the Last offset - which is Offset 9 using OffsetCommit
API


Now, when I want to going into the next iteration to pull the next batch of
messages(in this case from 10th to 19th Offsets), I will do a Current
Offset Fetch.

In this case, the offset fetch will give me 9th.

Since I have already processed the message @ 9th Offset, I will end up
processing it again.

Ideally, I want to get 10 as Offset when I want to start the next iteration
of starting the message processing.

I navigated through the latest Kafka documents, but I dont see any method
to find out the Next Offset from Kafka rather than the current manually
committed offset.

*Simply put: I processed "Nth" Offset and committed "N". I want the NEXT
OFFSET FROM the Nth OFFSET. I dont want to assume it as (N+1).*

*Or is it correct if I assume its N+1 ?*


Thanks for your time !
Krishna Raj

Re: Question about manual tracking of Offset

Posted by Krishna Raj <re...@gmail.com>.

Thanks, Jun.  Greatly helps and that was simple solution which I overlooked
:)

Regards,
KR


On Tue, Mar 25, 2014 at 9:03 AM, Jun Rao <ju...@gmail.com> wrote:

> While you are consuming messages, you should use
> MessageAndOffset.nextOffset() when saving offsets.
>
> Thanks,
>
> Jun
>
>
> On Mon, Mar 24, 2014 at 10:43 PM, Krishna Raj <reach.krishnaraj@gmail.com
> >wrote:
>
> > Hi experts & Kafka Dev team,
> >
> > Have a very quick question and need your help in designing a consumer. I
> am
> > trying to keep the email short and simple.
> >
> > Scenario, Lets say:
> >
> > 1) Have a Kafka with 20 messages(from Offset 0 - 19)
> >
> > 2) Pulling the 1st 10 message from 0th to 9th Offset(assuming the desired
> > size pulls 1st 10 message)
> >
> > 3) Manually commit the Last offset - which is Offset 9 using OffsetCommit
> > API
> >
> >
> > Now, when I want to going into the next iteration to pull the next batch
> of
> > messages(in this case from 10th to 19th Offsets), I will do a Current
> > Offset Fetch.
> >
> > In this case, the offset fetch will give me 9th.
> >
> > Since I have already processed the message @ 9th Offset, I will end up
> > processing it again.
> >
> > Ideally, I want to get 10 as Offset when I want to start the next
> iteration
> > of starting the message processing.
> >
> > I navigated through the latest Kafka documents, but I dont see any method
> > to find out the Next Offset from Kafka rather than the current manually
> > committed offset.
> >
> > *Simply put: I processed "Nth" Offset and committed "N". I want the NEXT
> > OFFSET FROM the Nth OFFSET. I dont want to assume it as (N+1).*
> >
> > *Or is it correct if I assume its N+1 ?*
> >
> >
> > Thanks for your time !
> > Krishna Raj
> >
>

Re: Question about manual tracking of Offset

Posted by Jun Rao <ju...@gmail.com>.

While you are consuming messages, you should use
MessageAndOffset.nextOffset() when saving offsets.

Thanks,

Jun


On Mon, Mar 24, 2014 at 10:43 PM, Krishna Raj <re...@gmail.com>wrote:

> Hi experts & Kafka Dev team,
>
> Have a very quick question and need your help in designing a consumer. I am
> trying to keep the email short and simple.
>
> Scenario, Lets say:
>
> 1) Have a Kafka with 20 messages(from Offset 0 - 19)
>
> 2) Pulling the 1st 10 message from 0th to 9th Offset(assuming the desired
> size pulls 1st 10 message)
>
> 3) Manually commit the Last offset - which is Offset 9 using OffsetCommit
> API
>
>
> Now, when I want to going into the next iteration to pull the next batch of
> messages(in this case from 10th to 19th Offsets), I will do a Current
> Offset Fetch.
>
> In this case, the offset fetch will give me 9th.
>
> Since I have already processed the message @ 9th Offset, I will end up
> processing it again.
>
> Ideally, I want to get 10 as Offset when I want to start the next iteration
> of starting the message processing.
>
> I navigated through the latest Kafka documents, but I dont see any method
> to find out the Next Offset from Kafka rather than the current manually
> committed offset.
>
> *Simply put: I processed "Nth" Offset and committed "N". I want the NEXT
> OFFSET FROM the Nth OFFSET. I dont want to assume it as (N+1).*
>
> *Or is it correct if I assume its N+1 ?*
>
>
> Thanks for your time !
> Krishna Raj
>