You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Zhen Zhang <zz...@twilio.com.INVALID> on 2020/09/23 07:53:16 UTC
Question Regarding Offset Behavior When Calling Poll()
Hi there,
I am new to Kafka and I would like to get some clarifications for a newbie
question,
Let's say if I have set up my consumer's "enable.auto.commit" to false, and
then poll the records one at a time. So when calling poll(), starting from
offset 0, if any exception is thrown, should I expect to get the record at
offset 0 or offset 1 when I call poll() again? The reason I'm asking for
this is bc in the Kafka Doc, it says that,
"The position of the consumer gives the offset of the next record that will
be given out. It will be one larger than the highest offset the consumer
has seen in that partition. It automatically advances every time the
consumer receives messages in a call to poll(Duration)."
But in my described situation above, an exception is thrown, I'm not sure
if this is counted as a successful poll (meaning that the next poll() will
give the next record) or a failed one (meaning that the next poll() will
give the same record again).
I would really appreciate it for your help.
Thanks,
Zhen Zhang
Software Engineer
[image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
MOBILE (949) 771-6073
EMAIL zzhang@twilio.com
Re: Question Regarding Offset Behavior When Calling Poll()
Posted by Zhen Zhang <zz...@twilio.com.INVALID>.
Thank you.
Regards,
Zhen Zhang
Software Engineer
[image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
MOBILE (949) 771-6073
EMAIL zzhang@twilio.com
On Wed, Oct 7, 2020 at 11:35 PM Matthias J. Sax <mj...@apache.org> wrote:
> I guess it's a question of the client implementation and you cannot
> infer the behavior from the Java client.
>
> I would _assume_ that the offset is _not_ advanced. But the behavior is
> not define by Kafka itself, but it's up the the client implementation
> itself. Thus, only the client's docs or community can clarify.
>
> Sorry that I cannot provide a better answer.
>
> -Matthias
>
>
> On 9/25/20 1:03 AM, Zhen Zhang wrote:
> > Hi,
> >
> > Sorry for the late reply, let me clarify on this.
> >
> > I am developing using Golang so I used a library based on librdkafka, and
> > there's one function, ReadMesage(), which is a wrapper on top of the
> poll()
> > function, except that it will only poll one message(record) at a time and
> > return either one of the following,
> >
> > 1. (msg, nil) -> normal situation with no error;
> > 2. (nil, err) -> err is a Kafka timeout error;
> > 3. (nil, err) -> err is general error;
> > 4. (msg, err) -> err is partition-specific error.
> >
> > When I was browsing the javadocs
> >
> https://kafka.apache.org/26/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
> ,
> > I noticed the concept of `Offsets and Consumer Position`, so I am
> wondering
> > that in the event of the situation 2, 3 and 4, will the position still
> > increment by 1 even if what I get from ReadMessage() is an error?
> >
> > I tried to read the docs but didn't find anything useful I can relate
> to, I
> > tried to ask for help from the contributors of the library I was using
> but
> > didn't get an answer, not sure if this question is too obvious or too
> > noobie lolol. Since that library I used is an implementation of the Kafka
> > protocol, I decided to ask it here and I sincerely hope I can get some
> > insights from a Kafka guru.
> >
> > Thanks,
> > Zhen Zhang
> > Software Engineer
> > [image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
> > MOBILE (949) 771-6073
> > EMAIL zzhang@twilio.com
> >
> >
> > On Wed, Sep 23, 2020 at 9:45 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> >> I guess it depends where the exception comes from? Can you clarify?
> >>
> >> -Matthias
> >>
> >> On 9/23/20 12:53 AM, Zhen Zhang wrote:
> >>> Hi there,
> >>>
> >>> I am new to Kafka and I would like to get some clarifications for a
> >> newbie
> >>> question,
> >>>
> >>> Let's say if I have set up my consumer's "enable.auto.commit" to false,
> >> and
> >>> then poll the records one at a time. So when calling poll(), starting
> >> from
> >>> offset 0, if any exception is thrown, should I expect to get the record
> >> at
> >>> offset 0 or offset 1 when I call poll() again? The reason I'm asking
> for
> >>> this is bc in the Kafka Doc, it says that,
> >>> "The position of the consumer gives the offset of the next record that
> >> will
> >>> be given out. It will be one larger than the highest offset the
> consumer
> >>> has seen in that partition. It automatically advances every time the
> >>> consumer receives messages in a call to poll(Duration)."
> >>>
> >>> But in my described situation above, an exception is thrown, I'm not
> sure
> >>> if this is counted as a successful poll (meaning that the next poll()
> >> will
> >>> give the next record) or a failed one (meaning that the next poll()
> will
> >>> give the same record again).
> >>>
> >>> I would really appreciate it for your help.
> >>>
> >>> Thanks,
> >>> Zhen Zhang
> >>> Software Engineer
> >>> [image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
> >>> MOBILE (949) 771-6073
> >>> EMAIL zzhang@twilio.com
> >>>
> >>
> >
>
>
Re: Question Regarding Offset Behavior When Calling Poll()
Posted by "Matthias J. Sax" <mj...@apache.org>.
I guess it's a question of the client implementation and you cannot
infer the behavior from the Java client.
I would _assume_ that the offset is _not_ advanced. But the behavior is
not define by Kafka itself, but it's up the the client implementation
itself. Thus, only the client's docs or community can clarify.
Sorry that I cannot provide a better answer.
-Matthias
On 9/25/20 1:03 AM, Zhen Zhang wrote:
> Hi,
>
> Sorry for the late reply, let me clarify on this.
>
> I am developing using Golang so I used a library based on librdkafka, and
> there's one function, ReadMesage(), which is a wrapper on top of the poll()
> function, except that it will only poll one message(record) at a time and
> return either one of the following,
>
> 1. (msg, nil) -> normal situation with no error;
> 2. (nil, err) -> err is a Kafka timeout error;
> 3. (nil, err) -> err is general error;
> 4. (msg, err) -> err is partition-specific error.
>
> When I was browsing the javadocs
> https://kafka.apache.org/26/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html,
> I noticed the concept of `Offsets and Consumer Position`, so I am wondering
> that in the event of the situation 2, 3 and 4, will the position still
> increment by 1 even if what I get from ReadMessage() is an error?
>
> I tried to read the docs but didn't find anything useful I can relate to, I
> tried to ask for help from the contributors of the library I was using but
> didn't get an answer, not sure if this question is too obvious or too
> noobie lolol. Since that library I used is an implementation of the Kafka
> protocol, I decided to ask it here and I sincerely hope I can get some
> insights from a Kafka guru.
>
> Thanks,
> Zhen Zhang
> Software Engineer
> [image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
> MOBILE (949) 771-6073
> EMAIL zzhang@twilio.com
>
>
> On Wed, Sep 23, 2020 at 9:45 AM Matthias J. Sax <mj...@apache.org> wrote:
>
>> I guess it depends where the exception comes from? Can you clarify?
>>
>> -Matthias
>>
>> On 9/23/20 12:53 AM, Zhen Zhang wrote:
>>> Hi there,
>>>
>>> I am new to Kafka and I would like to get some clarifications for a
>> newbie
>>> question,
>>>
>>> Let's say if I have set up my consumer's "enable.auto.commit" to false,
>> and
>>> then poll the records one at a time. So when calling poll(), starting
>> from
>>> offset 0, if any exception is thrown, should I expect to get the record
>> at
>>> offset 0 or offset 1 when I call poll() again? The reason I'm asking for
>>> this is bc in the Kafka Doc, it says that,
>>> "The position of the consumer gives the offset of the next record that
>> will
>>> be given out. It will be one larger than the highest offset the consumer
>>> has seen in that partition. It automatically advances every time the
>>> consumer receives messages in a call to poll(Duration)."
>>>
>>> But in my described situation above, an exception is thrown, I'm not sure
>>> if this is counted as a successful poll (meaning that the next poll()
>> will
>>> give the next record) or a failed one (meaning that the next poll() will
>>> give the same record again).
>>>
>>> I would really appreciate it for your help.
>>>
>>> Thanks,
>>> Zhen Zhang
>>> Software Engineer
>>> [image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
>>> MOBILE (949) 771-6073
>>> EMAIL zzhang@twilio.com
>>>
>>
>
Re: Question Regarding Offset Behavior When Calling Poll()
Posted by Zhen Zhang <zz...@twilio.com.INVALID>.
Hi,
Sorry for the late reply, let me clarify on this.
I am developing using Golang so I used a library based on librdkafka, and
there's one function, ReadMesage(), which is a wrapper on top of the poll()
function, except that it will only poll one message(record) at a time and
return either one of the following,
1. (msg, nil) -> normal situation with no error;
2. (nil, err) -> err is a Kafka timeout error;
3. (nil, err) -> err is general error;
4. (msg, err) -> err is partition-specific error.
When I was browsing the javadocs
https://kafka.apache.org/26/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html,
I noticed the concept of `Offsets and Consumer Position`, so I am wondering
that in the event of the situation 2, 3 and 4, will the position still
increment by 1 even if what I get from ReadMessage() is an error?
I tried to read the docs but didn't find anything useful I can relate to, I
tried to ask for help from the contributors of the library I was using but
didn't get an answer, not sure if this question is too obvious or too
noobie lolol. Since that library I used is an implementation of the Kafka
protocol, I decided to ask it here and I sincerely hope I can get some
insights from a Kafka guru.
Thanks,
Zhen Zhang
Software Engineer
[image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
MOBILE (949) 771-6073
EMAIL zzhang@twilio.com
On Wed, Sep 23, 2020 at 9:45 AM Matthias J. Sax <mj...@apache.org> wrote:
> I guess it depends where the exception comes from? Can you clarify?
>
> -Matthias
>
> On 9/23/20 12:53 AM, Zhen Zhang wrote:
> > Hi there,
> >
> > I am new to Kafka and I would like to get some clarifications for a
> newbie
> > question,
> >
> > Let's say if I have set up my consumer's "enable.auto.commit" to false,
> and
> > then poll the records one at a time. So when calling poll(), starting
> from
> > offset 0, if any exception is thrown, should I expect to get the record
> at
> > offset 0 or offset 1 when I call poll() again? The reason I'm asking for
> > this is bc in the Kafka Doc, it says that,
> > "The position of the consumer gives the offset of the next record that
> will
> > be given out. It will be one larger than the highest offset the consumer
> > has seen in that partition. It automatically advances every time the
> > consumer receives messages in a call to poll(Duration)."
> >
> > But in my described situation above, an exception is thrown, I'm not sure
> > if this is counted as a successful poll (meaning that the next poll()
> will
> > give the next record) or a failed one (meaning that the next poll() will
> > give the same record again).
> >
> > I would really appreciate it for your help.
> >
> > Thanks,
> > Zhen Zhang
> > Software Engineer
> > [image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
> > MOBILE (949) 771-6073
> > EMAIL zzhang@twilio.com
> >
>
Re: Question Regarding Offset Behavior When Calling Poll()
Posted by "Matthias J. Sax" <mj...@apache.org>.
I guess it depends where the exception comes from? Can you clarify?
-Matthias
On 9/23/20 12:53 AM, Zhen Zhang wrote:
> Hi there,
>
> I am new to Kafka and I would like to get some clarifications for a newbie
> question,
>
> Let's say if I have set up my consumer's "enable.auto.commit" to false, and
> then poll the records one at a time. So when calling poll(), starting from
> offset 0, if any exception is thrown, should I expect to get the record at
> offset 0 or offset 1 when I call poll() again? The reason I'm asking for
> this is bc in the Kafka Doc, it says that,
> "The position of the consumer gives the offset of the next record that will
> be given out. It will be one larger than the highest offset the consumer
> has seen in that partition. It automatically advances every time the
> consumer receives messages in a call to poll(Duration)."
>
> But in my described situation above, an exception is thrown, I'm not sure
> if this is counted as a successful poll (meaning that the next poll() will
> give the next record) or a failed one (meaning that the next poll() will
> give the same record again).
>
> I would really appreciate it for your help.
>
> Thanks,
> Zhen Zhang
> Software Engineer
> [image: Twilio] <https://www.twilio.com/?utm_source=email_signature>
> MOBILE (949) 771-6073
> EMAIL zzhang@twilio.com
>