You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Boyuan Zhang <bo...@apache.org> on 2020/05/07 22:51:57 UTC
Is committing offset required for Consumer
Hi team,
I'm building an application which uses Kafka Consumer APIs to read messages
from topics. I plan to manually assign TopicPartitions to my consumer and
seek a certain offset before starting to read. I'll also materialize the
last read offset and reuse it when creating the consumer later.
Within my usage, I' curious whether I need to commit offset automatically
or manually. While going through the doc, it seems like committing offset
is only important to dynamic assignment.
Another question around manual assignment is, is it still true that I need
to call poll() continuously to keep the consumer in the group described as
below?
> It is also possible that the consumer could encounter a "livelock"
> situation where it is continuing to send heartbeats, but no progress is
> being made. To prevent the consumer from holding onto its partitions
> indefinitely in this case, we provide a liveness detection mechanism using
> the max.poll.interval.ms setting. Basically if you don't call poll at
> least as frequently as the configured max interval, then the client will
> proactively leave the group so that another consumer can take over its
> partitions. When this happens, you may see an offset commit failure (as
> indicated by a CommitFailedException
> <https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html> thrown
> from a call to commitSync()
> <https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync-->).
> This is a safety mechanism which guarantees that only active members of the
> group are able to commit offsets. So to stay in the group, you must
> continue to call poll.
What will happen to poll() with manual assignment if the consumer is
removed from the group?
Thanks for your help!
Boyuan
Re: Is committing offset required for Consumer
Posted by Boyuan Zhang <bo...@google.com.INVALID>.
Thanks! That helps a lot.
On Thu, May 7, 2020 at 8:10 PM Chris Toomey <ct...@gmail.com> wrote:
> Right -- offset storage is an optional feature with Kafka, you can always
> choose to not use it and instead manage offsets yourself.
>
>
> On Thu, May 7, 2020 at 8:07 PM Boyuan Zhang <bo...@apache.org> wrote:
>
> > Thanks for the pointer! Does that mean I don't need to commit the offset
> > with managing partitions and offset manually?
> >
> > On Thu, May 7, 2020 at 8:02 PM Chris Toomey <ct...@gmail.com> wrote:
> >
> > > If you choose to manually assign topic partitions, then you won't be
> > using
> > > the group protocol to dynamically manage partition assignments and thus
> > > don't have a need to poll or heartbeat at any interval. See "Manual
> > > Partition Assignment" in
> > >
> > >
> >
> https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > .
> > >
> > > On Thu, May 7, 2020 at 3:52 PM Boyuan Zhang <bo...@apache.org>
> wrote:
> > >
> > > > Hi team,
> > > >
> > > > I'm building an application which uses Kafka Consumer APIs to read
> > > messages
> > > > from topics. I plan to manually assign TopicPartitions to my consumer
> > and
> > > > seek a certain offset before starting to read. I'll also materialize
> > the
> > > > last read offset and reuse it when creating the consumer later.
> > > >
> > > > Within my usage, I' curious whether I need to commit offset
> > automatically
> > > > or manually. While going through the doc, it seems like committing
> > offset
> > > > is only important to dynamic assignment.
> > > >
> > > > Another question around manual assignment is, is it still true that I
> > > need
> > > > to call poll() continuously to keep the consumer in the group
> described
> > > as
> > > > below?
> > > >
> > > > > It is also possible that the consumer could encounter a "livelock"
> > > > > situation where it is continuing to send heartbeats, but no
> progress
> > is
> > > > > being made. To prevent the consumer from holding onto its
> partitions
> > > > > indefinitely in this case, we provide a liveness detection
> mechanism
> > > > using
> > > > > the max.poll.interval.ms setting. Basically if you don't call poll
> > at
> > > > > least as frequently as the configured max interval, then the client
> > > will
> > > > > proactively leave the group so that another consumer can take over
> > its
> > > > > partitions. When this happens, you may see an offset commit failure
> > (as
> > > > > indicated by a CommitFailedException
> > > > > <
> > > >
> > >
> >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html
> > > >
> > > > thrown
> > > > > from a call to commitSync()
> > > > > <
> > > >
> > >
> >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync--
> > > > >).
> > > > > This is a safety mechanism which guarantees that only active
> members
> > of
> > > > the
> > > > > group are able to commit offsets. So to stay in the group, you must
> > > > > continue to call poll.
> > > >
> > > > What will happen to poll() with manual assignment if the consumer is
> > > > removed from the group?
> > > >
> > > > Thanks for your help!
> > > >
> > > > Boyuan
> > > >
> > >
> >
>
Re: Is committing offset required for Consumer
Posted by Chris Toomey <ct...@gmail.com>.
Right -- offset storage is an optional feature with Kafka, you can always
choose to not use it and instead manage offsets yourself.
On Thu, May 7, 2020 at 8:07 PM Boyuan Zhang <bo...@apache.org> wrote:
> Thanks for the pointer! Does that mean I don't need to commit the offset
> with managing partitions and offset manually?
>
> On Thu, May 7, 2020 at 8:02 PM Chris Toomey <ct...@gmail.com> wrote:
>
> > If you choose to manually assign topic partitions, then you won't be
> using
> > the group protocol to dynamically manage partition assignments and thus
> > don't have a need to poll or heartbeat at any interval. See "Manual
> > Partition Assignment" in
> >
> >
> https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > .
> >
> > On Thu, May 7, 2020 at 3:52 PM Boyuan Zhang <bo...@apache.org> wrote:
> >
> > > Hi team,
> > >
> > > I'm building an application which uses Kafka Consumer APIs to read
> > messages
> > > from topics. I plan to manually assign TopicPartitions to my consumer
> and
> > > seek a certain offset before starting to read. I'll also materialize
> the
> > > last read offset and reuse it when creating the consumer later.
> > >
> > > Within my usage, I' curious whether I need to commit offset
> automatically
> > > or manually. While going through the doc, it seems like committing
> offset
> > > is only important to dynamic assignment.
> > >
> > > Another question around manual assignment is, is it still true that I
> > need
> > > to call poll() continuously to keep the consumer in the group described
> > as
> > > below?
> > >
> > > > It is also possible that the consumer could encounter a "livelock"
> > > > situation where it is continuing to send heartbeats, but no progress
> is
> > > > being made. To prevent the consumer from holding onto its partitions
> > > > indefinitely in this case, we provide a liveness detection mechanism
> > > using
> > > > the max.poll.interval.ms setting. Basically if you don't call poll
> at
> > > > least as frequently as the configured max interval, then the client
> > will
> > > > proactively leave the group so that another consumer can take over
> its
> > > > partitions. When this happens, you may see an offset commit failure
> (as
> > > > indicated by a CommitFailedException
> > > > <
> > >
> >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html
> > >
> > > thrown
> > > > from a call to commitSync()
> > > > <
> > >
> >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync--
> > > >).
> > > > This is a safety mechanism which guarantees that only active members
> of
> > > the
> > > > group are able to commit offsets. So to stay in the group, you must
> > > > continue to call poll.
> > >
> > > What will happen to poll() with manual assignment if the consumer is
> > > removed from the group?
> > >
> > > Thanks for your help!
> > >
> > > Boyuan
> > >
> >
>
Re: Is committing offset required for Consumer
Posted by Boyuan Zhang <bo...@apache.org>.
Thanks for the pointer! Does that mean I don't need to commit the offset
with managing partitions and offset manually?
On Thu, May 7, 2020 at 8:02 PM Chris Toomey <ct...@gmail.com> wrote:
> If you choose to manually assign topic partitions, then you won't be using
> the group protocol to dynamically manage partition assignments and thus
> don't have a need to poll or heartbeat at any interval. See "Manual
> Partition Assignment" in
>
> https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> .
>
> On Thu, May 7, 2020 at 3:52 PM Boyuan Zhang <bo...@apache.org> wrote:
>
> > Hi team,
> >
> > I'm building an application which uses Kafka Consumer APIs to read
> messages
> > from topics. I plan to manually assign TopicPartitions to my consumer and
> > seek a certain offset before starting to read. I'll also materialize the
> > last read offset and reuse it when creating the consumer later.
> >
> > Within my usage, I' curious whether I need to commit offset automatically
> > or manually. While going through the doc, it seems like committing offset
> > is only important to dynamic assignment.
> >
> > Another question around manual assignment is, is it still true that I
> need
> > to call poll() continuously to keep the consumer in the group described
> as
> > below?
> >
> > > It is also possible that the consumer could encounter a "livelock"
> > > situation where it is continuing to send heartbeats, but no progress is
> > > being made. To prevent the consumer from holding onto its partitions
> > > indefinitely in this case, we provide a liveness detection mechanism
> > using
> > > the max.poll.interval.ms setting. Basically if you don't call poll at
> > > least as frequently as the configured max interval, then the client
> will
> > > proactively leave the group so that another consumer can take over its
> > > partitions. When this happens, you may see an offset commit failure (as
> > > indicated by a CommitFailedException
> > > <
> >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html
> >
> > thrown
> > > from a call to commitSync()
> > > <
> >
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync--
> > >).
> > > This is a safety mechanism which guarantees that only active members of
> > the
> > > group are able to commit offsets. So to stay in the group, you must
> > > continue to call poll.
> >
> > What will happen to poll() with manual assignment if the consumer is
> > removed from the group?
> >
> > Thanks for your help!
> >
> > Boyuan
> >
>
Re: Is committing offset required for Consumer
Posted by Chris Toomey <ct...@gmail.com>.
If you choose to manually assign topic partitions, then you won't be using
the group protocol to dynamically manage partition assignments and thus
don't have a need to poll or heartbeat at any interval. See "Manual
Partition Assignment" in
https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
.
On Thu, May 7, 2020 at 3:52 PM Boyuan Zhang <bo...@apache.org> wrote:
> Hi team,
>
> I'm building an application which uses Kafka Consumer APIs to read messages
> from topics. I plan to manually assign TopicPartitions to my consumer and
> seek a certain offset before starting to read. I'll also materialize the
> last read offset and reuse it when creating the consumer later.
>
> Within my usage, I' curious whether I need to commit offset automatically
> or manually. While going through the doc, it seems like committing offset
> is only important to dynamic assignment.
>
> Another question around manual assignment is, is it still true that I need
> to call poll() continuously to keep the consumer in the group described as
> below?
>
> > It is also possible that the consumer could encounter a "livelock"
> > situation where it is continuing to send heartbeats, but no progress is
> > being made. To prevent the consumer from holding onto its partitions
> > indefinitely in this case, we provide a liveness detection mechanism
> using
> > the max.poll.interval.ms setting. Basically if you don't call poll at
> > least as frequently as the configured max interval, then the client will
> > proactively leave the group so that another consumer can take over its
> > partitions. When this happens, you may see an offset commit failure (as
> > indicated by a CommitFailedException
> > <
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html>
> thrown
> > from a call to commitSync()
> > <
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync--
> >).
> > This is a safety mechanism which guarantees that only active members of
> the
> > group are able to commit offsets. So to stay in the group, you must
> > continue to call poll.
>
> What will happen to poll() with manual assignment if the consumer is
> removed from the group?
>
> Thanks for your help!
>
> Boyuan
>
Fwd: Is committing offset required for Consumer
Posted by Boyuan Zhang <bo...@apache.org>.
Add dev mailing list as well.
Thanks for your help!
---------- Forwarded message ---------
From: Boyuan Zhang <bo...@apache.org>
Date: Thu, May 7, 2020 at 3:51 PM
Subject: Is committing offset required for Consumer
To: <us...@kafka.apache.org>
Hi team,
I'm building an application which uses Kafka Consumer APIs to read messages
from topics. I plan to manually assign TopicPartitions to my consumer and
seek a certain offset before starting to read. I'll also materialize the
last read offset and reuse it when creating the consumer later.
Within my usage, I' curious whether I need to commit offset automatically
or manually. While going through the doc, it seems like committing offset
is only important to dynamic assignment.
Another question around manual assignment is, is it still true that I need
to call poll() continuously to keep the consumer in the group described as
below?
> It is also possible that the consumer could encounter a "livelock"
> situation where it is continuing to send heartbeats, but no progress is
> being made. To prevent the consumer from holding onto its partitions
> indefinitely in this case, we provide a liveness detection mechanism using
> the max.poll.interval.ms setting. Basically if you don't call poll at
> least as frequently as the configured max interval, then the client will
> proactively leave the group so that another consumer can take over its
> partitions. When this happens, you may see an offset commit failure (as
> indicated by a CommitFailedException
> <https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html> thrown
> from a call to commitSync()
> <https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync-->).
> This is a safety mechanism which guarantees that only active members of the
> group are able to commit offsets. So to stay in the group, you must
> continue to call poll.
What will happen to poll() with manual assignment if the consumer is
removed from the group?
Thanks for your help!
Boyuan