You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vadim Bobrov <va...@gmail.com> on 2015/07/13 22:04:25 UTC
Offset not committed
I am trying to replace ActiveMQ with Kafka in our environment however I
have encountered a strange problem that basically prevents from using Kafka
in production. The problem is that sometimes the offsets are not committed.
I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer,
auto-commit = off. Every N messages I issue commitOffsets(). Now here is
the problem - if N is below a certain number (180 000 for me) it works and
the offset is moving. If N is 180 000 or more the offset is not updated
after commitOffsets
I am looking at offsets using kafka-run-class.sh
kafka.tools.ConsumerOffsetChecker
Any help?
Re: Fwd: Offset not committed
Posted by Vadim Bobrov <va...@gmail.com>.
I am using ZookeeperConsumerConnector
actually I set up a consumer for __consumer_offsets the way you suggested
and now I cannot reproduce the situation any longer. Offsets are committed
every time.
On Tue, Jul 14, 2015 at 1:49 PM, Joel Koshy <jj...@gmail.com> wrote:
> Actually, how are you committing offsets? Are you using the old
> (zookeeperconsumerconnector) or new KafkaConsumer?
>
> It is true that the current APIs don't return any result, but it would
> help to check if anything is getting into the offsets topic - unless
> you are seeing errors in the logs, the offset commit should succeed
> (if you are indeed explicitly committing offsets).
>
> Thanks,
>
> Joel
>
> On Tue, Jul 14, 2015 at 12:19:01PM -0400, Vadim Bobrov wrote:
> > Thanks, Joel, I will but regardless of my findings the basic problem will
> > still be there: there is no guarantee that the offsets will be committed
> > after commitOffsets. Because commitOffsets does not return its exit
> status,
> > nor does it block as I understand until offsets are committed. In other
> > words, there is no way to know that it has, in fact, commited the offsets
> >
> > or am I missing something? And then another question - why does it seem
> to
> > depend on the number of consumed messages?
> >
> > On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy <jj...@gmail.com>
> wrote:
> >
> > > Can you take a look at the kafka commit rate mbean on your consumer?
> > > Also, can you consume the offsets topic while you are committing
> > > offsets and see if/what offsets are getting committed?
> > > (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32)
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote:
> > > > I am trying to replace ActiveMQ with Kafka in our environment
> however I
> > > > have encountered a strange problem that basically prevents from using
> > > Kafka
> > > > in production. The problem is that sometimes the offsets are not
> > > committed.
> > > >
> > > > I am using Kafka 0.8.2.1, offset storage = kafka, high level
> consumer,
> > > > auto-commit = off. Every N messages I issue commitOffsets(). Now
> here is
> > > > the problem - if N is below a certain number (180 000 for me) it
> works
> > > and
> > > > the offset is moving. If N is 180 000 or more the offset is not
> updated
> > > > after commitOffsets
> > > >
> > > > I am looking at offsets using kafka-run-class.sh
> > > > kafka.tools.ConsumerOffsetChecker
> > > > Any help?
> > >
> > >
>
>
Re: Fwd: Offset not committed
Posted by Vadim Bobrov <va...@gmail.com>.
just caught this error again. I issue commitOffsets - no error but no
committng offsets either. __consumer_offsets watching shows no new messages
either. Then in a few minutes I issue commitOffsets again - all committed.
Unless I am doing something terribly wrong this is very unreliable
On Tue, Jul 14, 2015 at 1:49 PM, Joel Koshy <jj...@gmail.com> wrote:
> Actually, how are you committing offsets? Are you using the old
> (zookeeperconsumerconnector) or new KafkaConsumer?
>
> It is true that the current APIs don't return any result, but it would
> help to check if anything is getting into the offsets topic - unless
> you are seeing errors in the logs, the offset commit should succeed
> (if you are indeed explicitly committing offsets).
>
> Thanks,
>
> Joel
>
> On Tue, Jul 14, 2015 at 12:19:01PM -0400, Vadim Bobrov wrote:
> > Thanks, Joel, I will but regardless of my findings the basic problem will
> > still be there: there is no guarantee that the offsets will be committed
> > after commitOffsets. Because commitOffsets does not return its exit
> status,
> > nor does it block as I understand until offsets are committed. In other
> > words, there is no way to know that it has, in fact, commited the offsets
> >
> > or am I missing something? And then another question - why does it seem
> to
> > depend on the number of consumed messages?
> >
> > On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy <jj...@gmail.com>
> wrote:
> >
> > > Can you take a look at the kafka commit rate mbean on your consumer?
> > > Also, can you consume the offsets topic while you are committing
> > > offsets and see if/what offsets are getting committed?
> > > (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32)
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote:
> > > > I am trying to replace ActiveMQ with Kafka in our environment
> however I
> > > > have encountered a strange problem that basically prevents from using
> > > Kafka
> > > > in production. The problem is that sometimes the offsets are not
> > > committed.
> > > >
> > > > I am using Kafka 0.8.2.1, offset storage = kafka, high level
> consumer,
> > > > auto-commit = off. Every N messages I issue commitOffsets(). Now
> here is
> > > > the problem - if N is below a certain number (180 000 for me) it
> works
> > > and
> > > > the offset is moving. If N is 180 000 or more the offset is not
> updated
> > > > after commitOffsets
> > > >
> > > > I am looking at offsets using kafka-run-class.sh
> > > > kafka.tools.ConsumerOffsetChecker
> > > > Any help?
> > >
> > >
>
>
Re: Fwd: Offset not committed
Posted by Joel Koshy <jj...@gmail.com>.
Actually, how are you committing offsets? Are you using the old
(zookeeperconsumerconnector) or new KafkaConsumer?
It is true that the current APIs don't return any result, but it would
help to check if anything is getting into the offsets topic - unless
you are seeing errors in the logs, the offset commit should succeed
(if you are indeed explicitly committing offsets).
Thanks,
Joel
On Tue, Jul 14, 2015 at 12:19:01PM -0400, Vadim Bobrov wrote:
> Thanks, Joel, I will but regardless of my findings the basic problem will
> still be there: there is no guarantee that the offsets will be committed
> after commitOffsets. Because commitOffsets does not return its exit status,
> nor does it block as I understand until offsets are committed. In other
> words, there is no way to know that it has, in fact, commited the offsets
>
> or am I missing something? And then another question - why does it seem to
> depend on the number of consumed messages?
>
> On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy <jj...@gmail.com> wrote:
>
> > Can you take a look at the kafka commit rate mbean on your consumer?
> > Also, can you consume the offsets topic while you are committing
> > offsets and see if/what offsets are getting committed?
> > (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32)
> >
> > Thanks,
> >
> > Joel
> >
> > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote:
> > > I am trying to replace ActiveMQ with Kafka in our environment however I
> > > have encountered a strange problem that basically prevents from using
> > Kafka
> > > in production. The problem is that sometimes the offsets are not
> > committed.
> > >
> > > I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer,
> > > auto-commit = off. Every N messages I issue commitOffsets(). Now here is
> > > the problem - if N is below a certain number (180 000 for me) it works
> > and
> > > the offset is moving. If N is 180 000 or more the offset is not updated
> > > after commitOffsets
> > >
> > > I am looking at offsets using kafka-run-class.sh
> > > kafka.tools.ConsumerOffsetChecker
> > > Any help?
> >
> >
Re: Fwd: Offset not committed
Posted by Vadim Bobrov <va...@gmail.com>.
Thanks, Joel, I will but regardless of my findings the basic problem will
still be there: there is no guarantee that the offsets will be committed
after commitOffsets. Because commitOffsets does not return its exit status,
nor does it block as I understand until offsets are committed. In other
words, there is no way to know that it has, in fact, commited the offsets
or am I missing something? And then another question - why does it seem to
depend on the number of consumed messages?
On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy <jj...@gmail.com> wrote:
> Can you take a look at the kafka commit rate mbean on your consumer?
> Also, can you consume the offsets topic while you are committing
> offsets and see if/what offsets are getting committed?
> (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32)
>
> Thanks,
>
> Joel
>
> On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote:
> > I am trying to replace ActiveMQ with Kafka in our environment however I
> > have encountered a strange problem that basically prevents from using
> Kafka
> > in production. The problem is that sometimes the offsets are not
> committed.
> >
> > I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer,
> > auto-commit = off. Every N messages I issue commitOffsets(). Now here is
> > the problem - if N is below a certain number (180 000 for me) it works
> and
> > the offset is moving. If N is 180 000 or more the offset is not updated
> > after commitOffsets
> >
> > I am looking at offsets using kafka-run-class.sh
> > kafka.tools.ConsumerOffsetChecker
> > Any help?
>
>
Re: Fwd: Offset not committed
Posted by Joel Koshy <jj...@gmail.com>.
Can you take a look at the kafka commit rate mbean on your consumer?
Also, can you consume the offsets topic while you are committing
offsets and see if/what offsets are getting committed?
(http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32)
Thanks,
Joel
On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote:
> I am trying to replace ActiveMQ with Kafka in our environment however I
> have encountered a strange problem that basically prevents from using Kafka
> in production. The problem is that sometimes the offsets are not committed.
>
> I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer,
> auto-commit = off. Every N messages I issue commitOffsets(). Now here is
> the problem - if N is below a certain number (180 000 for me) it works and
> the offset is moving. If N is 180 000 or more the offset is not updated
> after commitOffsets
>
> I am looking at offsets using kafka-run-class.sh
> kafka.tools.ConsumerOffsetChecker
> Any help?
Fwd: Offset not committed
Posted by Vadim Bobrov <va...@gmail.com>.
I am trying to replace ActiveMQ with Kafka in our environment however I
have encountered a strange problem that basically prevents from using Kafka
in production. The problem is that sometimes the offsets are not committed.
I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer,
auto-commit = off. Every N messages I issue commitOffsets(). Now here is
the problem - if N is below a certain number (180 000 for me) it works and
the offset is moving. If N is 180 000 or more the offset is not updated
after commitOffsets
I am looking at offsets using kafka-run-class.sh
kafka.tools.ConsumerOffsetChecker
Any help?