[0.8] Autocommit bug? Or am I doing it wrong? :)

["Hargett, Phil" <ph...@mirror-image.com>]

In one of our applications using Kafka, we are using the high-level consumer to pull messages from our topic.

Because we pull messages from topics in discrete units (e.g., an hour's worth of messages), we want to control explicitly when offsets are committed.

Even though "auto.commit.enable" is set to false, during consumer group rebalancing, offsets are committed anyway, regardless of the setting of this flag.

Is this a bug? Or just a known gap in offset management? I do see plenty of notes on the wiki suggesting future releases may enable applications using the high-level consumer to have more fine-grained control over offset management.

I also fully realize that different applications have different needs, and meeting all of them with a clean API can be challenging.

In the case of this application, the high-level consumer solves the problem of locating the correct in a cluster for a given topic, so there are advantages to using it, even if we are not using it to balance fetch load across multiple consumers. We ideally have only 1 consumer active per consumer group, and can tolerate some duplicate messages. But, the consumer groups make it easy for 1 consumer to recover at the correct starting point, should the prior consumer in the group have failed before doing a commit.

Thoughts?

Re: [0.8] Autocommit bug? Or am I doing it wrong? :)

[Neha Narkhede <ne...@gmail.com>]

That sounds odd. Can you turn on DEBUG for
kafka.consumer.ZookeeperConsumerConnector in your consumer and check if you
see the following log message -
"Committed offset <consumer-offset> for topic <your-topic>"

Which version of Kafka are you using?

Thanks,
Neha


On Fri, May 24, 2013 at 8:06 AM, Hargett, Phil <
phil.hargett@mirror-image.com> wrote:

> In one of our applications using Kafka, we are using the high-level
> consumer to pull messages from our topic.
>
> Because we pull messages from topics in discrete units (e.g., an hour's
> worth of messages), we want to control explicitly when offsets are
> committed.
>
> Even though "auto.commit.enable" is set to false, during consumer group
> rebalancing, offsets are committed anyway, regardless of the setting of
> this flag.
>
> Is this a bug? Or just a known gap in offset management? I do see plenty
> of notes on the wiki suggesting future releases may enable applications
> using the high-level consumer to have more fine-grained control over offset
> management.
>
> I also fully realize that different applications have different needs, and
> meeting all of them with a clean API can be challenging.
>
> In the case of this application, the high-level consumer solves the
> problem of locating the correct in a cluster for a given topic, so there
> are advantages to using it, even if we are not using it to balance fetch
> load across multiple consumers. We ideally have only 1 consumer active per
> consumer group, and can tolerate some duplicate messages. But, the consumer
> groups make it easy for 1 consumer to recover at the correct starting
> point, should the prior consumer in the group have failed before doing a
> commit.
>
> Thoughts?

Re: [0.8] Autocommit bug? Or am I doing it wrong? :)

["Hargett, Phil" <ph...@mirror-image.com>]

Done: https://issues.apache.org/jira/browse/KAFKA-919.

I attached a very simple patch to the bug.  I did not change any comments around it, but I can verify that my use case now works as expected.

:)

On May 24, 2013, at 12:33 PM, "Jun Rao" <ju...@gmail.com>> wrote:

This turns out to be a bug. It seems that we always call commitOffsets() in
ZookeeperConsumerConnector.closeFetchersForQueues() (which is called during
rebalance), whether auto commit is enabled or not. Could you file a jira?

Thanks,

Jun


On Fri, May 24, 2013 at 8:06 AM, Hargett, Phil <
phil.hargett@mirror-image.com<ma...@mirror-image.com>> wrote:

In one of our applications using Kafka, we are using the high-level
consumer to pull messages from our topic.

Because we pull messages from topics in discrete units (e.g., an hour's
worth of messages), we want to control explicitly when offsets are
committed.

Even though "auto.commit.enable" is set to false, during consumer group
rebalancing, offsets are committed anyway, regardless of the setting of
this flag.

Is this a bug? Or just a known gap in offset management? I do see plenty
of notes on the wiki suggesting future releases may enable applications
using the high-level consumer to have more fine-grained control over offset
management.

I also fully realize that different applications have different needs, and
meeting all of them with a clean API can be challenging.

In the case of this application, the high-level consumer solves the
problem of locating the correct in a cluster for a given topic, so there
are advantages to using it, even if we are not using it to balance fetch
load across multiple consumers. We ideally have only 1 consumer active per
consumer group, and can tolerate some duplicate messages. But, the consumer
groups make it easy for 1 consumer to recover at the correct starting
point, should the prior consumer in the group have failed before doing a
commit.

Thoughts?

Re: [0.8] Autocommit bug? Or am I doing it wrong? :)

[Jun Rao <ju...@gmail.com>]

This turns out to be a bug. It seems that we always call commitOffsets() in
ZookeeperConsumerConnector.closeFetchersForQueues() (which is called during
rebalance), whether auto commit is enabled or not. Could you file a jira?

Thanks,

Jun


On Fri, May 24, 2013 at 8:06 AM, Hargett, Phil <
phil.hargett@mirror-image.com> wrote:

> In one of our applications using Kafka, we are using the high-level
> consumer to pull messages from our topic.
>
> Because we pull messages from topics in discrete units (e.g., an hour's
> worth of messages), we want to control explicitly when offsets are
> committed.
>
> Even though "auto.commit.enable" is set to false, during consumer group
> rebalancing, offsets are committed anyway, regardless of the setting of
> this flag.
>
> Is this a bug? Or just a known gap in offset management? I do see plenty
> of notes on the wiki suggesting future releases may enable applications
> using the high-level consumer to have more fine-grained control over offset
> management.
>
> I also fully realize that different applications have different needs, and
> meeting all of them with a clean API can be challenging.
>
> In the case of this application, the high-level consumer solves the
> problem of locating the correct in a cluster for a given topic, so there
> are advantages to using it, even if we are not using it to balance fetch
> load across multiple consumers. We ideally have only 1 consumer active per
> consumer group, and can tolerate some duplicate messages. But, the consumer
> groups make it easy for 1 consumer to recover at the correct starting
> point, should the prior consumer in the group have failed before doing a
> commit.
>
> Thoughts?