Cleanup partition offsets that exist for consumer groups but not in broker

[Jeff Widman <je...@netskope.com>]

We hit an error in some custom monitoring code for our Kafka cluster where
the root cause was zookeeper was storing for some partition offsets for
consumer groups, but those partitions didn't actually exist on the brokers.

Apparently in the past, some colleagues needed to reset a stuck cluster
caused by corrupted data. So they wiped out the data log files on disk for
some topics, but didn't wipe the consumer offsets.

In an ideal world this situation should never happen. However, things like
this do happen in the real world.

Couple of questions:
1) This is pretty easy to cleanup through the Zookeeper CLI, but how do you
clean this up if we were instead storing offsets in Kafka?

2) From an operational perspective, I'm sure we're not the only ones to hit
this, so I think there should be a simple command/script to clean this up
that is a) packaged with Kafka, and b) documented. Does this currently
exist?

3) I also think it'd be nice if Kafka automatically checked for this error
case and logged a warning. I wouldn't want automatic cleaning, because if
this situation occurs, something is screwy and I'd want to minimize what's
changing while I tried to debug. Is this a reasonable request?

Cheers,
Jeff

Re: Cleanup partition offsets that exist for consumer groups but not in broker

[Guozhang Wang <wa...@gmail.com>]

Hello Jeff,

Generally speaking ZK's stored offset paths should not be used as the
"source-of-truth" to determine which topic-partitions exist in the Kafka
cluster, but instead the broker topics path should be treated as the
"source-of-truth". More specifically, the common usage pattern would be:

1. check if the topic-partition exists from the ZK, or via the
MetadataRequest / Response from any of the brokers (note the former is
still the source of truth, while the latter is only caching a recent
snapshot of the metadata in ZK).

2. then try to fetch the offsets from ZK (Kafka, if you are on 0.8.2+
versions) with the known topic-partitions.


As for deleting the stored offsets in Kafka, there are indeed some other
use cases for this feature (e.g. you would not want to resume from the
committed offsets anymore but rather start from beginning or the LEO, etc),
and currently we do not yet have an admin request for deleting such
offsets, but one can do this via 1) starting a consumer group with the same
group id, 2) reset position to 0 or LEO, etc, 3) commit offsets so that the
offsets are effectively being reset.

Guozhang

On Thu, Nov 3, 2016 at 5:25 PM, Jeff Widman <je...@netskope.com> wrote:

> We hit an error in some custom monitoring code for our Kafka cluster where
> the root cause was zookeeper was storing for some partition offsets for
> consumer groups, but those partitions didn't actually exist on the brokers.
>
> Apparently in the past, some colleagues needed to reset a stuck cluster
> caused by corrupted data. So they wiped out the data log files on disk for
> some topics, but didn't wipe the consumer offsets.
>
> In an ideal world this situation should never happen. However, things like
> this do happen in the real world.
>
> Couple of questions:
> 1) This is pretty easy to cleanup through the Zookeeper CLI, but how do you
> clean this up if we were instead storing offsets in Kafka?
>
> 2) From an operational perspective, I'm sure we're not the only ones to hit
> this, so I think there should be a simple command/script to clean this up
> that is a) packaged with Kafka, and b) documented. Does this currently
> exist?
>
> 3) I also think it'd be nice if Kafka automatically checked for this error
> case and logged a warning. I wouldn't want automatic cleaning, because if
> this situation occurs, something is screwy and I'd want to minimize what's
> changing while I tried to debug. Is this a reasonable request?
>
> Cheers,
> Jeff
>



-- 
-- Guozhang