consumer_offest partition increased.

[David Ballano Fernandez <df...@demonware.net>]

Hi guys,

So, we have a Kafka cluster v2.8, and by mistake, we have increased the
partition number from 50 to 52.

And now we are having some coordinator inconsistencies when consumers try
to consume from the cluster.

Any advice on how to untangle this mess? would a rolling restart of the
cluster help? or even deleting the consumer_offest topic?

thanks.

Re: consumer_offest partition increased.

[David Ballano Fernandez <df...@demonware.net>]

Hi Divij, Thanks for your reply! Will get a rolling restart going!

On Tue, Apr 4, 2023 at 8:54 AM Divij Vaidya <di...@gmail.com> wrote:

> Hi David Sorry to hear about your situation. You need a restart of all the
> brokers. This is because to determine the group coordinator for a consumer
> group, a formula is used [1] which is dependent on the number of partitions
> in the consumer
> ZjQcmQRYFpfptBannerStart
> This Message Is From an Untrusted Sender
> You have not previously corresponded with this sender.
>
> ZjQcmQRYFpfptBannerEnd
>
> Hi David
>
> Sorry to hear about your situation. You need a restart of all the brokers.
> This is because to determine the group coordinator for a consumer group, a
> formula is used [1] which is dependent on the number of partitions in the
> consumer group. Unfortunately, once you change the number of partitions for
> __consumer_offsets, the new number of partitions is not
> automatically picked up by this formula since it's only calculated at
> startup of the broker.
> Hence, even if you do a rolling restart, you will face periods of
> uncertainty until all your brokers have finished restarting. You can find
> more information in the background section of KIP 895 [2].
>
> So, my suggestion would be to perform a rolling restart with an expectation
> that your cluster may not be able to consume the topic until all brokers
> have restarted. After the completion of the restart, everything should be
> ok.
>
> Regards,
> Divij
>
> [1] leader_of_partition(abs(hash(CONSUMER_GROUP_NAME)) %
> __CONSUMER_OFFSETS_PARTITION_COUNT)
> [2]https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D895-253A-2BDynamically-2Brefresh-2Bpartition-2Bcount-2Bof-2B-5F-5Fconsumer-5Foffsets&d=DwIFaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=9DpGX5TXJQUt8QvQvJNrVcwZK1-pYTdpgPjYwQkXkYA752EKHdc2kbLnYC6c9xyj&s=5ncO1fPHJBb_TeHewskIs2vXiZMOexEVhduDZ2cUceY&e=
>
>
>
>
> Divij Vaidya
>
>
>
> On Mon, Apr 3, 2023 at 7:42 PM David Ballano Fernandez <df...@demonware.net> wrote:
>
> > Hi guys,
> >
> > So, we have a Kafka cluster v2.8, and by mistake, we have increased the
> > partition number from 50 to 52.
> >
> > And now we are having some coordinator inconsistencies when consumers try
> > to consume from the cluster.
> >
> > Any advice on how to untangle this mess? would a rolling restart of the
> > cluster help? or even deleting the consumer_offest topic?
> >
> > thanks.
> >
>
>

Re: consumer_offest partition increased.

[Divij Vaidya <di...@gmail.com>]

Hi David

Sorry to hear about your situation. You need a restart of all the brokers.
This is because to determine the group coordinator for a consumer group, a
formula is used [1] which is dependent on the number of partitions in the
consumer group. Unfortunately, once you change the number of partitions for
__consumer_offsets, the new number of partitions is not
automatically picked up by this formula since it's only calculated at
startup of the broker.
Hence, even if you do a rolling restart, you will face periods of
uncertainty until all your brokers have finished restarting. You can find
more information in the background section of KIP 895 [2].

So, my suggestion would be to perform a rolling restart with an expectation
that your cluster may not be able to consume the topic until all brokers
have restarted. After the completion of the restart, everything should be
ok.

Regards,
Divij

[1] leader_of_partition(abs(hash(CONSUMER_GROUP_NAME)) %
__CONSUMER_OFFSETS_PARTITION_COUNT)
[2]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-895%3A+Dynamically+refresh+partition+count+of+__consumer_offsets




Divij Vaidya



On Mon, Apr 3, 2023 at 7:42 PM David Ballano Fernandez <
dfernandez@demonware.net> wrote:

> Hi guys,
>
> So, we have a Kafka cluster v2.8, and by mistake, we have increased the
> partition number from 50 to 52.
>
> And now we are having some coordinator inconsistencies when consumers try
> to consume from the cluster.
>
> Any advice on how to untangle this mess? would a rolling restart of the
> cluster help? or even deleting the consumer_offest topic?
>
> thanks.
>