Re: Kafka 3.5.2 is able to read __consumer_offsets from partitions with metadata version 3 when inter.broker.protocol.version is set to 2.0

[dethmix dethmix <de...@gmail.com>]

Hi all,

I do understand that everybody is excited about the Kafka 3.8 release, but
this question is still actual :)
Any updates.

Thanks,

On Tue, Jul 16, 2024 at 2:54 PM dethmix dethmix <de...@gmail.com> wrote:

> Hi all,
>
> I am working on upgrading the Kafka cluster from the version 2.0.1 to
> 3.5.2. The main concern in that update is passing version 2.1 which has the
> following note in the upgrade document:
> ```
> Note that 2.1.x contains a change to the internal schema used to store
> consumer offsets. Once the upgrade is complete, it will not be possible to
> downgrade to previous versions.
> ```
> In the rolling upgrade section the documentation says the following:
> ```
> Restart the brokers one by one for the new protocol version to take
> effect. Once the brokers begin using the latest protocol version, it will
> no longer be possible to downgrade the cluster to an older version.
> ```
> So, as soon as inter.broker.protocol.version is set to 2.1+ you should not
> be able to rollback to the old version.
> I did reproduce the issue with the rollback with the following steps:
> 1. Start cluster with Kafka 2.0.1 binaries and
> inter.broker.protocol.version=2.0. Produce some messages in the test topic
> 2. Restart cluster with Kafka 3.5.2 binaries
> and inter.broker.protocol.version=3.5. Consume messages.
> 3. Restart cluster once again with Kafka 2.0.1 binaries and
> inter.broker.protocol.version=2.0. During start you will get error like
> this:
> ```
> ERROR [GroupMetadataManager brokerId=0] Error loading offsets from
> __consumer_offsets-43 (kafka.coordinator.group.GroupMetadataManager)
> org.apache.kafka.common.KafkaException: Unknown group metadata version 3
> ```
>
> The interesting thing is that this doesn't happen if I rollback
> inter.broker.protocol.version to 2.0 using Kafka 3.5.2 binaries. Cluster is
> able to start and read all the data from the __consumer_offsets partitions
> with metadata version 3.
> The question is that expected behavior? Below are the steps that I have
> used:
> 1. Start cluster with Kafka 2.0.1 binaries and
> inter.broker.protocol.version=2.0. Produce some messages in the test topic
> 2. Restart cluster with Kafka 3.5.2 binaries
> and inter.broker.protocol.version=3.5. Consume messages.
> 3. Restart cluster once again with Kafka 3.5.2 binaries and
> inter.broker.protocol.version=2.0. Cluster starts properly and is able to
> read partitions from the __consumer_offsets topic.
>
> I did double check Kafka startup logs and make sure that it uses
> configured inter.broker.protocol.version versions.
> if this is okay, then I guess that can be used as a rollback in case of
> problems after switching to the inter.broker.protocol.version 3.X
>
> Thanks in advance,
> Dmytro
>

Re: Kafka 3.5.2 is able to read __consumer_offsets from partitions with metadata version 3 when inter.broker.protocol.version is set to 2.0

[Luke Chen <sh...@gmail.com>]

Hi Dethmix,

> The interesting thing is that this doesn't happen if I rollback
inter.broker.protocol.version to 2.0 using Kafka 3.5.2 binaries. Cluster is
able to start and read all the data from the __consumer_offsets partitions
with metadata version 3.
The question is that expected behavior?

I think you should understand that even if it works well so far, we can't
guarantee there will be no unexpected issue happening later.
I think the document clearly said that downgrading is not supported, please
try not to do that.

Thanks.
Luke

On Wed, Aug 14, 2024 at 3:36 AM dethmix dethmix <de...@gmail.com> wrote:

> Hi all,
>
> I do understand that everybody is excited about the Kafka 3.8 release, but
> this question is still actual :)
> Any updates.
>
> Thanks,
>
> On Tue, Jul 16, 2024 at 2:54 PM dethmix dethmix <de...@gmail.com> wrote:
>
> > Hi all,
> >
> > I am working on upgrading the Kafka cluster from the version 2.0.1 to
> > 3.5.2. The main concern in that update is passing version 2.1 which has
> the
> > following note in the upgrade document:
> > ```
> > Note that 2.1.x contains a change to the internal schema used to store
> > consumer offsets. Once the upgrade is complete, it will not be possible
> to
> > downgrade to previous versions.
> > ```
> > In the rolling upgrade section the documentation says the following:
> > ```
> > Restart the brokers one by one for the new protocol version to take
> > effect. Once the brokers begin using the latest protocol version, it will
> > no longer be possible to downgrade the cluster to an older version.
> > ```
> > So, as soon as inter.broker.protocol.version is set to 2.1+ you should
> not
> > be able to rollback to the old version.
> > I did reproduce the issue with the rollback with the following steps:
> > 1. Start cluster with Kafka 2.0.1 binaries and
> > inter.broker.protocol.version=2.0. Produce some messages in the test
> topic
> > 2. Restart cluster with Kafka 3.5.2 binaries
> > and inter.broker.protocol.version=3.5. Consume messages.
> > 3. Restart cluster once again with Kafka 2.0.1 binaries and
> > inter.broker.protocol.version=2.0. During start you will get error like
> > this:
> > ```
> > ERROR [GroupMetadataManager brokerId=0] Error loading offsets from
> > __consumer_offsets-43 (kafka.coordinator.group.GroupMetadataManager)
> > org.apache.kafka.common.KafkaException: Unknown group metadata version 3
> > ```
> >
> > The interesting thing is that this doesn't happen if I rollback
> > inter.broker.protocol.version to 2.0 using Kafka 3.5.2 binaries. Cluster
> is
> > able to start and read all the data from the __consumer_offsets
> partitions
> > with metadata version 3.
> > The question is that expected behavior? Below are the steps that I have
> > used:
> > 1. Start cluster with Kafka 2.0.1 binaries and
> > inter.broker.protocol.version=2.0. Produce some messages in the test
> topic
> > 2. Restart cluster with Kafka 3.5.2 binaries
> > and inter.broker.protocol.version=3.5. Consume messages.
> > 3. Restart cluster once again with Kafka 3.5.2 binaries and
> > inter.broker.protocol.version=2.0. Cluster starts properly and is able to
> > read partitions from the __consumer_offsets topic.
> >
> > I did double check Kafka startup logs and make sure that it uses
> > configured inter.broker.protocol.version versions.
> > if this is okay, then I guess that can be used as a rollback in case of
> > problems after switching to the inter.broker.protocol.version 3.X
> >
> > Thanks in advance,
> > Dmytro
> >
>