You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Atul Kumar (atkumar3)" <at...@appdynamics.com.INVALID> on 2022/12/16 12:54:51 UTC
Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.
Hi Divij,
Thanks for the reply!
We were using the Kafka-manager tool (with an older Kafka client) which was the root cause of this issue.
Regards,
Atul
________________________________
From: Divij Vaidya <di...@gmail.com>
Sent: 08 December 2022 17:55
To: atul.kumar@appdynamics.com.invalid <at...@appdynamics.com.invalid>
Cc: users@kafka.apache.org <us...@kafka.apache.org>
Subject: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.
Hi Atul
There is a known bug which has similar symptoms that you observed. See:
https://issues.apache.org/jira/browse/KAFKA-14190
The bug manifests when you use a client < 2.8.0 with `--zookeeper` flag. It
could be avoided by either upgrading the client or by using
`--bootstrap-server` instead of `--zookeeper`.
Does the above explanation match your Kafka setup?
--
Divij Vaidya
On Thu, Dec 8, 2022 at 1:16 PM Atul Kumar (atkumar3)
<at...@appdynamics.com.invalid> wrote:
> Hello everyone,
>
> We are using Kafka 2.8.1, and recently while rolling restart of one of our
> clusters, we encountered the following issue on the Kafka cluster
>
> topic Id in memory: <> does not match the topic Id for partition <>
> provided in the request: <>.
>
> All the produce requests to the impacted topic were failing. We were able
> to fix the issue by deleting partition.metadata from each broker for the
> impacted topic and doing a rolling restart of the Kafka cluster. We are
> using the Kafka stream application.
>
> The same issue was reported in Jira:
> https://issues.apache.org/jira/browse/KAFKA-12835 and is marked fixed
> with version 2.8.1 but we can still see this issue.
>
> Any help would be highly appreciated on any fix for this issue.
>
> Thanks,
> Atul Kumar
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Regards,
> Atul
>
Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.
Posted by Divij Vaidya <di...@gmail.com>.
I forgot to mention earlier, another way to detect this is to check the
logs for the following lines:
[2022-08-25 17:44:05,308] ERROR [Broker id=0] Topic Id in memory:
jKTRaM_TSNqoc1233I2aYOQ does not match the topic Id for partition
myTopic-0 provided in the request: nI-J1234wQwGiylMfm8k13w.
(state.change.logger)
--
Divij Vaidya
On Thu, Dec 22, 2022 at 11:32 AM Divij Vaidya <di...@gmail.com>
wrote:
> Oh yes! You can run a script which checks that the topic IDs in Zk are
> consistent with the ones that are available on partition.metadata.
>
> An example script would look something like this:
> https://gist.github.com/divijvaidya/8a940a3d67bc8b215c95da54f5eff961
>
> I will also add this script to the JIRA I linked earlier. Please add a +1
> to the JIRA since it would help measure the impact of this bug and hence,
> will help drive prioritization of a fix in the community.
>
> --
> Divij Vaidya
>
> On Mon, Dec 19, 2022 at 1:23 PM Atul Kumar (atkumar3)
> <at...@appdynamics.com.invalid> wrote:
>
>> Hi Divij,
>>
>> As this issue manifests after the Kafka controller restart, is there a
>> way to detect this corrupted Kafka cluster before restarting the controller?
>>
>>
>>
>>
>>
>> Regards,
>> Atul
>> ________________________________
>> From: Atul Kumar (atkumar3) <at...@appdynamics.com>
>> Sent: 16 December 2022 18:24
>> To: atul.kumar@appdynamics.com.invalid <at...@appdynamics.com.INVALID>;
>> users@kafka.apache.org <us...@kafka.apache.org>
>> Subject: Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id
>> in memory: <> does not match the topic Id for partition <> provided in the
>> request: <>.
>>
>> Hi Divij,
>>
>> Thanks for the reply!
>>
>> We were using the Kafka-manager tool (with an older Kafka client) which
>> was the root cause of this issue.
>>
>>
>>
>>
>> Regards,
>> Atul
>> ________________________________
>> From: Divij Vaidya <di...@gmail.com>
>> Sent: 08 December 2022 17:55
>> To: atul.kumar@appdynamics.com.invalid <atul.kumar@appdynamics.com
>> .invalid>
>> Cc: users@kafka.apache.org <us...@kafka.apache.org>
>> Subject: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in
>> memory: <> does not match the topic Id for partition <> provided in the
>> request: <>.
>>
>> Hi Atul
>>
>> There is a known bug which has similar symptoms that you observed. See:
>> https://issues.apache.org/jira/browse/KAFKA-14190
>> The bug manifests when you use a client < 2.8.0 with `--zookeeper` flag.
>> It
>> could be avoided by either upgrading the client or by using
>> `--bootstrap-server` instead of `--zookeeper`.
>>
>> Does the above explanation match your Kafka setup?
>>
>> --
>> Divij Vaidya
>>
>>
>>
>> On Thu, Dec 8, 2022 at 1:16 PM Atul Kumar (atkumar3)
>> <at...@appdynamics.com.invalid> wrote:
>>
>> > Hello everyone,
>> >
>> > We are using Kafka 2.8.1, and recently while rolling restart of one of
>> our
>> > clusters, we encountered the following issue on the Kafka cluster
>> >
>> > topic Id in memory: <> does not match the topic Id for partition <>
>> > provided in the request: <>.
>> >
>> > All the produce requests to the impacted topic were failing. We were
>> able
>> > to fix the issue by deleting partition.metadata from each broker for the
>> > impacted topic and doing a rolling restart of the Kafka cluster. We are
>> > using the Kafka stream application.
>> >
>> > The same issue was reported in Jira:
>> > https://issues.apache.org/jira/browse/KAFKA-12835 and is marked fixed
>> > with version 2.8.1 but we can still see this issue.
>> >
>> > Any help would be highly appreciated on any fix for this issue.
>> >
>> > Thanks,
>> > Atul Kumar
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Regards,
>> > Atul
>> >
>>
>
Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.
Posted by Divij Vaidya <di...@gmail.com>.
Oh yes! You can run a script which checks that the topic IDs in Zk are
consistent with the ones that are available on partition.metadata.
An example script would look something like this:
https://gist.github.com/divijvaidya/8a940a3d67bc8b215c95da54f5eff961
I will also add this script to the JIRA I linked earlier. Please add a +1
to the JIRA since it would help measure the impact of this bug and hence,
will help drive prioritization of a fix in the community.
--
Divij Vaidya
On Mon, Dec 19, 2022 at 1:23 PM Atul Kumar (atkumar3)
<at...@appdynamics.com.invalid> wrote:
> Hi Divij,
>
> As this issue manifests after the Kafka controller restart, is there a
> way to detect this corrupted Kafka cluster before restarting the controller?
>
>
>
>
>
> Regards,
> Atul
> ________________________________
> From: Atul Kumar (atkumar3) <at...@appdynamics.com>
> Sent: 16 December 2022 18:24
> To: atul.kumar@appdynamics.com.invalid <at...@appdynamics.com.INVALID>;
> users@kafka.apache.org <us...@kafka.apache.org>
> Subject: Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id
> in memory: <> does not match the topic Id for partition <> provided in the
> request: <>.
>
> Hi Divij,
>
> Thanks for the reply!
>
> We were using the Kafka-manager tool (with an older Kafka client) which
> was the root cause of this issue.
>
>
>
>
> Regards,
> Atul
> ________________________________
> From: Divij Vaidya <di...@gmail.com>
> Sent: 08 December 2022 17:55
> To: atul.kumar@appdynamics.com.invalid <atul.kumar@appdynamics.com
> .invalid>
> Cc: users@kafka.apache.org <us...@kafka.apache.org>
> Subject: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in
> memory: <> does not match the topic Id for partition <> provided in the
> request: <>.
>
> Hi Atul
>
> There is a known bug which has similar symptoms that you observed. See:
> https://issues.apache.org/jira/browse/KAFKA-14190
> The bug manifests when you use a client < 2.8.0 with `--zookeeper` flag. It
> could be avoided by either upgrading the client or by using
> `--bootstrap-server` instead of `--zookeeper`.
>
> Does the above explanation match your Kafka setup?
>
> --
> Divij Vaidya
>
>
>
> On Thu, Dec 8, 2022 at 1:16 PM Atul Kumar (atkumar3)
> <at...@appdynamics.com.invalid> wrote:
>
> > Hello everyone,
> >
> > We are using Kafka 2.8.1, and recently while rolling restart of one of
> our
> > clusters, we encountered the following issue on the Kafka cluster
> >
> > topic Id in memory: <> does not match the topic Id for partition <>
> > provided in the request: <>.
> >
> > All the produce requests to the impacted topic were failing. We were able
> > to fix the issue by deleting partition.metadata from each broker for the
> > impacted topic and doing a rolling restart of the Kafka cluster. We are
> > using the Kafka stream application.
> >
> > The same issue was reported in Jira:
> > https://issues.apache.org/jira/browse/KAFKA-12835 and is marked fixed
> > with version 2.8.1 but we can still see this issue.
> >
> > Any help would be highly appreciated on any fix for this issue.
> >
> > Thanks,
> > Atul Kumar
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Regards,
> > Atul
> >
>
Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.
Posted by "Atul Kumar (atkumar3)" <at...@appdynamics.com.INVALID>.
Hi Divij,
As this issue manifests after the Kafka controller restart, is there a way to detect this corrupted Kafka cluster before restarting the controller?
Regards,
Atul
________________________________
From: Atul Kumar (atkumar3) <at...@appdynamics.com>
Sent: 16 December 2022 18:24
To: atul.kumar@appdynamics.com.invalid <at...@appdynamics.com.INVALID>; users@kafka.apache.org <us...@kafka.apache.org>
Subject: Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.
Hi Divij,
Thanks for the reply!
We were using the Kafka-manager tool (with an older Kafka client) which was the root cause of this issue.
Regards,
Atul
________________________________
From: Divij Vaidya <di...@gmail.com>
Sent: 08 December 2022 17:55
To: atul.kumar@appdynamics.com.invalid <at...@appdynamics.com.invalid>
Cc: users@kafka.apache.org <us...@kafka.apache.org>
Subject: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.
Hi Atul
There is a known bug which has similar symptoms that you observed. See:
https://issues.apache.org/jira/browse/KAFKA-14190
The bug manifests when you use a client < 2.8.0 with `--zookeeper` flag. It
could be avoided by either upgrading the client or by using
`--bootstrap-server` instead of `--zookeeper`.
Does the above explanation match your Kafka setup?
--
Divij Vaidya
On Thu, Dec 8, 2022 at 1:16 PM Atul Kumar (atkumar3)
<at...@appdynamics.com.invalid> wrote:
> Hello everyone,
>
> We are using Kafka 2.8.1, and recently while rolling restart of one of our
> clusters, we encountered the following issue on the Kafka cluster
>
> topic Id in memory: <> does not match the topic Id for partition <>
> provided in the request: <>.
>
> All the produce requests to the impacted topic were failing. We were able
> to fix the issue by deleting partition.metadata from each broker for the
> impacted topic and doing a rolling restart of the Kafka cluster. We are
> using the Kafka stream application.
>
> The same issue was reported in Jira:
> https://issues.apache.org/jira/browse/KAFKA-12835 and is marked fixed
> with version 2.8.1 but we can still see this issue.
>
> Any help would be highly appreciated on any fix for this issue.
>
> Thanks,
> Atul Kumar
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Regards,
> Atul
>