[jira] [Commented] (KAFKA-12835) Topic IDs can mismatch on brokers (after interbroker protocol version update)

["ZhenChun Pan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555852#comment-17555852 ] 

ZhenChun Pan commented on KAFKA-12835:
--------------------------------------

[~jolshan] Hello, I find an error similar to this issue mentioned below when I update my kafka from 2.7.0 to 2.8.1.

[2022-06-17 09:34:56,599] ERROR [Broker id=0] Topic Id in memory: PV_7diK7RJaS45KsEH4sng does not match the topic Id for partition __consumer_offsets-14 provided in the request: X_hrEUetShKyQ_RR_pEo9A. (state.change.logger)

I found that the topic id of __consumer_offsets in partition.metadata is PV_7diK7RJaS45KsEH4sng. And in zookeeper the topic id is X_hrEUetShKyQ_RR_pEo9A.

Then I delete all partition.metadata of topic __consumer_offsets, restart kafka, and fix this probelm.

I try to recurrence this probelm the second time, but failed.

I want to ask in which situation will make the topic id of __consumer_offsets diffirent between partition.metadata and zookeeper. Because when topic __consumer_offsets have this problem, the consumer will not work completely. I had to make attention on it.

> Topic IDs can mismatch on brokers (after interbroker protocol version update)
> -----------------------------------------------------------------------------
>
>                 Key: KAFKA-12835
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12835
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 2.8.0
>            Reporter: Ivan Yurchenko
>            Assignee: Justine Olshan
>            Priority: Major
>             Fix For: 2.8.1, 3.0.0
>
>
> We had a Kafka cluster running 2.8 version with interbroker protocol set to 2.7. It had a number of topics and everything was fine.
> Then we decided to update the interbroker protocol to 2.8 by the following procedure:
> 1. Run new brokers with the interbroker protocol set to 2.8.
> 2. Move the data from the old brokers to the new ones (normal partition reassignment API).
> 3. Decommission the old brokers.
> At the stage 2 we had the problem: old brokers started failing on {{LeaderAndIsrRequest}} handling with
> {code:java}
> ERROR [Broker id=<...>] Topic Id in memory: <...> does not match the topic Id for partition <...> provided in the request: <...>. (state.change.logger)
> {code}
> for multiple topics. Topics were not recreated.
> We checked {{partition.metadata}} files and IDs there were indeed different from the values in ZooKeeper. It was fixed by deleting the metadata files (and letting them be recreated).
>  
> The logs, unfortunately, didn't show anything that might point to the cause of the issue (or it happened longer ago than we store the logs).
> We tried to reproduce this also, but no success.
> If the community can point out what to check or beware of in future, it will be great. We'll be happy to provide additional information if needed. Thank you! 
> Sorry for the ticket that might be not very actionable. We hope to at least rise awareness of this issue.
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)