[jira] [Commented] (KAFKA-12835) Topic IDs can mismatch on brokers (after interbroker protocol version update)

["Justine Olshan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17349352#comment-17349352 ] 

Justine Olshan commented on KAFKA-12835:
----------------------------------------

Hi [~ivanyu]. Thanks for pointing this out.
I was curious about your upgrade process. Is there a reason you moved from old brokers to new brokers rather than doing a rolling restart of the same brokers? (Described in the documentation here: https://kafka.apache.org/documentation/#upgrade_2_7_0)

> Topic IDs can mismatch on brokers (after interbroker protocol version update)
> -----------------------------------------------------------------------------
>
>                 Key: KAFKA-12835
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12835
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 2.8.0
>            Reporter: Ivan Yurchenko
>            Priority: Major
>
> We had a Kafka cluster running 2.8 version with interbroker protocol set to 2.7. It had a number of topics and everything was fine.
> Then we decided to update the interbroker protocol to 2.8 by the following procedure:
> 1. Run new brokers with the interbroker protocol set to 2.8.
> 2. Move the data from the old brokers to the new ones (normal partition reassignment API).
> 3. Decommission the old brokers.
> At the stage 2 we had the problem: old brokers started failing on {{LeaderAndIsrRequest}} handling with
> {code:java}
> ERROR [Broker id=<...>] Topic Id in memory: <...> does not match the topic Id for partition <...> provided in the request: <...>. (state.change.logger)
> {code}
> for multiple topics. Topics were not recreated.
> We checked {{partition.metadata}} files and IDs there were indeed different from the values in ZooKeeper. It was fixed by deleting the metadata files (and letting them be recreated).
>  
> The logs, unfortunately, didn't show anything that might point to the cause of the issue (or it happened longer ago than we store the logs).
> We tried to reproduce this also, but no success.
> If the community can point out what to check or beware of in future, it will be great. We'll be happy to provide additional information if needed. Thank you! 
> Sorry for the ticket that might be not very actionable. We hope to at least rise awareness of this issue.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)