[jira] [Created] (KAFKA-13132) Upgrading to topic IDs in LISR requests has gaps introduced in 3.0

["Justine Olshan (Jira)" <ji...@apache.org>]

Justine Olshan created KAFKA-13132:
--------------------------------------

             Summary: Upgrading to topic IDs in LISR requests has gaps introduced in 3.0
                 Key: KAFKA-13132
                 URL: https://issues.apache.org/jira/browse/KAFKA-13132
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 3.0.0
            Reporter: Justine Olshan
            Assignee: Justine Olshan


With the change in 3.0 to how topic IDs are assigned to logs, a bug was inadvertently introduced. Now, topic IDs will only be assigned on the load of the log to a partition in LISR requests. This means we will only assign topic IDs for newly created topics/partitions, on broker startup, or potentially when a partition is reassigned.

 

In the case of upgrading from an IBP before 2.8, we may have a scenario where we upgrade the controller to IBP 3.0 last. (Ie, the controller is IBP < 2.8 and all other brokers are on IBP 3.0) Upon the last broker upgrading, we will elect a new controller but its LISR request may not result in topic IDs being assigned to logs. 

 Keep in mind, in this scenario, topic IDs will be assigned to all pre-existing topics and will show up in metadata. It is just the LISR/partition.metadata part of the code that is affected. This means we are not ensured the same guarantees we had in 2.8.

 

The problem is two-fold
 1. We ignore LISR requests when the partition leader epoch has not increased (previously we assigned the ID before this check)
 2. We only assign the topic ID when we are associating the log with the partition in replicamanager for the first time. Though in the scenario described above, we have logs associated with partitions that need to be upgraded.

 

We should check the if the LISR request is resulting in a topic ID addition and add logic to logs already associated to partitions in replica manager.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)