[jira] [Updated] (KAFKA-16205) Reduce number of metadata requests during hybrid mode

["David Arthur (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Arthur updated KAFKA-16205:
---------------------------------
    Component/s: migration

> Reduce number of metadata requests during hybrid mode
> -----------------------------------------------------
>
>                 Key: KAFKA-16205
>                 URL: https://issues.apache.org/jira/browse/KAFKA-16205
>             Project: Kafka
>          Issue Type: Improvement
>          Components: controller, kraft, migration
>    Affects Versions: 3.4.0, 3.5.0, 3.6.0, 3.7.0
>            Reporter: David Arthur
>            Priority: Major
>
> When migrating a cluster with a high number of brokers and partitions, it is possible for the controller channel manager queue to get backed up. This can happen when many small RPCs are generated in response to several small MetadataDeltas being handled MigrationPropagator.
>  
> In the ZK controller, various optimizations have been made over the years to reduce the number of UMR and LISR sent during controlled shutdown or other large metadata events. For the ZK to KRaft migration, we use the MetadataLoader infrastructure to learn about and propagate metadata to ZK brokers.
>  
> We need to improve the batching in MigrationPropagator to avoid performance issues during the migration of large clusters.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)