[jira] [Updated] (KAFKA-12254) MirrorMaker 2.0 creates destination topic with default configs

["Dhruvil Shah (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dhruvil Shah updated KAFKA-12254:
---------------------------------
    Affects Version/s: 2.8.0

> MirrorMaker 2.0 creates destination topic with default configs
> --------------------------------------------------------------
>
>                 Key: KAFKA-12254
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12254
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 2.8.0
>            Reporter: Dhruvil Shah
>            Assignee: Dhruvil Shah
>            Priority: Blocker
>             Fix For: 3.0.0
>
>
> `MirrorSourceConnector` implements the logic for replicating data, configurations, and other metadata between the source and destination clusters. This includes the tasks below:
>  # `refreshTopicPartitions` for syncing topics / partitions from source to destination.
>  # `syncTopicConfigs` for syncing topic configurations from source to destination.
> A limitation is that `computeAndCreateTopicPartitions` creates topics with default configurations on the destination cluster. A separate async task `syncTopicConfigs` is responsible for syncing the topic configs. Before that sync happens, topic configurations could be out of sync between the two clusters.
> In the worst case, this could lead to data loss eg. when we have a compacted topic being mirrored between clusters which is incorrectly created with the default configuration of `cleanup.policy = delete` on the destination before the configurations are sync'd via `syncTopicConfigs`.
> Here is an example of the divergence:
> Source Topic:
> ```
> Topic: foobar PartitionCount: 1 ReplicationFactor: 1 Configs: cleanup.policy=compact,segment.bytes=1073741824
> ```
> Destination Topic:
> ```
> Topic: A.foobar PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
> ```
> A safer approach is to ensure that the right configurations are set on the destination cluster before data is replicated to it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)