You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Dhruvil Shah (Jira)" <ji...@apache.org> on 2021/01/29 20:07:00 UTC
[jira] [Created] (KAFKA-12254) MirrorMaker 2.0 creates destination
topic with default configs
Dhruvil Shah created KAFKA-12254:
------------------------------------
Summary: MirrorMaker 2.0 creates destination topic with default configs
Key: KAFKA-12254
URL: https://issues.apache.org/jira/browse/KAFKA-12254
Project: Kafka
Issue Type: Bug
Reporter: Dhruvil Shah
`MirrorSourceConnector` implements the logic for replicating data, configurations, and other metadata between the source and destination clusters. This includes the tasks below:
# `refreshTopicPartitions` for syncing topics / partitions from source to destination.
# `syncTopicConfigs` for syncing topic configurations from source to destination.
A limitation is that `computeAndCreateTopicPartitions` creates topics with default configurations on the destination cluster. A separate async task `syncTopicConfigs` is responsible for syncing the topic configs. Before that sync happens, topic configurations could be out of sync between the two clusters.
In the worst case, this could lead to data loss eg. when we have a compacted topic being mirrored between clusters which is incorrectly created with the default configuration of `cleanup.policy = delete` on the destination before the configurations are sync'd via `syncTopicConfigs`.
Here is an example of the divergence:
Source Topic:
```
Topic: foobar PartitionCount: 1 ReplicationFactor: 1 Configs: cleanup.policy=compact,segment.bytes=1073741824
```
Destination Topic:
```
Topic: A.foobar PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)