回复: how to restore kafka and mm2 after disaster

[Calvin Chen <pi...@hotmail.com>]

even I successfully rebuild src kafka and topic/consumer group/acl in src kafka, when I test, I noticed consumer offset in src kafka is out of sync with dst kafka's, so how do we fail-back to src kafka if disaster is recovered, thanks

________________________________
发件人: Calvin Chen <pi...@hotmail.com>
发送时间: 2021年10月12日 11:49
收件人: users@kafka.apache.org <us...@kafka.apache.org>
主题: how to restore kafka and mm2 after disaster

Hi all

I know kafka mm2 works good, it can mirror topic/message/offset to remote cluster, it is quite good, but I have question on how we restore cluster and mirror after disaster, let's say:

I have src kafka mirror to dst kafka, and mm2 works well, then src kafka/zookeeper get disaster and completely gone, all users switched to dst kafka and continue working there.
then we rebuild src kafka, and recreate user topic/consumer group/acl from database, and restart mm2 from src to dst kafka, however, I noticed there is problem to produce message, it complains topic partition count not as expected:

%5|1634038375.826|PARTCNT|rdkafka#producer-1| [thrd:sasl_ssl://sc-dev-kafka01a.eng.vmware.com:9093/bootstrap]: Topic nimbus-vc-mirror partition count changed from 3 to 0

I double check topic name/partition/replica are all same with original one:

Topic: nimbus-vc-mirror TopicId: 6ETa5HJWRsKtDbuBfqkjcw PartitionCount: 3 ReplicationFactor: 2 Configs: flush.ms=3600000,segment.bytes=536870912,retention.bytes=1073741824
Topic: nimbus-vc-mirror Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: nimbus-vc-mirror Partition: 1 Leader: 2 Replicas: 2,3 Isr: 2,3
Topic: nimbus-vc-mirror Partition: 2 Leader: 3 Replicas: 3,1 Isr: 3,1


so question is how do we restore kafka and its original topic/consumer group/acl after disaster? thanks

-Calvin