You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "rjain2778 (via GitHub)" <gi...@apache.org> on 2026/03/21 15:32:42 UTC
[PR] Feat/mm2 fault tolerance [kafka]
rjain2778 opened a new pull request, #21848:
URL: https://github.com/apache/kafka/pull/21848
# Kafka Data Replication Project
## Problem Statement
### Overview
This project implements fault-tolerant data replication between two Kafka clusters using an enhanced MirrorMaker 2 (MM2) connector. The solution addresses two critical failure modes that vanilla MM2 does not handle:
1. **Log Truncation**: When Kafka's retention policy deletes messages before MM2 replicates them, vanilla MM2 silently skips those messages, leaving the disaster recovery (DR) cluster with permanent, invisible gaps in its Write-Ahead Log (WAL).
**Solution**: Added `checkForLogTruncation()` method called at the top of every `poll()` cycle:
**Why fail-fast?** Log truncation represents irrecoverable data loss. The DR cluster is now permanently incomplete. Continuing replication would create a DR cluster that appears healthy but cannot fully reconstruct system state on failover.
2. **Topic Reset**: When an operator deletes and recreates a source topic, vanilla MM2 crashes with `OffsetOutOfRangeException` or silently skips new messages because its stored checkpoint offsets no longer exist.
**Solution**: Two complementary detection mechanisms:
1. **Leader Epoch Regression**: Every `ConsumerRecord` carries the partition's leader epoch. A backward jump triggers
2. **OffsetOutOfRangeException Fallback**: Catches resets while MM2 was paused:
**Why auto-recover?** A topic reset is deliberate. The operator chose to wipe the source. The correct response is to replicate the new topic from the beginning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feat/mm2 fault tolerance [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21848:
URL: https://github.com/apache/kafka/pull/21848#issuecomment-4149375902
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feat/mm2 fault tolerance [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison closed pull request #21848: Feat/mm2 fault tolerance
URL: https://github.com/apache/kafka/pull/21848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feat/mm2 fault tolerance [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #21848:
URL: https://github.com/apache/kafka/pull/21848#issuecomment-4496900535
Closing as it's a low effort AI contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org