You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Mickael Maison <mi...@gmail.com> on 2026/07/03 15:26:01 UTC
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Hi,
Thanks for the KIP!
This will definitively be a great addition to Kafka.
MM1: In the description of the main components,
ClusterMirrorCoordinator has this responsibility:
"Metadata Refresh Scheduling: The coordinator schedules periodic
metadata refresh operations by invoking a metadata manager every 30
seconds by default. This ensures that configuration updates and group
offset commits in the source cluster are continuously propagated to
the destination cluster. The refresh interval is configurable via
mirror.metadata.refresh.interval.ms."
Is that accurate because MirrorMetadataManager has pretty much the
same responsibility.
MM2: About topic deletions, the KIP states: "the operator would need
to manually remove the topic from the mirror"
Is there a way to find topics that have been deleted in the source
cluster for an operator to do that action?
MM3: About group offsets: It states: "Groups with active members on
the destination are skipped."
Should these be logged too?
MM4: In the security control section, the DeleteClusterMirror RPC
requires Alter on the ClusterMirror resource. Should we use Delete
instead of Alter?
On non-cluster resources, we typically use Delete for deletion. See
https://kafka.apache.org/43/security/authorization-and-acls/#operations-and-resources-on-protocols
MM5: In Destination cluster permissions, I think FindCoordinator
should also need Describe on the Group resources for regular consumer
groups.
MM6: In Exactly-Once Semantics, it states: "The mirror fetcher thread
uses READ_UNCOMMITTED isolation"
Could this be configurable? If so we can do this in a follow-up KIP
MM7: It seems quotas are not mirrored between the clusters?
Is that omission intentional?
MM8: In StopMirrorTopicsRequest I'm not sure I understand what the
Patterns field is?
Is it a pattern for the topics to stop? If so the description seems odd
MM9: In StopMirrorTopicsResponse/StartMirrorTopicsResponse/PauseMirrorTopicsResponse/ResumeMirrorTopicsResponse
do we really need the mirror name?
MM10: In Configuration / Broker:
Can you clarify why we need sasl.mechanism.mirror.admin.protocol?
Can't we retrieve the protocol from mirror.admin.listener.name or
inter.broker.listener.name?
If we really need it, should it be named
mirror.admin.sasl.mechanism.protocol to match the other names?
MM11: In Configuration / Mirror
Should it be mirror.acls.include instead of mirror.acl.include to
match mirror.groups.include and mirror.topics.include?
MM12: Is the description of mirror.support.unclean.leader.election inverted?
MM13: The description of CLUSTER_MIRROR_NOT_EMPTY states: "The cluster
mirror still has active or non-removed topics"
Should it be "The cluster mirror still has active or non-stopped topics"?
MM14: We have the MIRROR_TOPIC_BEING_STOPPED error when stopping topic
already in stopping state? What happens when pausing topics already in
pausing state?
Related, I assume when stopping a stopped topic, you get
TOPIC_NOT_IN_CLUSTER_MIRROR
Thanks,
Mickael
On Mon, Jun 22, 2026 at 7:47 PM Andrew Schofield <sc...@apache.org> wrote:
>
> Hi Fede,
> Thanks for your replies.
>
> AS30: Yes, that's right. Two different end markers for the same transaction, and essentially unbounded duration of the inconsistency since KIP-939. I agree that it's a class of transaction state divergence, like KAFKA-20716. For KIP-1279, the most important thing is that consumers of the mirrored topic must not get confused by the markers. For example, when the Fetch response contains the list of aborted transactions, provided we are sure that it can be built properly without confusion from the extra marker, then we are good. It came to mind because the way this is represented in the protocol for the responses is somewhat fragile I feel, so I'm a bit paranoid that a mis-alignment could break it.
>
> Thanks,
> Andrew
>
> On 2026/06/19 12:16:52 Federico Valeri wrote:
> > Hi Andrew, me, again.
> >
> > Given that READ_COMMITTED consumers are unaffected because the ABORT
> > is already recorded in the transaction index and the dangling COMMIT
> > produces no CompletedTxn (no index and LSO updates), the practical
> > impact is limited. However, __transaction_state records the
> > transaction as COMMITTED while the partition log has it as aborted.
> > While analyzing this, we discovered that this type of divergence is
> > not unique to mirroring. KAFKA-20716 documents the same category of
> > inconsistency after unclean leader elections, where lost transaction
> > markers cause the LSO to get stuck (much bigger problem). We believe
> > that solving the transaction state divergence issue would also close
> > this mirroring scenario. Wdyt?
> >
> > On Thu, Jun 18, 2026 at 10:36 PM Federico Valeri <fe...@gmail.com> wrote:
> > >
> > > Hi Andrew, about AS30, I had a closer look at KIP-939, and I think
> > > this is what you mean:
> > >
> > > The problem is that a WriteTxnMarker request for a PID unknown to the
> > > ProducerStateManager (cleared by MIRROR_PID_RESET) does not fail. The
> > > end transaction marker is still appended, but transaction index and
> > > LSO are unaffected (ProducerStateManager.prepareUpdate,
> > > ProducerAppendInfo.appendEndTxnMarker). This may lead to a situation
> > > where we have two different end txn markers for the same transaction:
> > > ABORT from the stop mirroring operation, followed by a COMMIT from the
> > > local Txn Coordinator retrying its WriteTxnMarker after the partition
> > > becomes writable again.
> > >
> > > READ_COMMITTED consumers still see the earlier ABORT in the
> > > transaction index and skip the aborted data. The visible outcome for
> > > consumers is correct. However, __transaction_state on the original
> > > source cluster now records the transaction as COMMITTED while the
> > > partition log has it as aborted, creating a silent inconsistency that
> > > tools querying transaction state will not detect. The window for this
> > > race is bounded by transaction.max.timeout.ms (default 15 minutes),
> > > but becomes unbounded with KIP-939 because it allows prepared
> > > transactions to sit indefinitely.
> > >
> > > We could move the coordinator state to “complete abort” at the cost of
> > > an RPC. This would prevent any retry, but that would break the 2PC
> > > contract (Kafka must hold the prepared state until the external
> > > coordinator decides). Also, before stop there would be a number of
> > > retries constantly going on. We definitely need a better solution.
> > >
> > > On Thu, Jun 18, 2026 at 11:36 AM Federico Valeri <fe...@gmail.com> wrote:
> > > >
> > > > Hi Andrew, thanks for your careful review.
> > > >
> > > > AS28: We reworked that section and added some diagrams to illustrate
> > > > problem and solutions. Let us know if it is better.
> > > >
> > > > AS30: We updated that table, but we are not sure about the problem you
> > > > mention. Can you clarify and send us the sequence of actions that
> > > > would lead to that problem?
> > > >
> > > > AS32: You are right. Replaced StartMirrorTopicsRequestData.TopicData
> > > > with a Controller.MirrorTopicMetadata record. Also renamed TopicData
> > > > to TopicMetadata in all schemas.
> > > >
> > > > AS33: Good idea. Added per-topic error messages in the RPC responses
> > > > you mentioned.
> > > >
> > > > AS29, AS31, AS34 (both of them), AS35, AS36: These small issues are
> > > > all fixed now.
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Jun 9, 2026 at 9:58 PM Andrew Schofield <sc...@apache.org> wrote:
> > > > >
> > > > > Hi Fede and friends,
> > > > > This KIP is shaping up nicely. The change to the way that state management for the mirrors is done is much more elegant. The removal of the PID mapping is also a nice improvement.
> > > > >
> > > > > Inevitably, I have some more comments.
> > > > >
> > > > > AS28: I understand KAFKA-18723 and how leader epochs work in that case, but the enhancements that you made to the Fetch RPC and how they work escapes me. Please could we have some diagrams to illustrate the various epochs?
> > > > >
> > > > > AS29: I like the addition of share group offset syncing. You need AlterShareGroupOffsets and DescribeShareGroupOffsets in the tables of RPCs and permissions.
> > > > >
> > > > > AS30: It is certainly true that EOS is not supported. Using cluster mirroring on transactional data does risk breaking transactional atomicity for transactions in flight at the time that mirroring was stopped. That's a known limitation, and it's an effect of per-topic async replication. Writing ABORT markers when stop mirroring is triggered is interesting. I just wonder whether it's truly safe.
> > > > >
> > > > > Could you enhance the tables of records in the Exactly-Once Semantics section to include leader epochs? I think that the leader epoch for the ABORT markers will have been bumped as the mirroring stopped and the partitions became writable. However, I worry that someone might switch the mirroring back in the other direction before the transaction which was rudely partially aborted has completed on the original source cluster, and that the effect of the transaction coordinator completing the transaction, potentially writing duplicate transaction markers after the ABORT markers which will have been replicated back again.
> > > > >
> > > > > This is probably only a realistic possibility when KIP-939 is present, but it is accepted and I'm sure it will land soon enough. I would say this KIP should be prepared for the existence of long-running transactions so we don't dig a hole to fall into later.
> > > > >
> > > > > AS31: In the section on the Admin Client, the description for Admin.createClusterMirror(String, Map, CreateClusterMirrorOptions) is duplicated.
> > > > >
> > > > > AS32: The use of StartMirrorTopicsRequestData.TopicData in an external interface looks odd.
> > > > >
> > > > > AS33: I suggest per-topic error messages in the RPC responses such as PauseMirrorTopics so you can pass back more context, such as the state if the request failed due to a state mismatch.
> > > > >
> > > > > AS34: There is an unnamed field in DescribeClusterMirrorsResponse. The same in LastMirrorEpochsValue.
> > > > >
> > > > > AS34: Some of the fields in BumpLeaderEpochsRequest start with lower-case letters. Also, there are quite a few fields in the RPCs in general that should be version "0+" which are "0".
> > > > >
> > > > > AS35: Sometimes the partition states in the RPCs are string, but sometimes int8.
> > > > >
> > > > > AS36: The base level for share group support is probably 4.1. All of the RPCs were present in 4.1, although they were only enabled by default in 4.2.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > On 2026/06/04 11:01:36 Federico Valeri wrote:
> > > > > > Hello, additional updates and replies:
> > > > > >
> > > > > > VK12: Consumer data loss risk is a good point. We updated the "Group
> > > > > > Offsets" paragraph.
> > > > > >
> > > > > > Some of you raised the point that using mirror.name suffixes for pause
> > > > > > and stop operations is a bit inelegant and we agreed. Now we propose a
> > > > > > better approach:
> > > > > >
> > > > > > We replaced the topic config based approach (mirror.name and
> > > > > > .stopped/.paused suffix conventions) with a first class metadata
> > > > > > record for tracking mirror topic state changes. A new
> > > > > > MirrorTopicStateChangeRecord is added to the metadata log with three
> > > > > > fields: TopicId (uuid), MirrorName (string), and DesiredState (int8,
> > > > > > where 0 = start mirroring, 1 = stop, 2 = pause). When a user calls
> > > > > > startMirrorTopics, stopMirrorTopics, or pauseMirrorTopics, the
> > > > > > controller writes this record to the metadata log instead of
> > > > > > manipulating config suffixes. Brokers receive the record via metadata
> > > > > > updates and react accordingly, triggering partition level state
> > > > > > transitions through the existing MirrorPartitionState state machine.
> > > > > > More details in the updated KIP.
> > > > > >
> > > > > > Let us know what you think.
> > > > > >
> > > > > > Thanks
> > > > > > Fede
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 2, 2026 at 8:50 AM Luke Chen <sh...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Viquar,
> > > > > > >
> > > > > > > Thanks for the comment.
> > > > > > >
> > > > > > > Regarding VK12, thanks for raising the issue.
> > > > > > > We didn't think about that.
> > > > > > > I agree that compared with data re-processing, it's worse to have data loss.
> > > > > > > The proposed formula makes sense to me.
> > > > > > > *syncedOffset = max(destinationLogStartOffset, min(sourceCommitted,
> > > > > > > destinationLEO))*
> > > > > > >
> > > > > > > Let us have some discussion internally and then update the KIP and reply to
> > > > > > > the thread.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Luke
> > > > > > >
> > > > > > > On Wed, May 20, 2026 at 2:39 AM Federico Valeri <fe...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini,
> > > > > > > >
> > > > > > > > RS14: Done. Missed that, sorry.
> > > > > > > >
> > > > > > > > Thanks again.
> > > > > > > >
> > > > > > > > On Tue, May 19, 2026 at 7:28 PM Rajini Sivaram <ra...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Federico,
> > > > > > > > >
> > > > > > > > > Thanks for the updates. Just one minor point, apart from that, looks
> > > > > > > > good.
> > > > > > > > >
> > > > > > > > > RS14: KIP still shows "bin/kafka-configs.sh --bootstrap-server :9094
> > > > > > > > > --entity-type mirrors".
> > > > > > > > > Will be good to change that to `--entity-type cluster-mirrors`.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, May 19, 2026 at 5:28 PM vaquar khan <va...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Federico and team,
> > > > > > > > > >
> > > > > > > > > > Thank you for your detailed response on May 11, 2026. I greatly
> > > > > > > > appreciate
> > > > > > > > > > the collaborative effort over the past few months to harden the
> > > > > > > > KIP-1279
> > > > > > > > > > architecture.
> > > > > > > > > >
> > > > > > > > > > * 1. Resolved Architectural & Stability Vulnerabilities*
> > > > > > > > > >
> > > > > > > > > > I am pleased to confirm that the KIP has successfully integrated fixes
> > > > > > > > for
> > > > > > > > > > the critical vulnerabilities I flagged, thereby protecting the
> > > > > > > > cluster's
> > > > > > > > > > state machine, memory bounds, and control plane. Consider the following
> > > > > > > > > > items fully resolved on my end:
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > >
> > > > > > > > > > *VK4 / VK8 (Negative PID Bug & State Machine Failure):* I previously
> > > > > > > > > > identified that the proposed -(sourceProducerId + 2) PID rewriting
> > > > > > > > > > formula would fundamentally break hasProducerId() in
> > > > > > > > > > AbstractRecordBatch.java, causing the broker to bypass
> > > > > > > > > > ProducerStateManager.update() and default the Last Stable Offset
> > > > > > > > (LSO)
> > > > > > > > > > to the High Watermark. I am glad to see the KIP authors abandoned
> > > > > > > > PID
> > > > > > > > > > mapping entirely and adopted the deterministic MIRROR_PID_RESET
> > > > > > > > control
> > > > > > > > > > record barrier. This perfectly protects Kafka's exactly-once
> > > > > > > > semantics.
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > >
> > > > > > > > > > *VK11 (TransactionIndex Rebuild Ambiguity):* Thank you for
> > > > > > > > officially
> > > > > > > > > > confirming that the TransactionIndex is strictly rebuilt locally
> > > > > > > > during
> > > > > > > > > > log appends rather than copied byte-for-byte. This resolves my
> > > > > > > > concern
> > > > > > > > > > regarding PID mismatches inducing consumer read anomalies.
> > > > > > > > > > -
> > > > > > > > > >
> > > > > > > > > > *VK1 (Thundering Herd / OOM Heap Allocation):* Your clarification
> > > > > > > > that
> > > > > > > > > > fetcher threads multiplex partitions meaning the memory footprint is
> > > > > > > > > > strictly bounded by num_fetcher_threads * response_max_bytes rather
> > > > > > > > than
> > > > > > > > > > a concurrent 1MB buffer per partition fully resolves my concern
> > > > > > > > > > regarding
> > > > > > > > > > 50GB broker-wide OOM spikes during mass partition wake-ups.
> > > > > > > > > > -
> > > > > > > > > >
> > > > > > > > > > *VK3 (Control Plane Hotspots):* I had flagged the severe risk of
> > > > > > > > > > metadata saturation on a single broker during a "link flap" event.
> > > > > > > > Your
> > > > > > > > > > confirmation that the __mirror_state topic utilizes a compound hash
> > > > > > > > > > of (mirrorName,
> > > > > > > > > > topicId, partition number) mathematically ensures that the 50,000+
> > > > > > > > state
> > > > > > > > > > transitions will be safely distributed across the cluster,
> > > > > > > > neutralizing
> > > > > > > > > > the
> > > > > > > > > > single-node hotspot risk.
> > > > > > > > > >
> > > > > > > > > > 2. Outstanding Critical Blocker: VK12 (Offset Sync Data Loss)
> > > > > > > > > >
> > > > > > > > > > Regarding VK12, there is a fundamental misunderstanding in your
> > > > > > > > previous
> > > > > > > > > > reply. You stated: *"The scenario described can only occur if offsets
> > > > > > > > are
> > > > > > > > > > force-written to an active group, which the design prevents."*
> > > > > > > > > >
> > > > > > > > > > My concern has absolutely nothing to do with overwriting active
> > > > > > > > groups. My
> > > > > > > > > > concern applies strictly to the normal synchronization of inactive/dead
> > > > > > > > > > groups, and is based directly on the race condition currently
> > > > > > > > documented in
> > > > > > > > > > the official KIP-1279 text:
> > > > > > > > > >
> > > > > > > > > > *"During offset synchronization, the committed offset in the
> > > > > > > > destination
> > > > > > > > > > cluster may temporarily exceed the current log end offset (LEO) of the
> > > > > > > > > > mirror topic... consumers attempting to resume from offset 100 will
> > > > > > > > receive
> > > > > > > > > > an OffsetOutOfRangeException. To handle this gracefully, consumers
> > > > > > > > should
> > > > > > > > > > configure auto.offset.reset=latest..."*
> > > > > > > > > >
> > > > > > > > > > If a failover happens during this documented divergence window, a
> > > > > > > > > > reconnecting consumer will hit the OffsetOutOfRangeException. If the
> > > > > > > > > > downstream consumer follows the KIP's official advice and relies on
> > > > > > > > > > auto.offset.reset=latest, the consumer will jump to the absolute newest
> > > > > > > > > > offset on the partition. *This completely skips any newly produced
> > > > > > > > records
> > > > > > > > > > that arrived between the failover and the consumer reconnecting,
> > > > > > > > resulting
> > > > > > > > > > in silent, unrecoverable data loss for the downstream application.*
> > > > > > > > > > *Proposed Resolution: Double-Clamped Offset Safety Invariant* Instead
> > > > > > > > of
> > > > > > > > > > requiring consumers to use auto.offset.reset=latest and endorsing a
> > > > > > > > known
> > > > > > > > > > data loss vector, I propose that the ClusterMirrorCoordinator enforce a
> > > > > > > > > > pre-persist offset validation invariant during offset synchronization.
> > > > > > > > > > Before any translated offset is committed to the destination cluster,
> > > > > > > > the
> > > > > > > > > > coordinator must apply a double-clamp:
> > > > > > > > > >
> > > > > > > > > > *syncedOffset = max(destinationLSO, min(sourceCommitted,
> > > > > > > > destinationLEO))*
> > > > > > > > > >
> > > > > > > > > > Where destinationLSO is the Log Start Offset (earliest readable
> > > > > > > > position
> > > > > > > > > > post-retention) and destinationLEO is the Log End Offset (latest
> > > > > > > > replicated
> > > > > > > > > > position) of the destination partition.
> > > > > > > > > >
> > > > > > > > > > This guarantees that every persisted offset falls within the physically
> > > > > > > > > > valid range eliminating both the OffsetOutOfRangeException caused by
> > > > > > > > > > exceeding the LEO during replication lag, and the expired-offset rewind
> > > > > > > > > > caused by falling below the LSO due to destination retention policies.
> > > > > > > > > >
> > > > > > > > > > The worst-case trade-off under this invariant is bounded re-processing
> > > > > > > > > > proportional to the replication lag at failover time not total
> > > > > > > > partition
> > > > > > > > > > depth which is perfectly consistent with Kafka's documented
> > > > > > > > at-least-once
> > > > > > > > > > delivery guarantees. Broad enterprise production evidence from
> > > > > > > > large-scale
> > > > > > > > > > cross-cluster failovers confirms that state checks alone (preventing
> > > > > > > > active
> > > > > > > > > > group overwrites) are insufficient; strict offset bounds clamping is
> > > > > > > > > > required to achieve enterprise-grade data integrity.
> > > > > > > > > >
> > > > > > > > > > I look forward to your thoughts on implementing this final offset
> > > > > > > > capping
> > > > > > > > > > logic. Once VK12 is patched, I believe this architecture will be
> > > > > > > > > > exceptionally robust and ready for enterprise deployment.
> > > > > > > > > >
> > > > > > > > > > Best Regards,
> > > > > > > > > >
> > > > > > > > > > Viquar Khan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, 18 May 2026 at 14:23, Rajini Sivaram <ra...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Federico,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the updates! The KIP is looking good. A few more small
> > > > > > > > > > comments.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > RS13: A couple of places still refer to `kafka-mirror.sh` like under
> > > > > > > > > > > `Failover Process`. Can we change them to
> > > > > > > > `*kafka-cluster-mirrors.sh*`?
> > > > > > > > > > >
> > > > > > > > > > > RS14: Should we change `--entity-type mirrors` for kafka-configs to
> > > > > > > > be `
> > > > > > > > > > > --entity-type *cluster-mirrors*` to be consistent? Also,
> > > > > > > > > > > CLUSTER_MIRROR((byte)
> > > > > > > > > > > 64, "mirror"); could be `*cluster-mirror*`?
> > > > > > > > > > >
> > > > > > > > > > > RS15: It may be useful to rename `mirror.topic.num.partitions` and `
> > > > > > > > > > > mirror.topic.replication.factor` since they are very similar to `
> > > > > > > > > > > mirror.topic.properties.exclude`, but the `mirror.topic` prefix
> > > > > > > > refers to
> > > > > > > > > > > different topics (the internal topic for the first two and actual
> > > > > > > > mirror
> > > > > > > > > > > topics for the other one).
> > > > > > > > > > >
> > > > > > > > > > > RS16: ACL Sync: KIP says "Deletes ACLs that exist in destination but
> > > > > > > > not
> > > > > > > > > > in
> > > > > > > > > > > source using DeleteAcls request."
> > > > > > > > > > > What happens if someone creates an ACL on the destination to deny
> > > > > > > > > > > User:Alice access to all topics?
> > > > > > > > > > >
> > > > > > > > > > > 1. If that ACL also existed on the source cluster and then it was
> > > > > > > > > > > removed, will it get removed from the destination?
> > > > > > > > > > > 2. If that ACL never existed on the source cluster, will it get
> > > > > > > > > > removed
> > > > > > > > > > > from the destination?
> > > > > > > > > > >
> > > > > > > > > > > RS17: The table in the Source ACLs section says:
> > > > > > > > "DescribeClusterMirrors
> > > > > > > > > > > MC Read Cluster Log truncation"
> > > > > > > > > > > Should that be `ClusterMirror:Read` instead of `Cluster:Read`?
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > Rajini
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 15, 2026 at 8:41 AM Federico Valeri <
> > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Rajini, we finally addressed the API and tool naming
> > > > > > > > refactoring as
> > > > > > > > > > > > you suggested in RS6. Please take a look when you have time.
> > > > > > > > Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 11, 2026 at 6:17 PM Federico Valeri <
> > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hello all, I want to highlight a couple of new paragraphs:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Leader Epoch Invariant: Cluster mirroring enforces the
> > > > > > > > invariant
> > > > > > > > > > > > > that the destination leader epoch must always be greater than or
> > > > > > > > > > equal
> > > > > > > > > > > > > to the source leader epoch (DLE>=SLE). Without this, consumers
> > > > > > > > on the
> > > > > > > > > > > > > destination cluster can get stuck in an infinite metadata refresh
> > > > > > > > > > loop
> > > > > > > > > > > > > when they encounter committed offsets carrying source epochs
> > > > > > > > higher
> > > > > > > > > > > > > than the local epoch. The invariant is maintained through three
> > > > > > > > > > > > > mechanisms: reactive bumping (epoch fencing triggered when SLE >
> > > > > > > > DLE
> > > > > > > > > > > > > during fetch), proactive bumping (scheduled when SLE approaches
> > > > > > > > DLE
> > > > > > > > > > > > > within a threshold), and periodic bumping (checked during
> > > > > > > > coordinator
> > > > > > > > > > > > > metadata sync).
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973#KIP1279:ClusterMirroring-LeaderEpochInvariant
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. Group Offsets: The coordinator periodically syncs consumer and
> > > > > > > > > > > > > share group offsets from the source cluster to the destination
> > > > > > > > for
> > > > > > > > > > all
> > > > > > > > > > > > > mirrored topics. Groups are filtered by configurable
> > > > > > > > include/exclude
> > > > > > > > > > > > > patterns, and offsets are only synced for groups that are not
> > > > > > > > > > > > > currently active on the destination cluster, preventing
> > > > > > > > overwrites of
> > > > > > > > > > > > > local consumer progress. Because source and destination share the
> > > > > > > > > > same
> > > > > > > > > > > > > topic offsets (no offset translation), synced offsets can be used
> > > > > > > > > > > > > directly without mapping.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973#KIP1279:ClusterMirroring-GroupOffsets
> > > > > > > > > > > > >
> > > > > > > > > > > > > These new paragraphs directly address some of your questions,
> > > > > > > > but let
> > > > > > > > > > > > > me list them here:
> > > > > > > > > > > > >
> > > > > > > > > > > > > JR2: Yes, we removed the incorrect phrase and added more details
> > > > > > > > to
> > > > > > > > > > > > > the paragraph.
> > > > > > > > > > > > >
> > > > > > > > > > > > > JR4: When source cluster topic has tiered storage enabled, CM
> > > > > > > > works
> > > > > > > > > > by
> > > > > > > > > > > > > mirroring remote and local log into destination cluster. When
> > > > > > > > > > > > > destination cluster topic has tiered storage enabled, CM fails in
> > > > > > > > > > > > > PREPARING state because the LME may be in remote storage, but
> > > > > > > > works
> > > > > > > > > > > > > fine if already MIRRORING because no truncation is needed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > JR11: See "Leader Epoch Invariant" paragraph mentioned above.
> > > > > > > > > > > > >
> > > > > > > > > > > > > JR13: We can't support stateful Streams application because
> > > > > > > > > > > > > asynchronous replication cannot preserve the transactional
> > > > > > > > boundaries
> > > > > > > > > > > > > between input offset commits, state store mutations written to
> > > > > > > > > > > > > changelog topics, and intermediate records written to repartition
> > > > > > > > > > > > > topics. The synchronous extension of this design will be able to
> > > > > > > > > > > > > support them. Existing Features Integration paragraph updated.
> > > > > > > > > > > > >
> > > > > > > > > > > > > JR18: See "Group Offsets" paragraph mentioned above.
> > > > > > > > > > > > >
> > > > > > > > > > > > > IY1: See "Group Offsets" paragraph mentioned above.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > Fede
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, May 11, 2026 at 6:08 PM Federico Valeri <
> > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Vaquar,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > VK4/VK8: We don't do PID mapping anymore. The KIP was updated
> > > > > > > > some
> > > > > > > > > > > > > > time ago with the new approach based on the new PID reset
> > > > > > > > control
> > > > > > > > > > > > > > record.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > VK11: The transaction index is always built locally during log
> > > > > > > > > > > append,
> > > > > > > > > > > > > > never copied.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > VK1: The 50,000 * 1MB = 50GB calculation misunderstands the
> > > > > > > > fetch
> > > > > > > > > > > > > > model. Fetcher threads don't allocate one buffer per partition.
> > > > > > > > > > > Actual
> > > > > > > > > > > > > > peak memory is roughly num_fetcher_threads *
> > > > > > > > response_max_bytes,
> > > > > > > > > > not
> > > > > > > > > > > > > > num_partitions * partition_max_bytes. With 1 fetcher thread
> > > > > > > > and the
> > > > > > > > > > > > > > default response max, the memory footprint is modest
> > > > > > > > regardless of
> > > > > > > > > > > > > > partition count. We are leveraging the same proven pattern
> > > > > > > > used by
> > > > > > > > > > > the
> > > > > > > > > > > > > > internal replication.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > VK3: The __mirror_state topic uses hash-based partitioning
> > > > > > > > based on
> > > > > > > > > > > > > > mirrorName, topicId and partition number. With the production
> > > > > > > > > > default
> > > > > > > > > > > > > > of 50 partitions, 50,000 partition transitions distribute
> > > > > > > > across
> > > > > > > > > > ~50
> > > > > > > > > > > > > > partition leaders on different brokers, not a single broker.
> > > > > > > > This
> > > > > > > > > > is
> > > > > > > > > > > > > > the same proven pattern as __consumer_offsets, which handles
> > > > > > > > > > millions
> > > > > > > > > > > > > > of commits.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > VK12: The scenario described can only occur if offsets are
> > > > > > > > > > > > > > force-written to an active group, which the design prevents.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > > Fede
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> >
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Rajini, thanks for the info. KIP updated.
Regards
Fede
On Tue, Nov 3, 2026 at 2:11 AM Rajini Sivaram <ra...@gmail.com> wrote:
>
> Hi Federico,
>
> Truncation-on-fetch was introduced in 2.7, but had edge cases, which were
> fixed later. Commits like fcc7c2de391 and 8e211eb72f9 didn't go into 2.7.
> There were also other fixes like KAFKA-12996 that came later than 2.8.0,
> though that one is in 2.8.1. There may be others too. 3.0.x seems like a
> safe option where we can expect truncation-on-fetch to be reliable.
>
> Regards,
>
> Rajini
>
>
> On Tue, Nov 3, 2026 at 5:40 AM Federico Valeri <fe...@gmail.com> wrote:
>
> > Hi Rajini and Michael. Maintenance is a fair point and we agree on
> > that. Just one question for my understanding, why 3.0 and not 2.7 that
> > is where truncation-on-fetch was introduced? Do you see an any
> > specific problem related to the Fetch protocol?
> >
> >
> > On Mon, Nov 2, 2026 at 11:46 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > I agree with Rajini, I'm not sure it's worth re-adding that logic.
> > > If the migration/upgrade use case is important for you, targeting 3.0+
> > > should already cover a large portion of outdated clusters.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > > On Mon, Nov 2, 2026 at 7:41 PM Rajini Sivaram <ra...@gmail.com>
> > wrote:
> > > >
> > > > Hi Federico,
> > > >
> > > > Thanks for the response. I think we removed the old code in
> > > > AbstractFetcherThread to reduce complexity and maintenance cost. It
> > seems
> > > > odd that we would bring back all that complexity in a subclass. Not
> > sure if
> > > > there are users who are still on old broker versions like 2.1 (unlike
> > > > clients). I would have thought that it is sufficient to support 3.0+.
> > It
> > > > will be good to understand how much compatibility we expect to
> > maintain for
> > > > 5.0 and beyond. Let's see what others think too.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Mon, Nov 2, 2026 at 5:58 PM Federico Valeri <fe...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Rajini, thanks for letting us know. The schema is still there and
> > we
> > > > > think it is important to keep support for 2.1 to help old Kafka
> > versions
> > > > > users to migrate to latest version, so we would like to reintroduce
> > that
> > > > > logic just for Cluster Mirroring. Wdyt?
> > > > >
> > > > > On Mon, Nov 2, 2026 at 2:39 AM Rajini Sivaram <
> > rajinisivaram@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Federico,
> > > > > >
> > > > > > KIP-1279 currently states that cluster mirroring source clusters
> > may be
> > > > > > version >= 2.1. We have been removing support for older metadata
> > versions
> > > > > > and
> > > > > >
> > > > > >
> > > > >
> > https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8
> > > > > > removes the explicit truncating phase that sends
> > > > > > OffsetForLeaderEpochRequest from AbstractFetcherThread. So
> > inter-broker
> > > > > > replication now relies entirely on truncate-on-fetch. Are we
> > planning to
> > > > > > introduce OffsetForLeaderEpochRequest in the cluster mirroring
> > fetcher
> > > > > > because we think there is a requirement to migrate from older
> > clusters?
> > > > > Or
> > > > > > are we planning to keep the code simpler and support newer
> > clusters only?
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram <
> > rajinisivaram@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Federico! The solutions for both RS25 and RS27 sound good.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <
> > fedevaleri@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Rajini,
> > > > > > >>
> > > > > > >> >Not sure about RS25, but you mentioned you are looking into it
> > again,
> > > > > > so
> > > > > > >> will wait for the response after that.
> > > > > > >>
> > > > > > >> The clamping first attempts to read LSO and LEO from the local
> > > > > > >> ReplicaManager, which is an in-memory lookup against the local
> > > > > > >> UnifiedLog instance. This works for any partition the
> > coordinator
> > > > > > >> broker happens to host a replica for. For partitions without a
> > local
> > > > > > >> replica, the coordinator falls back to the Admin listOffsets
> > API,
> > > > > > >> which internally routes the request to the partition leader and
> > > > > > >> returns the offsets. We resolve all mirror partition offsets
> > once at
> > > > > > >> the beginning of each sync cycle, then reuse them for every
> > group. Not
> > > > > > >> one call per partition or per group.
> > > > > > >>
> > > > > > >> KIP Updated.
> > > > > > >>
> > > > > > >> > For RS27, not supporting unclean leader election for older
> > clients
> > > > > may
> > > > > > >> be ok unless the fix is straightforward.
> > > > > > >>
> > > > > > >> We think it is possible to support old versions with the
> > following
> > > > > > >> strategy:
> > > > > > >>
> > > > > > >> We detect the source cluster's fetch version dynamically. On
> > the first
> > > > > > >> fetch request, the fetcher thread always includes
> > lastFetchedEpoch in
> > > > > > >> the request builder. The NetworkClient then queries ApiVersions
> > from
> > > > > > >> the source cluster and selects the latestUsableVersion. Two
> > outcomes
> > > > > > >> are possible:
> > > > > > >>
> > > > > > >> 1. Source supports fetch v12+: the request succeeds with
> > > > > > >> lastFetchedEpoch, and truncation proceeds via the
> > truncation-on-fetch
> > > > > > >> path.
> > > > > > >> 2. Source does not support fetch v12: NetworkClient throws
> > > > > > >> UnsupportedVersionException. We catch it, disable
> > truncation-on-fetch,
> > > > > > >> and fall back to the OffsetsForLeaderEpochRequest path.
> > > > > > >>
> > > > > > >> This handles both old and new source clusters, including live
> > upgrades
> > > > > > >> and downgrades.
> > > > > > >>
> > > > > > >> Unclean leader election handling:
> > > > > > >>
> > > > > > >> Previously we only handled unclean leader election (ULE) in the
> > > > > > >> truncation-on-fetch case. To support older clusters via
> > > > > > >> OffsetsForLeaderEpochRequest, we apply the same principle: log
> > > > > > >> truncation should never happen on the destination cluster during
> > > > > > >> mirroring because we only fetch committed data. If an
> > > > > > >> OffsetsForLeaderEpochResponse does trigger truncation, that
> > truncation
> > > > > > >> must originate from a ULE on the source cluster. In that case,
> > we
> > > > > > >> transition to the ULE_RECOVERY state and wait for all replicas
> > to
> > > > > > >> catch up (when mirror.unclean.leader.election.enable=true).
> > > > > > >>
> > > > > > >> To guarantee that all destination truncations come from source
> > ULEs,
> > > > > > >> we store the "last mirror offset and last mirror epoch" in the
> > LME
> > > > > > >> record. Originally, the LME phase one truncated records with
> > epoch
> > > > > > >> greater than the LME epoch, then phase two relied on normal
> > > > > > >> replication to handle log divergence. The problem: some
> > truncations
> > > > > > >> are side effects of mirroring setup (e.g. reverse mirroring),
> > not
> > > > > > >> ULEs.
> > > > > > >>
> > > > > > >> Example (cluster A mirrors from cluster B):
> > > > > > >>
> > > > > > >> Cluster A (dest) <---- Cluster B (source)
> > > > > > >> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> > > > > > >> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> > > > > > >> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
> > > > > > >>
> > > > > > >> When cluster A receives the first OffsetsForLeaderEpochResponse
> > or
> > > > > > >> fetch response from cluster B, offset 2 on cluster A gets
> > truncated.
> > > > > > >> This truncation is not caused by a ULE. Without the last mirror
> > offset
> > > > > > >> and epoch in the LME record, we cannot distinguish it from a
> > real ULE.
> > > > > > >> By storing these values in the LME record, the LOG_ALIGNMENT
> > state
> > > > > > >> truncates only records where: record offset <= last mirror
> > offset, AND
> > > > > > >> record epoch <= last mirror epoch. This correctly truncates
> > offset 2
> > > > > > >> in the example above during LOG_ALIGNMENT. After alignment
> > completes,
> > > > > > >> any subsequent truncation on the destination is guaranteed to
> > > > > > >> originate from a source cluster ULE.
> > > > > > >>
> > > > > > >> Let us know if you are good with this solution. KIP update will
> > > > > follow.
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >> Fede
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> >
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Federico,
Truncation-on-fetch was introduced in 2.7, but had edge cases, which were
fixed later. Commits like fcc7c2de391 and 8e211eb72f9 didn't go into 2.7.
There were also other fixes like KAFKA-12996 that came later than 2.8.0,
though that one is in 2.8.1. There may be others too. 3.0.x seems like a
safe option where we can expect truncation-on-fetch to be reliable.
Regards,
Rajini
On Tue, Nov 3, 2026 at 5:40 AM Federico Valeri <fe...@gmail.com> wrote:
> Hi Rajini and Michael. Maintenance is a fair point and we agree on
> that. Just one question for my understanding, why 3.0 and not 2.7 that
> is where truncation-on-fetch was introduced? Do you see an any
> specific problem related to the Fetch protocol?
>
>
> On Mon, Nov 2, 2026 at 11:46 AM Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I agree with Rajini, I'm not sure it's worth re-adding that logic.
> > If the migration/upgrade use case is important for you, targeting 3.0+
> > should already cover a large portion of outdated clusters.
> >
> > Thanks,
> > Mickael
> >
> >
> > On Mon, Nov 2, 2026 at 7:41 PM Rajini Sivaram <ra...@gmail.com>
> wrote:
> > >
> > > Hi Federico,
> > >
> > > Thanks for the response. I think we removed the old code in
> > > AbstractFetcherThread to reduce complexity and maintenance cost. It
> seems
> > > odd that we would bring back all that complexity in a subclass. Not
> sure if
> > > there are users who are still on old broker versions like 2.1 (unlike
> > > clients). I would have thought that it is sufficient to support 3.0+.
> It
> > > will be good to understand how much compatibility we expect to
> maintain for
> > > 5.0 and beyond. Let's see what others think too.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Mon, Nov 2, 2026 at 5:58 PM Federico Valeri <fe...@gmail.com>
> wrote:
> > >
> > > > Hi Rajini, thanks for letting us know. The schema is still there and
> we
> > > > think it is important to keep support for 2.1 to help old Kafka
> versions
> > > > users to migrate to latest version, so we would like to reintroduce
> that
> > > > logic just for Cluster Mirroring. Wdyt?
> > > >
> > > > On Mon, Nov 2, 2026 at 2:39 AM Rajini Sivaram <
> rajinisivaram@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Federico,
> > > > >
> > > > > KIP-1279 currently states that cluster mirroring source clusters
> may be
> > > > > version >= 2.1. We have been removing support for older metadata
> versions
> > > > > and
> > > > >
> > > > >
> > > >
> https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8
> > > > > removes the explicit truncating phase that sends
> > > > > OffsetForLeaderEpochRequest from AbstractFetcherThread. So
> inter-broker
> > > > > replication now relies entirely on truncate-on-fetch. Are we
> planning to
> > > > > introduce OffsetForLeaderEpochRequest in the cluster mirroring
> fetcher
> > > > > because we think there is a requirement to migrate from older
> clusters?
> > > > Or
> > > > > are we planning to keep the code simpler and support newer
> clusters only?
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram <
> rajinisivaram@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Federico! The solutions for both RS25 and RS27 sound good.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <
> fedevaleri@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Rajini,
> > > > > >>
> > > > > >> >Not sure about RS25, but you mentioned you are looking into it
> again,
> > > > > so
> > > > > >> will wait for the response after that.
> > > > > >>
> > > > > >> The clamping first attempts to read LSO and LEO from the local
> > > > > >> ReplicaManager, which is an in-memory lookup against the local
> > > > > >> UnifiedLog instance. This works for any partition the
> coordinator
> > > > > >> broker happens to host a replica for. For partitions without a
> local
> > > > > >> replica, the coordinator falls back to the Admin listOffsets
> API,
> > > > > >> which internally routes the request to the partition leader and
> > > > > >> returns the offsets. We resolve all mirror partition offsets
> once at
> > > > > >> the beginning of each sync cycle, then reuse them for every
> group. Not
> > > > > >> one call per partition or per group.
> > > > > >>
> > > > > >> KIP Updated.
> > > > > >>
> > > > > >> > For RS27, not supporting unclean leader election for older
> clients
> > > > may
> > > > > >> be ok unless the fix is straightforward.
> > > > > >>
> > > > > >> We think it is possible to support old versions with the
> following
> > > > > >> strategy:
> > > > > >>
> > > > > >> We detect the source cluster's fetch version dynamically. On
> the first
> > > > > >> fetch request, the fetcher thread always includes
> lastFetchedEpoch in
> > > > > >> the request builder. The NetworkClient then queries ApiVersions
> from
> > > > > >> the source cluster and selects the latestUsableVersion. Two
> outcomes
> > > > > >> are possible:
> > > > > >>
> > > > > >> 1. Source supports fetch v12+: the request succeeds with
> > > > > >> lastFetchedEpoch, and truncation proceeds via the
> truncation-on-fetch
> > > > > >> path.
> > > > > >> 2. Source does not support fetch v12: NetworkClient throws
> > > > > >> UnsupportedVersionException. We catch it, disable
> truncation-on-fetch,
> > > > > >> and fall back to the OffsetsForLeaderEpochRequest path.
> > > > > >>
> > > > > >> This handles both old and new source clusters, including live
> upgrades
> > > > > >> and downgrades.
> > > > > >>
> > > > > >> Unclean leader election handling:
> > > > > >>
> > > > > >> Previously we only handled unclean leader election (ULE) in the
> > > > > >> truncation-on-fetch case. To support older clusters via
> > > > > >> OffsetsForLeaderEpochRequest, we apply the same principle: log
> > > > > >> truncation should never happen on the destination cluster during
> > > > > >> mirroring because we only fetch committed data. If an
> > > > > >> OffsetsForLeaderEpochResponse does trigger truncation, that
> truncation
> > > > > >> must originate from a ULE on the source cluster. In that case,
> we
> > > > > >> transition to the ULE_RECOVERY state and wait for all replicas
> to
> > > > > >> catch up (when mirror.unclean.leader.election.enable=true).
> > > > > >>
> > > > > >> To guarantee that all destination truncations come from source
> ULEs,
> > > > > >> we store the "last mirror offset and last mirror epoch" in the
> LME
> > > > > >> record. Originally, the LME phase one truncated records with
> epoch
> > > > > >> greater than the LME epoch, then phase two relied on normal
> > > > > >> replication to handle log divergence. The problem: some
> truncations
> > > > > >> are side effects of mirroring setup (e.g. reverse mirroring),
> not
> > > > > >> ULEs.
> > > > > >>
> > > > > >> Example (cluster A mirrors from cluster B):
> > > > > >>
> > > > > >> Cluster A (dest) <---- Cluster B (source)
> > > > > >> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> > > > > >> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> > > > > >> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
> > > > > >>
> > > > > >> When cluster A receives the first OffsetsForLeaderEpochResponse
> or
> > > > > >> fetch response from cluster B, offset 2 on cluster A gets
> truncated.
> > > > > >> This truncation is not caused by a ULE. Without the last mirror
> offset
> > > > > >> and epoch in the LME record, we cannot distinguish it from a
> real ULE.
> > > > > >> By storing these values in the LME record, the LOG_ALIGNMENT
> state
> > > > > >> truncates only records where: record offset <= last mirror
> offset, AND
> > > > > >> record epoch <= last mirror epoch. This correctly truncates
> offset 2
> > > > > >> in the example above during LOG_ALIGNMENT. After alignment
> completes,
> > > > > >> any subsequent truncation on the destination is guaranteed to
> > > > > >> originate from a source cluster ULE.
> > > > > >>
> > > > > >> Let us know if you are good with this solution. KIP update will
> > > > follow.
> > > > > >>
> > > > > >> Thanks
> > > > > >> Fede
> > > > > >>
> > > > > >
> > > > >
> > > >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Rajini and Michael. Maintenance is a fair point and we agree on
that. Just one question for my understanding, why 3.0 and not 2.7 that
is where truncation-on-fetch was introduced? Do you see an any
specific problem related to the Fetch protocol?
On Mon, Nov 2, 2026 at 11:46 AM Mickael Maison <mi...@gmail.com> wrote:
>
> Hi,
>
> I agree with Rajini, I'm not sure it's worth re-adding that logic.
> If the migration/upgrade use case is important for you, targeting 3.0+
> should already cover a large portion of outdated clusters.
>
> Thanks,
> Mickael
>
>
> On Mon, Nov 2, 2026 at 7:41 PM Rajini Sivaram <ra...@gmail.com> wrote:
> >
> > Hi Federico,
> >
> > Thanks for the response. I think we removed the old code in
> > AbstractFetcherThread to reduce complexity and maintenance cost. It seems
> > odd that we would bring back all that complexity in a subclass. Not sure if
> > there are users who are still on old broker versions like 2.1 (unlike
> > clients). I would have thought that it is sufficient to support 3.0+. It
> > will be good to understand how much compatibility we expect to maintain for
> > 5.0 and beyond. Let's see what others think too.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Mon, Nov 2, 2026 at 5:58 PM Federico Valeri <fe...@gmail.com> wrote:
> >
> > > Hi Rajini, thanks for letting us know. The schema is still there and we
> > > think it is important to keep support for 2.1 to help old Kafka versions
> > > users to migrate to latest version, so we would like to reintroduce that
> > > logic just for Cluster Mirroring. Wdyt?
> > >
> > > On Mon, Nov 2, 2026 at 2:39 AM Rajini Sivaram <ra...@gmail.com>
> > > wrote:
> > >
> > > > Hi Federico,
> > > >
> > > > KIP-1279 currently states that cluster mirroring source clusters may be
> > > > version >= 2.1. We have been removing support for older metadata versions
> > > > and
> > > >
> > > >
> > > https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8
> > > > removes the explicit truncating phase that sends
> > > > OffsetForLeaderEpochRequest from AbstractFetcherThread. So inter-broker
> > > > replication now relies entirely on truncate-on-fetch. Are we planning to
> > > > introduce OffsetForLeaderEpochRequest in the cluster mirroring fetcher
> > > > because we think there is a requirement to migrate from older clusters?
> > > Or
> > > > are we planning to keep the code simpler and support newer clusters only?
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram <ra...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Federico! The solutions for both RS25 and RS27 sound good.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <fe...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi Rajini,
> > > > >>
> > > > >> >Not sure about RS25, but you mentioned you are looking into it again,
> > > > so
> > > > >> will wait for the response after that.
> > > > >>
> > > > >> The clamping first attempts to read LSO and LEO from the local
> > > > >> ReplicaManager, which is an in-memory lookup against the local
> > > > >> UnifiedLog instance. This works for any partition the coordinator
> > > > >> broker happens to host a replica for. For partitions without a local
> > > > >> replica, the coordinator falls back to the Admin listOffsets API,
> > > > >> which internally routes the request to the partition leader and
> > > > >> returns the offsets. We resolve all mirror partition offsets once at
> > > > >> the beginning of each sync cycle, then reuse them for every group. Not
> > > > >> one call per partition or per group.
> > > > >>
> > > > >> KIP Updated.
> > > > >>
> > > > >> > For RS27, not supporting unclean leader election for older clients
> > > may
> > > > >> be ok unless the fix is straightforward.
> > > > >>
> > > > >> We think it is possible to support old versions with the following
> > > > >> strategy:
> > > > >>
> > > > >> We detect the source cluster's fetch version dynamically. On the first
> > > > >> fetch request, the fetcher thread always includes lastFetchedEpoch in
> > > > >> the request builder. The NetworkClient then queries ApiVersions from
> > > > >> the source cluster and selects the latestUsableVersion. Two outcomes
> > > > >> are possible:
> > > > >>
> > > > >> 1. Source supports fetch v12+: the request succeeds with
> > > > >> lastFetchedEpoch, and truncation proceeds via the truncation-on-fetch
> > > > >> path.
> > > > >> 2. Source does not support fetch v12: NetworkClient throws
> > > > >> UnsupportedVersionException. We catch it, disable truncation-on-fetch,
> > > > >> and fall back to the OffsetsForLeaderEpochRequest path.
> > > > >>
> > > > >> This handles both old and new source clusters, including live upgrades
> > > > >> and downgrades.
> > > > >>
> > > > >> Unclean leader election handling:
> > > > >>
> > > > >> Previously we only handled unclean leader election (ULE) in the
> > > > >> truncation-on-fetch case. To support older clusters via
> > > > >> OffsetsForLeaderEpochRequest, we apply the same principle: log
> > > > >> truncation should never happen on the destination cluster during
> > > > >> mirroring because we only fetch committed data. If an
> > > > >> OffsetsForLeaderEpochResponse does trigger truncation, that truncation
> > > > >> must originate from a ULE on the source cluster. In that case, we
> > > > >> transition to the ULE_RECOVERY state and wait for all replicas to
> > > > >> catch up (when mirror.unclean.leader.election.enable=true).
> > > > >>
> > > > >> To guarantee that all destination truncations come from source ULEs,
> > > > >> we store the "last mirror offset and last mirror epoch" in the LME
> > > > >> record. Originally, the LME phase one truncated records with epoch
> > > > >> greater than the LME epoch, then phase two relied on normal
> > > > >> replication to handle log divergence. The problem: some truncations
> > > > >> are side effects of mirroring setup (e.g. reverse mirroring), not
> > > > >> ULEs.
> > > > >>
> > > > >> Example (cluster A mirrors from cluster B):
> > > > >>
> > > > >> Cluster A (dest) <---- Cluster B (source)
> > > > >> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> > > > >> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> > > > >> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
> > > > >>
> > > > >> When cluster A receives the first OffsetsForLeaderEpochResponse or
> > > > >> fetch response from cluster B, offset 2 on cluster A gets truncated.
> > > > >> This truncation is not caused by a ULE. Without the last mirror offset
> > > > >> and epoch in the LME record, we cannot distinguish it from a real ULE.
> > > > >> By storing these values in the LME record, the LOG_ALIGNMENT state
> > > > >> truncates only records where: record offset <= last mirror offset, AND
> > > > >> record epoch <= last mirror epoch. This correctly truncates offset 2
> > > > >> in the example above during LOG_ALIGNMENT. After alignment completes,
> > > > >> any subsequent truncation on the destination is guaranteed to
> > > > >> originate from a source cluster ULE.
> > > > >>
> > > > >> Let us know if you are good with this solution. KIP update will
> > > follow.
> > > > >>
> > > > >> Thanks
> > > > >> Fede
> > > > >>
> > > > >
> > > >
> > >
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Mickael Maison <mi...@gmail.com>.
Hi,
I agree with Rajini, I'm not sure it's worth re-adding that logic.
If the migration/upgrade use case is important for you, targeting 3.0+
should already cover a large portion of outdated clusters.
Thanks,
Mickael
On Mon, Nov 2, 2026 at 7:41 PM Rajini Sivaram <ra...@gmail.com> wrote:
>
> Hi Federico,
>
> Thanks for the response. I think we removed the old code in
> AbstractFetcherThread to reduce complexity and maintenance cost. It seems
> odd that we would bring back all that complexity in a subclass. Not sure if
> there are users who are still on old broker versions like 2.1 (unlike
> clients). I would have thought that it is sufficient to support 3.0+. It
> will be good to understand how much compatibility we expect to maintain for
> 5.0 and beyond. Let's see what others think too.
>
> Regards,
>
> Rajini
>
>
> On Mon, Nov 2, 2026 at 5:58 PM Federico Valeri <fe...@gmail.com> wrote:
>
> > Hi Rajini, thanks for letting us know. The schema is still there and we
> > think it is important to keep support for 2.1 to help old Kafka versions
> > users to migrate to latest version, so we would like to reintroduce that
> > logic just for Cluster Mirroring. Wdyt?
> >
> > On Mon, Nov 2, 2026 at 2:39 AM Rajini Sivaram <ra...@gmail.com>
> > wrote:
> >
> > > Hi Federico,
> > >
> > > KIP-1279 currently states that cluster mirroring source clusters may be
> > > version >= 2.1. We have been removing support for older metadata versions
> > > and
> > >
> > >
> > https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8
> > > removes the explicit truncating phase that sends
> > > OffsetForLeaderEpochRequest from AbstractFetcherThread. So inter-broker
> > > replication now relies entirely on truncate-on-fetch. Are we planning to
> > > introduce OffsetForLeaderEpochRequest in the cluster mirroring fetcher
> > > because we think there is a requirement to migrate from older clusters?
> > Or
> > > are we planning to keep the code simpler and support newer clusters only?
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > >
> > > On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram <ra...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Federico! The solutions for both RS25 and RS27 sound good.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > > On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <fe...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Rajini,
> > > >>
> > > >> >Not sure about RS25, but you mentioned you are looking into it again,
> > > so
> > > >> will wait for the response after that.
> > > >>
> > > >> The clamping first attempts to read LSO and LEO from the local
> > > >> ReplicaManager, which is an in-memory lookup against the local
> > > >> UnifiedLog instance. This works for any partition the coordinator
> > > >> broker happens to host a replica for. For partitions without a local
> > > >> replica, the coordinator falls back to the Admin listOffsets API,
> > > >> which internally routes the request to the partition leader and
> > > >> returns the offsets. We resolve all mirror partition offsets once at
> > > >> the beginning of each sync cycle, then reuse them for every group. Not
> > > >> one call per partition or per group.
> > > >>
> > > >> KIP Updated.
> > > >>
> > > >> > For RS27, not supporting unclean leader election for older clients
> > may
> > > >> be ok unless the fix is straightforward.
> > > >>
> > > >> We think it is possible to support old versions with the following
> > > >> strategy:
> > > >>
> > > >> We detect the source cluster's fetch version dynamically. On the first
> > > >> fetch request, the fetcher thread always includes lastFetchedEpoch in
> > > >> the request builder. The NetworkClient then queries ApiVersions from
> > > >> the source cluster and selects the latestUsableVersion. Two outcomes
> > > >> are possible:
> > > >>
> > > >> 1. Source supports fetch v12+: the request succeeds with
> > > >> lastFetchedEpoch, and truncation proceeds via the truncation-on-fetch
> > > >> path.
> > > >> 2. Source does not support fetch v12: NetworkClient throws
> > > >> UnsupportedVersionException. We catch it, disable truncation-on-fetch,
> > > >> and fall back to the OffsetsForLeaderEpochRequest path.
> > > >>
> > > >> This handles both old and new source clusters, including live upgrades
> > > >> and downgrades.
> > > >>
> > > >> Unclean leader election handling:
> > > >>
> > > >> Previously we only handled unclean leader election (ULE) in the
> > > >> truncation-on-fetch case. To support older clusters via
> > > >> OffsetsForLeaderEpochRequest, we apply the same principle: log
> > > >> truncation should never happen on the destination cluster during
> > > >> mirroring because we only fetch committed data. If an
> > > >> OffsetsForLeaderEpochResponse does trigger truncation, that truncation
> > > >> must originate from a ULE on the source cluster. In that case, we
> > > >> transition to the ULE_RECOVERY state and wait for all replicas to
> > > >> catch up (when mirror.unclean.leader.election.enable=true).
> > > >>
> > > >> To guarantee that all destination truncations come from source ULEs,
> > > >> we store the "last mirror offset and last mirror epoch" in the LME
> > > >> record. Originally, the LME phase one truncated records with epoch
> > > >> greater than the LME epoch, then phase two relied on normal
> > > >> replication to handle log divergence. The problem: some truncations
> > > >> are side effects of mirroring setup (e.g. reverse mirroring), not
> > > >> ULEs.
> > > >>
> > > >> Example (cluster A mirrors from cluster B):
> > > >>
> > > >> Cluster A (dest) <---- Cluster B (source)
> > > >> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> > > >> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> > > >> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
> > > >>
> > > >> When cluster A receives the first OffsetsForLeaderEpochResponse or
> > > >> fetch response from cluster B, offset 2 on cluster A gets truncated.
> > > >> This truncation is not caused by a ULE. Without the last mirror offset
> > > >> and epoch in the LME record, we cannot distinguish it from a real ULE.
> > > >> By storing these values in the LME record, the LOG_ALIGNMENT state
> > > >> truncates only records where: record offset <= last mirror offset, AND
> > > >> record epoch <= last mirror epoch. This correctly truncates offset 2
> > > >> in the example above during LOG_ALIGNMENT. After alignment completes,
> > > >> any subsequent truncation on the destination is guaranteed to
> > > >> originate from a source cluster ULE.
> > > >>
> > > >> Let us know if you are good with this solution. KIP update will
> > follow.
> > > >>
> > > >> Thanks
> > > >> Fede
> > > >>
> > > >
> > >
> >
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Federico,
Thanks for the response. I think we removed the old code in
AbstractFetcherThread to reduce complexity and maintenance cost. It seems
odd that we would bring back all that complexity in a subclass. Not sure if
there are users who are still on old broker versions like 2.1 (unlike
clients). I would have thought that it is sufficient to support 3.0+. It
will be good to understand how much compatibility we expect to maintain for
5.0 and beyond. Let's see what others think too.
Regards,
Rajini
On Mon, Nov 2, 2026 at 5:58 PM Federico Valeri <fe...@gmail.com> wrote:
> Hi Rajini, thanks for letting us know. The schema is still there and we
> think it is important to keep support for 2.1 to help old Kafka versions
> users to migrate to latest version, so we would like to reintroduce that
> logic just for Cluster Mirroring. Wdyt?
>
> On Mon, Nov 2, 2026 at 2:39 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Hi Federico,
> >
> > KIP-1279 currently states that cluster mirroring source clusters may be
> > version >= 2.1. We have been removing support for older metadata versions
> > and
> >
> >
> https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8
> > removes the explicit truncating phase that sends
> > OffsetForLeaderEpochRequest from AbstractFetcherThread. So inter-broker
> > replication now relies entirely on truncate-on-fetch. Are we planning to
> > introduce OffsetForLeaderEpochRequest in the cluster mirroring fetcher
> > because we think there is a requirement to migrate from older clusters?
> Or
> > are we planning to keep the code simpler and support newer clusters only?
> >
> > Thank you,
> >
> > Rajini
> >
> >
> > On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram <ra...@gmail.com>
> > wrote:
> >
> > > Thanks Federico! The solutions for both RS25 and RS27 sound good.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <fe...@gmail.com>
> > > wrote:
> > >
> > >> Hi Rajini,
> > >>
> > >> >Not sure about RS25, but you mentioned you are looking into it again,
> > so
> > >> will wait for the response after that.
> > >>
> > >> The clamping first attempts to read LSO and LEO from the local
> > >> ReplicaManager, which is an in-memory lookup against the local
> > >> UnifiedLog instance. This works for any partition the coordinator
> > >> broker happens to host a replica for. For partitions without a local
> > >> replica, the coordinator falls back to the Admin listOffsets API,
> > >> which internally routes the request to the partition leader and
> > >> returns the offsets. We resolve all mirror partition offsets once at
> > >> the beginning of each sync cycle, then reuse them for every group. Not
> > >> one call per partition or per group.
> > >>
> > >> KIP Updated.
> > >>
> > >> > For RS27, not supporting unclean leader election for older clients
> may
> > >> be ok unless the fix is straightforward.
> > >>
> > >> We think it is possible to support old versions with the following
> > >> strategy:
> > >>
> > >> We detect the source cluster's fetch version dynamically. On the first
> > >> fetch request, the fetcher thread always includes lastFetchedEpoch in
> > >> the request builder. The NetworkClient then queries ApiVersions from
> > >> the source cluster and selects the latestUsableVersion. Two outcomes
> > >> are possible:
> > >>
> > >> 1. Source supports fetch v12+: the request succeeds with
> > >> lastFetchedEpoch, and truncation proceeds via the truncation-on-fetch
> > >> path.
> > >> 2. Source does not support fetch v12: NetworkClient throws
> > >> UnsupportedVersionException. We catch it, disable truncation-on-fetch,
> > >> and fall back to the OffsetsForLeaderEpochRequest path.
> > >>
> > >> This handles both old and new source clusters, including live upgrades
> > >> and downgrades.
> > >>
> > >> Unclean leader election handling:
> > >>
> > >> Previously we only handled unclean leader election (ULE) in the
> > >> truncation-on-fetch case. To support older clusters via
> > >> OffsetsForLeaderEpochRequest, we apply the same principle: log
> > >> truncation should never happen on the destination cluster during
> > >> mirroring because we only fetch committed data. If an
> > >> OffsetsForLeaderEpochResponse does trigger truncation, that truncation
> > >> must originate from a ULE on the source cluster. In that case, we
> > >> transition to the ULE_RECOVERY state and wait for all replicas to
> > >> catch up (when mirror.unclean.leader.election.enable=true).
> > >>
> > >> To guarantee that all destination truncations come from source ULEs,
> > >> we store the "last mirror offset and last mirror epoch" in the LME
> > >> record. Originally, the LME phase one truncated records with epoch
> > >> greater than the LME epoch, then phase two relied on normal
> > >> replication to handle log divergence. The problem: some truncations
> > >> are side effects of mirroring setup (e.g. reverse mirroring), not
> > >> ULEs.
> > >>
> > >> Example (cluster A mirrors from cluster B):
> > >>
> > >> Cluster A (dest) <---- Cluster B (source)
> > >> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> > >> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> > >> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
> > >>
> > >> When cluster A receives the first OffsetsForLeaderEpochResponse or
> > >> fetch response from cluster B, offset 2 on cluster A gets truncated.
> > >> This truncation is not caused by a ULE. Without the last mirror offset
> > >> and epoch in the LME record, we cannot distinguish it from a real ULE.
> > >> By storing these values in the LME record, the LOG_ALIGNMENT state
> > >> truncates only records where: record offset <= last mirror offset, AND
> > >> record epoch <= last mirror epoch. This correctly truncates offset 2
> > >> in the example above during LOG_ALIGNMENT. After alignment completes,
> > >> any subsequent truncation on the destination is guaranteed to
> > >> originate from a source cluster ULE.
> > >>
> > >> Let us know if you are good with this solution. KIP update will
> follow.
> > >>
> > >> Thanks
> > >> Fede
> > >>
> > >
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Rajini, thanks for letting us know. The schema is still there and we
think it is important to keep support for 2.1 to help old Kafka versions
users to migrate to latest version, so we would like to reintroduce that
logic just for Cluster Mirroring. Wdyt?
On Mon, Nov 2, 2026 at 2:39 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Federico,
>
> KIP-1279 currently states that cluster mirroring source clusters may be
> version >= 2.1. We have been removing support for older metadata versions
> and
>
> https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8
> removes the explicit truncating phase that sends
> OffsetForLeaderEpochRequest from AbstractFetcherThread. So inter-broker
> replication now relies entirely on truncate-on-fetch. Are we planning to
> introduce OffsetForLeaderEpochRequest in the cluster mirroring fetcher
> because we think there is a requirement to migrate from older clusters? Or
> are we planning to keep the code simpler and support newer clusters only?
>
> Thank you,
>
> Rajini
>
>
> On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Thanks Federico! The solutions for both RS25 and RS27 sound good.
> >
> > Regards,
> >
> > Rajini
> >
> > On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <fe...@gmail.com>
> > wrote:
> >
> >> Hi Rajini,
> >>
> >> >Not sure about RS25, but you mentioned you are looking into it again,
> so
> >> will wait for the response after that.
> >>
> >> The clamping first attempts to read LSO and LEO from the local
> >> ReplicaManager, which is an in-memory lookup against the local
> >> UnifiedLog instance. This works for any partition the coordinator
> >> broker happens to host a replica for. For partitions without a local
> >> replica, the coordinator falls back to the Admin listOffsets API,
> >> which internally routes the request to the partition leader and
> >> returns the offsets. We resolve all mirror partition offsets once at
> >> the beginning of each sync cycle, then reuse them for every group. Not
> >> one call per partition or per group.
> >>
> >> KIP Updated.
> >>
> >> > For RS27, not supporting unclean leader election for older clients may
> >> be ok unless the fix is straightforward.
> >>
> >> We think it is possible to support old versions with the following
> >> strategy:
> >>
> >> We detect the source cluster's fetch version dynamically. On the first
> >> fetch request, the fetcher thread always includes lastFetchedEpoch in
> >> the request builder. The NetworkClient then queries ApiVersions from
> >> the source cluster and selects the latestUsableVersion. Two outcomes
> >> are possible:
> >>
> >> 1. Source supports fetch v12+: the request succeeds with
> >> lastFetchedEpoch, and truncation proceeds via the truncation-on-fetch
> >> path.
> >> 2. Source does not support fetch v12: NetworkClient throws
> >> UnsupportedVersionException. We catch it, disable truncation-on-fetch,
> >> and fall back to the OffsetsForLeaderEpochRequest path.
> >>
> >> This handles both old and new source clusters, including live upgrades
> >> and downgrades.
> >>
> >> Unclean leader election handling:
> >>
> >> Previously we only handled unclean leader election (ULE) in the
> >> truncation-on-fetch case. To support older clusters via
> >> OffsetsForLeaderEpochRequest, we apply the same principle: log
> >> truncation should never happen on the destination cluster during
> >> mirroring because we only fetch committed data. If an
> >> OffsetsForLeaderEpochResponse does trigger truncation, that truncation
> >> must originate from a ULE on the source cluster. In that case, we
> >> transition to the ULE_RECOVERY state and wait for all replicas to
> >> catch up (when mirror.unclean.leader.election.enable=true).
> >>
> >> To guarantee that all destination truncations come from source ULEs,
> >> we store the "last mirror offset and last mirror epoch" in the LME
> >> record. Originally, the LME phase one truncated records with epoch
> >> greater than the LME epoch, then phase two relied on normal
> >> replication to handle log divergence. The problem: some truncations
> >> are side effects of mirroring setup (e.g. reverse mirroring), not
> >> ULEs.
> >>
> >> Example (cluster A mirrors from cluster B):
> >>
> >> Cluster A (dest) <---- Cluster B (source)
> >> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
> >>
> >> When cluster A receives the first OffsetsForLeaderEpochResponse or
> >> fetch response from cluster B, offset 2 on cluster A gets truncated.
> >> This truncation is not caused by a ULE. Without the last mirror offset
> >> and epoch in the LME record, we cannot distinguish it from a real ULE.
> >> By storing these values in the LME record, the LOG_ALIGNMENT state
> >> truncates only records where: record offset <= last mirror offset, AND
> >> record epoch <= last mirror epoch. This correctly truncates offset 2
> >> in the example above during LOG_ALIGNMENT. After alignment completes,
> >> any subsequent truncation on the destination is guaranteed to
> >> originate from a source cluster ULE.
> >>
> >> Let us know if you are good with this solution. KIP update will follow.
> >>
> >> Thanks
> >> Fede
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Federico,
KIP-1279 currently states that cluster mirroring source clusters may be
version >= 2.1. We have been removing support for older metadata versions
and
https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8
removes the explicit truncating phase that sends
OffsetForLeaderEpochRequest from AbstractFetcherThread. So inter-broker
replication now relies entirely on truncate-on-fetch. Are we planning to
introduce OffsetForLeaderEpochRequest in the cluster mirroring fetcher
because we think there is a requirement to migrate from older clusters? Or
are we planning to keep the code simpler and support newer clusters only?
Thank you,
Rajini
On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram <ra...@gmail.com>
wrote:
> Thanks Federico! The solutions for both RS25 and RS27 sound good.
>
> Regards,
>
> Rajini
>
> On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <fe...@gmail.com>
> wrote:
>
>> Hi Rajini,
>>
>> >Not sure about RS25, but you mentioned you are looking into it again, so
>> will wait for the response after that.
>>
>> The clamping first attempts to read LSO and LEO from the local
>> ReplicaManager, which is an in-memory lookup against the local
>> UnifiedLog instance. This works for any partition the coordinator
>> broker happens to host a replica for. For partitions without a local
>> replica, the coordinator falls back to the Admin listOffsets API,
>> which internally routes the request to the partition leader and
>> returns the offsets. We resolve all mirror partition offsets once at
>> the beginning of each sync cycle, then reuse them for every group. Not
>> one call per partition or per group.
>>
>> KIP Updated.
>>
>> > For RS27, not supporting unclean leader election for older clients may
>> be ok unless the fix is straightforward.
>>
>> We think it is possible to support old versions with the following
>> strategy:
>>
>> We detect the source cluster's fetch version dynamically. On the first
>> fetch request, the fetcher thread always includes lastFetchedEpoch in
>> the request builder. The NetworkClient then queries ApiVersions from
>> the source cluster and selects the latestUsableVersion. Two outcomes
>> are possible:
>>
>> 1. Source supports fetch v12+: the request succeeds with
>> lastFetchedEpoch, and truncation proceeds via the truncation-on-fetch
>> path.
>> 2. Source does not support fetch v12: NetworkClient throws
>> UnsupportedVersionException. We catch it, disable truncation-on-fetch,
>> and fall back to the OffsetsForLeaderEpochRequest path.
>>
>> This handles both old and new source clusters, including live upgrades
>> and downgrades.
>>
>> Unclean leader election handling:
>>
>> Previously we only handled unclean leader election (ULE) in the
>> truncation-on-fetch case. To support older clusters via
>> OffsetsForLeaderEpochRequest, we apply the same principle: log
>> truncation should never happen on the destination cluster during
>> mirroring because we only fetch committed data. If an
>> OffsetsForLeaderEpochResponse does trigger truncation, that truncation
>> must originate from a ULE on the source cluster. In that case, we
>> transition to the ULE_RECOVERY state and wait for all replicas to
>> catch up (when mirror.unclean.leader.election.enable=true).
>>
>> To guarantee that all destination truncations come from source ULEs,
>> we store the "last mirror offset and last mirror epoch" in the LME
>> record. Originally, the LME phase one truncated records with epoch
>> greater than the LME epoch, then phase two relied on normal
>> replication to handle log divergence. The problem: some truncations
>> are side effects of mirroring setup (e.g. reverse mirroring), not
>> ULEs.
>>
>> Example (cluster A mirrors from cluster B):
>>
>> Cluster A (dest) <---- Cluster B (source)
>> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
>>
>> When cluster A receives the first OffsetsForLeaderEpochResponse or
>> fetch response from cluster B, offset 2 on cluster A gets truncated.
>> This truncation is not caused by a ULE. Without the last mirror offset
>> and epoch in the LME record, we cannot distinguish it from a real ULE.
>> By storing these values in the LME record, the LOG_ALIGNMENT state
>> truncates only records where: record offset <= last mirror offset, AND
>> record epoch <= last mirror epoch. This correctly truncates offset 2
>> in the example above during LOG_ALIGNMENT. After alignment completes,
>> any subsequent truncation on the destination is guaranteed to
>> originate from a source cluster ULE.
>>
>> Let us know if you are good with this solution. KIP update will follow.
>>
>> Thanks
>> Fede
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Rajini Sivaram <ra...@gmail.com>.
Thanks Federico! The solutions for both RS25 and RS27 sound good.
Regards,
Rajini
On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <fe...@gmail.com>
wrote:
> Hi Rajini,
>
> >Not sure about RS25, but you mentioned you are looking into it again, so
> will wait for the response after that.
>
> The clamping first attempts to read LSO and LEO from the local
> ReplicaManager, which is an in-memory lookup against the local
> UnifiedLog instance. This works for any partition the coordinator
> broker happens to host a replica for. For partitions without a local
> replica, the coordinator falls back to the Admin listOffsets API,
> which internally routes the request to the partition leader and
> returns the offsets. We resolve all mirror partition offsets once at
> the beginning of each sync cycle, then reuse them for every group. Not
> one call per partition or per group.
>
> KIP Updated.
>
> > For RS27, not supporting unclean leader election for older clients may
> be ok unless the fix is straightforward.
>
> We think it is possible to support old versions with the following
> strategy:
>
> We detect the source cluster's fetch version dynamically. On the first
> fetch request, the fetcher thread always includes lastFetchedEpoch in
> the request builder. The NetworkClient then queries ApiVersions from
> the source cluster and selects the latestUsableVersion. Two outcomes
> are possible:
>
> 1. Source supports fetch v12+: the request succeeds with
> lastFetchedEpoch, and truncation proceeds via the truncation-on-fetch
> path.
> 2. Source does not support fetch v12: NetworkClient throws
> UnsupportedVersionException. We catch it, disable truncation-on-fetch,
> and fall back to the OffsetsForLeaderEpochRequest path.
>
> This handles both old and new source clusters, including live upgrades
> and downgrades.
>
> Unclean leader election handling:
>
> Previously we only handled unclean leader election (ULE) in the
> truncation-on-fetch case. To support older clusters via
> OffsetsForLeaderEpochRequest, we apply the same principle: log
> truncation should never happen on the destination cluster during
> mirroring because we only fetch committed data. If an
> OffsetsForLeaderEpochResponse does trigger truncation, that truncation
> must originate from a ULE on the source cluster. In that case, we
> transition to the ULE_RECOVERY state and wait for all replicas to
> catch up (when mirror.unclean.leader.election.enable=true).
>
> To guarantee that all destination truncations come from source ULEs,
> we store the "last mirror offset and last mirror epoch" in the LME
> record. Originally, the LME phase one truncated records with epoch
> greater than the LME epoch, then phase two relied on normal
> replication to handle log divergence. The problem: some truncations
> are side effects of mirroring setup (e.g. reverse mirroring), not
> ULEs.
>
> Example (cluster A mirrors from cluster B):
>
> Cluster A (dest) <---- Cluster B (source)
> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
>
> When cluster A receives the first OffsetsForLeaderEpochResponse or
> fetch response from cluster B, offset 2 on cluster A gets truncated.
> This truncation is not caused by a ULE. Without the last mirror offset
> and epoch in the LME record, we cannot distinguish it from a real ULE.
> By storing these values in the LME record, the LOG_ALIGNMENT state
> truncates only records where: record offset <= last mirror offset, AND
> record epoch <= last mirror epoch. This correctly truncates offset 2
> in the example above during LOG_ALIGNMENT. After alignment completes,
> any subsequent truncation on the destination is guaranteed to
> originate from a source cluster ULE.
>
> Let us know if you are good with this solution. KIP update will follow.
>
> Thanks
> Fede
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Rajini,
>Not sure about RS25, but you mentioned you are looking into it again, so will wait for the response after that.
The clamping first attempts to read LSO and LEO from the local
ReplicaManager, which is an in-memory lookup against the local
UnifiedLog instance. This works for any partition the coordinator
broker happens to host a replica for. For partitions without a local
replica, the coordinator falls back to the Admin listOffsets API,
which internally routes the request to the partition leader and
returns the offsets. We resolve all mirror partition offsets once at
the beginning of each sync cycle, then reuse them for every group. Not
one call per partition or per group.
KIP Updated.
> For RS27, not supporting unclean leader election for older clients may be ok unless the fix is straightforward.
We think it is possible to support old versions with the following strategy:
We detect the source cluster's fetch version dynamically. On the first
fetch request, the fetcher thread always includes lastFetchedEpoch in
the request builder. The NetworkClient then queries ApiVersions from
the source cluster and selects the latestUsableVersion. Two outcomes
are possible:
1. Source supports fetch v12+: the request succeeds with
lastFetchedEpoch, and truncation proceeds via the truncation-on-fetch
path.
2. Source does not support fetch v12: NetworkClient throws
UnsupportedVersionException. We catch it, disable truncation-on-fetch,
and fall back to the OffsetsForLeaderEpochRequest path.
This handles both old and new source clusters, including live upgrades
and downgrades.
Unclean leader election handling:
Previously we only handled unclean leader election (ULE) in the
truncation-on-fetch case. To support older clusters via
OffsetsForLeaderEpochRequest, we apply the same principle: log
truncation should never happen on the destination cluster during
mirroring because we only fetch committed data. If an
OffsetsForLeaderEpochResponse does trigger truncation, that truncation
must originate from a ULE on the source cluster. In that case, we
transition to the ULE_RECOVERY state and wait for all replicas to
catch up (when mirror.unclean.leader.election.enable=true).
To guarantee that all destination truncations come from source ULEs,
we store the "last mirror offset and last mirror epoch" in the LME
record. Originally, the LME phase one truncated records with epoch
greater than the LME epoch, then phase two relied on normal
replication to handle log divergence. The problem: some truncations
are side effects of mirroring setup (e.g. reverse mirroring), not
ULEs.
Example (cluster A mirrors from cluster B):
Cluster A (dest) <---- Cluster B (source)
Offset 0: A (LE: 0) Offset 0: A (LE: 0)
Offset 1: A (LE: 0) Offset 1: A (LE: 0)
Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10)
When cluster A receives the first OffsetsForLeaderEpochResponse or
fetch response from cluster B, offset 2 on cluster A gets truncated.
This truncation is not caused by a ULE. Without the last mirror offset
and epoch in the LME record, we cannot distinguish it from a real ULE.
By storing these values in the LME record, the LOG_ALIGNMENT state
truncates only records where: record offset <= last mirror offset, AND
record epoch <= last mirror epoch. This correctly truncates offset 2
in the example above during LOG_ALIGNMENT. After alignment completes,
any subsequent truncation on the destination is guaranteed to
originate from a source cluster ULE.
Let us know if you are good with this solution. KIP update will follow.
Thanks
Fede
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Federico,
Thanks for the responses. Not sure about RS25, but you mentioned you are
looking into it again, so will wait for the response after that. For RS27,
not supporting unclean leader election for older clients may be ok unless
the fix is straightforward. The rest of the responses sound good.
Thanks,
Rajini
On Thu, Aug 27, 2026 at 11:27 AM Federico Valeri <fe...@gmail.com>
wrote:
> Hi Rajini, I missed on question yesterday:
>
> >RS22: What are the failure scenarios from which a mirror topic can be
> >recovered using RecoverMirrorTopics
>
> Transient failures that exhaust retries. Network partitions, temporary
> authentication issues, coordinator unavailability. The underlying
> cause may resolve on its own.
>
> 2. Tiered storage enabled on destination mirror topic during log
> truncation. Partitions move to FAILED. Once tiered storage is
> disabled, recovery makes sense.
> 3. Mirror loop detection. If bidirectional mirroring is detected for
> the same topics, affected partitions enter FAILED. Recovery works once
> the conflicting mirror is removed.
>
> > and what does recovery mean for log
> >consistency?
>
> Recovery does not deal with log consistency. It simply moves the
> partition back to its previous state (e.g., LOG_ALIGNMENT or
> MIRRORING) and resets the retry counter. Log consistency is handled by
> whichever state the partition returns to: if it returns to
> LOG_ALIGNMENT, truncation happens there; if it returns to MIRRORING,
> the fetch protocol's leader epoch comparison handles divergence.
>
> >Aren't there terminal failures which prevent recovery, e.g. if
> >source topic was deleted and recreated?
>
> Yes. The implementation distinguishes retryable from non-retryable
> failures (KIP updated). Source topic deletion is detected through two
> consecutive metadata syncs to avoid false positives when the source
> cluster is still loading metadata. Once confirmed, the failure is
> marked non-retryable. For non-retryable failures a manual recovery is
> required before invoking RecoverMirrorTopics.
>
> ---
>
> Looking again into RS25. Will let you know.
>
> We are still discussing RS27 to see if there is a way to support ULE
> with older versions. Will let you know.
>
> For RS28 we will adopt your proposal.
>
> Thanks
>
>
>
>
> On Wed, Aug 26, 2026 at 5:41 PM Federico Valeri <fe...@gmail.com>
> wrote:
> >
> > Hi Rajini, thanks for checking KIP updates.
> >
> > >RS20: Does `mirror.metadata.refresh.interval.ms` control how often
> consumer
> > >offsets, ACLs etc. are sync'ed as well, in addition to topic metadata?
> >
> > Yes, mirror.metadata.refresh.interval.ms controls how often ALL
> > metadata is synced, including topic configs, consumer offsets and
> > ACLs. To be precise, every broker syncs topic metadata, but only the
> > one hashing to the particular mirror name syncs topic configs, group
> > offsets, acls, and new new/deleted topics matching patterns. This
> > periodic metadata refresh process runs in a dedicated scheduler
> > thread.
> >
> > >RS21: If we are supporting source versions without topic ids, how are we
> > >detecting if a source topic was deleted and recreated? Will we end up
> with
> > >mirror topics that contain records from both the old and new
> incarnation of
> > >the topic?
> >
> > For pre-2.8 source clusters there is a fallback to name-based lookup
> > so the problem you mention can actually happen. I think there is no
> > solution to that gap if we want to support pre-2.8 releases. In that
> > case the operator needs to take extra care and manually stop the
> > mirror topic and delete the destination topic before re-creating it,
> > so mirroring starts fresh against the new incarnation. KIP updated.
> >
> > >RS23: ConfigType currently has `CLUSTER_MIRRORS("cluster-mirrors")`. To
> > >match existing entries, should we change to `
> > >CLUSTER_MIRROR("cluster-mirrors")`?
> >
> > Sure, KIP updated.
> >
> > >RS24: "*Group ids with active members on the destination cluster are
> > >retrieved with a ListGroups RPC and skipped with a warning log*"
> > >Is this round trip request to group coordinator necessary? It isn't
> > >watertight anyway since a group could add a member after the RPC before
> the
> > >commit. Since group coordinators reject OffsetCommitRequest anyway if
> the
> > >group has members, do we need the extra RPC, potentially to every broker
> > >per commit cycle, to check state?
> >
> > I agree, the pre-check could be removed. The alterConsumerGroupOffsets
> > would fail per-group with an appropriate error. That approach is both
> > simpler and more correct since the rejection happens atomically at the
> > coordinator. KIP updated.
> >
> > >RS25: KIP says: "*During offset synchronization, the committed offset is
> > >clamped to the valid range of the destination partition using the
> formula
> > >max(dest log start offset, min(dest LEO, source committed offset))"*
> > >Does this mean that every offset commit operation needs to get log
> offsets
> > >of partitions from leader brokers?
> >
> > No. The clamping reads LSO and LEO from the local ReplicaManager.
> > These are in-memory lookups against the local UnifiedLog instance, not
> > RPCs to leader brokers. This works because offset sync only runs on
> > the coordinator broker, and mirror topics are destination-local topics
> > whose log state is available on any broker that hosts a replica. If
> > the local broker doesn't host the partition (log is absent), that
> > partition is skipped.
> >
> > >RS26: Is "mirror.admin.listener.name" only used to send requests to
> other
> > >brokers or is it used for broker->controller as well? From the
> discussion
> > >on this thread, it sounded like it was both. And that doesn't seem right
> > >since listeners can be different on brokers and controllers. For
> controller
> > >connections, we could perhaps use the same configs as existing
> > >broker->controller connections since we also rely on ClusterAction ACLs,
> > >e.g. for BumpLeaderEpochs.
> >
> > We have dedicated channels with different listener configurations:
> >
> > 1. Controller requests (channelManager.sendRequest) uses the shared
> > broker-to-controller forwarding channel. This uses
> > control.plane.listener.name / the controller's listener, not
> > mirror.admin.listener.name.
> > 2. Destination Admin client (dstAdmin) uses mirror.admin.listener.name
> > to find the broker's own advertised endpoint and security config for
> > broker-to-broker RPCs (offset commits, start/stop mirror topics).
> >
> > >RS27: The KIP says that the minimum supported source version is 2.1.
> Which
> > >means we need to support source clusters without topic ids, which seems
> ok
> > >and the KIP describes this. It also means we need to support source
> > >clusters which don't support truncation-on-fetch. The KIP says: "*Mirror
> > >fetcher threads use the explicit OffsetsForLeaderEpochRequest path for
> log
> > >truncation rather than truncation-on-fetch (diverging epoch in fetch
> > >responses) for older version source clusters (Fetch API v11 or older).
> *"
> > >I don't think Fetch API v11 is sufficient to support
> truncation-on-fetch.
> > >If I remember correctly, we had some issues that were fixed later and
> the
> > >minimum version is 2.7. The KIP describes the truncation-on-fetch mode
> in
> > >detail, e.g. for handling unclean leader election, but not the older
> style
> > >*OffsetsForLeaderEpochRequest *path. How does the fetcher thread
> determine
> > >the mode
> >
> > It doesn't dynamically detect the source version. Mirror threads
> > always start with the explicit OffsetsForLeaderEpochRequest path. But
> > diverging epochs in fetch responses are still honored when present.
> >
> > We have two paths depending on the source cluster version:
> >
> > Proactive (diverging epoch, source >= 2.7): The fetcher includes
> > lastFetchedEpoch in every fetch request. The source compares this
> > against its own epoch lineage. If the epochs diverge (ULE changed the
> > leader lineage), the source returns a divergingEpoch in the response
> > immediately, even if the fetcher's offset is still valid. Detection
> > happens on the very next fetch, regardless of offset position.
> >
> > Reactive (OFFSET_OUT_OF_RANGE, source < 2.7): The source ignores
> > lastFetchedEpoch (doesn't understand the field). The only signal is
> > when the fetcher requests an offset that no longer exists on the
> > source because the ULE truncated the log below the fetcher's position.
> > If the fetcher is at offset 100 and the ULE truncated the source to
> > offset 50, the next fetch returns OFFSET_OUT_OF_RANGE and the fetcher
> > corrects. But if the ULE truncated to offset 90 and new records have
> > already been produced past 100, the fetcher sees no error and keeps
> > appending from the diverged log. The divergence is never detected.
> >
> > In short, we can only support ULE with source cluster on 2.7+. KIP
> updated.
> >
> > > - can it handle upgrade/downgrade of the source cluster?
> >
> > Not explicitly. The initial truncation path is fixed. If a source
> > cluster upgrades mid-mirroring, the fetcher won't switch to
> > truncation-on-fetch for initial truncation.
> >
> > >How will
> > >unclean leader election etc. differ? It will be good to understand the
> > >additional complexity for supporting pre-2.7 source clusters.
> >
> > Supporting pre-2.7 means the ULE recovery path described in the KIP
> > (which relies on diverging epochs in fetch responses) simply won't
> > work for those sources. I think it's fine to explicitly document that.
> >
> > >RS28: The semantics of the config
> > >`mirror.support.unclean.leader.election` seems
> > >odd. We perform unclean truncation below high watermark even when the
> > >config is true. For deployments where unclean leader election is
> disabled
> > >and likely to trigger alerts, it will be useful to have a config that
> > >prevents unclean truncation and moves the mirror topic to failed state.
> > >Have we considered just using a config `
> > >mirror.unclean.leader.election.enable=false` to prevent unclean
> truncation?
> > >Then `mirror.unclean.leader.election.enable=true` would transition the
> > >topic to ULE_RECOVERY. Not sure if the in-between state which truncates,
> > >but leaves the partition in MIRRORING state is that useful.
> >
> > Good point. With your proposal, the operator gets a clear choice:
> > accept data loss from ULE (mirror.unclean.leader.election.enable=true)
> > or halt mirroring and move to FAILED (=false). The tradeoff is that
> > both paths may require manual recovery. When ULE is enabled,
> > ULE_RECOVERY waits for all assigned replicas to converge with no
> > timeout. A single stuck or decommissioned replica that never rejoins
> > ISR blocks mirroring for that partition indefinitely.
> >
> > Let me discuss it with the team and we will update.
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Rajini, I missed on question yesterday:
>RS22: What are the failure scenarios from which a mirror topic can be
>recovered using RecoverMirrorTopics
Transient failures that exhaust retries. Network partitions, temporary
authentication issues, coordinator unavailability. The underlying
cause may resolve on its own.
2. Tiered storage enabled on destination mirror topic during log
truncation. Partitions move to FAILED. Once tiered storage is
disabled, recovery makes sense.
3. Mirror loop detection. If bidirectional mirroring is detected for
the same topics, affected partitions enter FAILED. Recovery works once
the conflicting mirror is removed.
> and what does recovery mean for log
>consistency?
Recovery does not deal with log consistency. It simply moves the
partition back to its previous state (e.g., LOG_ALIGNMENT or
MIRRORING) and resets the retry counter. Log consistency is handled by
whichever state the partition returns to: if it returns to
LOG_ALIGNMENT, truncation happens there; if it returns to MIRRORING,
the fetch protocol's leader epoch comparison handles divergence.
>Aren't there terminal failures which prevent recovery, e.g. if
>source topic was deleted and recreated?
Yes. The implementation distinguishes retryable from non-retryable
failures (KIP updated). Source topic deletion is detected through two
consecutive metadata syncs to avoid false positives when the source
cluster is still loading metadata. Once confirmed, the failure is
marked non-retryable. For non-retryable failures a manual recovery is
required before invoking RecoverMirrorTopics.
---
Looking again into RS25. Will let you know.
We are still discussing RS27 to see if there is a way to support ULE
with older versions. Will let you know.
For RS28 we will adopt your proposal.
Thanks
On Wed, Aug 26, 2026 at 5:41 PM Federico Valeri <fe...@gmail.com> wrote:
>
> Hi Rajini, thanks for checking KIP updates.
>
> >RS20: Does `mirror.metadata.refresh.interval.ms` control how often consumer
> >offsets, ACLs etc. are sync'ed as well, in addition to topic metadata?
>
> Yes, mirror.metadata.refresh.interval.ms controls how often ALL
> metadata is synced, including topic configs, consumer offsets and
> ACLs. To be precise, every broker syncs topic metadata, but only the
> one hashing to the particular mirror name syncs topic configs, group
> offsets, acls, and new new/deleted topics matching patterns. This
> periodic metadata refresh process runs in a dedicated scheduler
> thread.
>
> >RS21: If we are supporting source versions without topic ids, how are we
> >detecting if a source topic was deleted and recreated? Will we end up with
> >mirror topics that contain records from both the old and new incarnation of
> >the topic?
>
> For pre-2.8 source clusters there is a fallback to name-based lookup
> so the problem you mention can actually happen. I think there is no
> solution to that gap if we want to support pre-2.8 releases. In that
> case the operator needs to take extra care and manually stop the
> mirror topic and delete the destination topic before re-creating it,
> so mirroring starts fresh against the new incarnation. KIP updated.
>
> >RS23: ConfigType currently has `CLUSTER_MIRRORS("cluster-mirrors")`. To
> >match existing entries, should we change to `
> >CLUSTER_MIRROR("cluster-mirrors")`?
>
> Sure, KIP updated.
>
> >RS24: "*Group ids with active members on the destination cluster are
> >retrieved with a ListGroups RPC and skipped with a warning log*"
> >Is this round trip request to group coordinator necessary? It isn't
> >watertight anyway since a group could add a member after the RPC before the
> >commit. Since group coordinators reject OffsetCommitRequest anyway if the
> >group has members, do we need the extra RPC, potentially to every broker
> >per commit cycle, to check state?
>
> I agree, the pre-check could be removed. The alterConsumerGroupOffsets
> would fail per-group with an appropriate error. That approach is both
> simpler and more correct since the rejection happens atomically at the
> coordinator. KIP updated.
>
> >RS25: KIP says: "*During offset synchronization, the committed offset is
> >clamped to the valid range of the destination partition using the formula
> >max(dest log start offset, min(dest LEO, source committed offset))"*
> >Does this mean that every offset commit operation needs to get log offsets
> >of partitions from leader brokers?
>
> No. The clamping reads LSO and LEO from the local ReplicaManager.
> These are in-memory lookups against the local UnifiedLog instance, not
> RPCs to leader brokers. This works because offset sync only runs on
> the coordinator broker, and mirror topics are destination-local topics
> whose log state is available on any broker that hosts a replica. If
> the local broker doesn't host the partition (log is absent), that
> partition is skipped.
>
> >RS26: Is "mirror.admin.listener.name" only used to send requests to other
> >brokers or is it used for broker->controller as well? From the discussion
> >on this thread, it sounded like it was both. And that doesn't seem right
> >since listeners can be different on brokers and controllers. For controller
> >connections, we could perhaps use the same configs as existing
> >broker->controller connections since we also rely on ClusterAction ACLs,
> >e.g. for BumpLeaderEpochs.
>
> We have dedicated channels with different listener configurations:
>
> 1. Controller requests (channelManager.sendRequest) uses the shared
> broker-to-controller forwarding channel. This uses
> control.plane.listener.name / the controller's listener, not
> mirror.admin.listener.name.
> 2. Destination Admin client (dstAdmin) uses mirror.admin.listener.name
> to find the broker's own advertised endpoint and security config for
> broker-to-broker RPCs (offset commits, start/stop mirror topics).
>
> >RS27: The KIP says that the minimum supported source version is 2.1. Which
> >means we need to support source clusters without topic ids, which seems ok
> >and the KIP describes this. It also means we need to support source
> >clusters which don't support truncation-on-fetch. The KIP says: "*Mirror
> >fetcher threads use the explicit OffsetsForLeaderEpochRequest path for log
> >truncation rather than truncation-on-fetch (diverging epoch in fetch
> >responses) for older version source clusters (Fetch API v11 or older). *"
> >I don't think Fetch API v11 is sufficient to support truncation-on-fetch.
> >If I remember correctly, we had some issues that were fixed later and the
> >minimum version is 2.7. The KIP describes the truncation-on-fetch mode in
> >detail, e.g. for handling unclean leader election, but not the older style
> >*OffsetsForLeaderEpochRequest *path. How does the fetcher thread determine
> >the mode
>
> It doesn't dynamically detect the source version. Mirror threads
> always start with the explicit OffsetsForLeaderEpochRequest path. But
> diverging epochs in fetch responses are still honored when present.
>
> We have two paths depending on the source cluster version:
>
> Proactive (diverging epoch, source >= 2.7): The fetcher includes
> lastFetchedEpoch in every fetch request. The source compares this
> against its own epoch lineage. If the epochs diverge (ULE changed the
> leader lineage), the source returns a divergingEpoch in the response
> immediately, even if the fetcher's offset is still valid. Detection
> happens on the very next fetch, regardless of offset position.
>
> Reactive (OFFSET_OUT_OF_RANGE, source < 2.7): The source ignores
> lastFetchedEpoch (doesn't understand the field). The only signal is
> when the fetcher requests an offset that no longer exists on the
> source because the ULE truncated the log below the fetcher's position.
> If the fetcher is at offset 100 and the ULE truncated the source to
> offset 50, the next fetch returns OFFSET_OUT_OF_RANGE and the fetcher
> corrects. But if the ULE truncated to offset 90 and new records have
> already been produced past 100, the fetcher sees no error and keeps
> appending from the diverged log. The divergence is never detected.
>
> In short, we can only support ULE with source cluster on 2.7+. KIP updated.
>
> > - can it handle upgrade/downgrade of the source cluster?
>
> Not explicitly. The initial truncation path is fixed. If a source
> cluster upgrades mid-mirroring, the fetcher won't switch to
> truncation-on-fetch for initial truncation.
>
> >How will
> >unclean leader election etc. differ? It will be good to understand the
> >additional complexity for supporting pre-2.7 source clusters.
>
> Supporting pre-2.7 means the ULE recovery path described in the KIP
> (which relies on diverging epochs in fetch responses) simply won't
> work for those sources. I think it's fine to explicitly document that.
>
> >RS28: The semantics of the config
> >`mirror.support.unclean.leader.election` seems
> >odd. We perform unclean truncation below high watermark even when the
> >config is true. For deployments where unclean leader election is disabled
> >and likely to trigger alerts, it will be useful to have a config that
> >prevents unclean truncation and moves the mirror topic to failed state.
> >Have we considered just using a config `
> >mirror.unclean.leader.election.enable=false` to prevent unclean truncation?
> >Then `mirror.unclean.leader.election.enable=true` would transition the
> >topic to ULE_RECOVERY. Not sure if the in-between state which truncates,
> >but leaves the partition in MIRRORING state is that useful.
>
> Good point. With your proposal, the operator gets a clear choice:
> accept data loss from ULE (mirror.unclean.leader.election.enable=true)
> or halt mirroring and move to FAILED (=false). The tradeoff is that
> both paths may require manual recovery. When ULE is enabled,
> ULE_RECOVERY waits for all assigned replicas to converge with no
> timeout. A single stuck or decommissioned replica that never rejoins
> ISR blocks mirroring for that partition indefinitely.
>
> Let me discuss it with the team and we will update.
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Rajini, thanks for checking KIP updates.
>RS20: Does `mirror.metadata.refresh.interval.ms` control how often consumer
>offsets, ACLs etc. are sync'ed as well, in addition to topic metadata?
Yes, mirror.metadata.refresh.interval.ms controls how often ALL
metadata is synced, including topic configs, consumer offsets and
ACLs. To be precise, every broker syncs topic metadata, but only the
one hashing to the particular mirror name syncs topic configs, group
offsets, acls, and new new/deleted topics matching patterns. This
periodic metadata refresh process runs in a dedicated scheduler
thread.
>RS21: If we are supporting source versions without topic ids, how are we
>detecting if a source topic was deleted and recreated? Will we end up with
>mirror topics that contain records from both the old and new incarnation of
>the topic?
For pre-2.8 source clusters there is a fallback to name-based lookup
so the problem you mention can actually happen. I think there is no
solution to that gap if we want to support pre-2.8 releases. In that
case the operator needs to take extra care and manually stop the
mirror topic and delete the destination topic before re-creating it,
so mirroring starts fresh against the new incarnation. KIP updated.
>RS23: ConfigType currently has `CLUSTER_MIRRORS("cluster-mirrors")`. To
>match existing entries, should we change to `
>CLUSTER_MIRROR("cluster-mirrors")`?
Sure, KIP updated.
>RS24: "*Group ids with active members on the destination cluster are
>retrieved with a ListGroups RPC and skipped with a warning log*"
>Is this round trip request to group coordinator necessary? It isn't
>watertight anyway since a group could add a member after the RPC before the
>commit. Since group coordinators reject OffsetCommitRequest anyway if the
>group has members, do we need the extra RPC, potentially to every broker
>per commit cycle, to check state?
I agree, the pre-check could be removed. The alterConsumerGroupOffsets
would fail per-group with an appropriate error. That approach is both
simpler and more correct since the rejection happens atomically at the
coordinator. KIP updated.
>RS25: KIP says: "*During offset synchronization, the committed offset is
>clamped to the valid range of the destination partition using the formula
>max(dest log start offset, min(dest LEO, source committed offset))"*
>Does this mean that every offset commit operation needs to get log offsets
>of partitions from leader brokers?
No. The clamping reads LSO and LEO from the local ReplicaManager.
These are in-memory lookups against the local UnifiedLog instance, not
RPCs to leader brokers. This works because offset sync only runs on
the coordinator broker, and mirror topics are destination-local topics
whose log state is available on any broker that hosts a replica. If
the local broker doesn't host the partition (log is absent), that
partition is skipped.
>RS26: Is "mirror.admin.listener.name" only used to send requests to other
>brokers or is it used for broker->controller as well? From the discussion
>on this thread, it sounded like it was both. And that doesn't seem right
>since listeners can be different on brokers and controllers. For controller
>connections, we could perhaps use the same configs as existing
>broker->controller connections since we also rely on ClusterAction ACLs,
>e.g. for BumpLeaderEpochs.
We have dedicated channels with different listener configurations:
1. Controller requests (channelManager.sendRequest) uses the shared
broker-to-controller forwarding channel. This uses
control.plane.listener.name / the controller's listener, not
mirror.admin.listener.name.
2. Destination Admin client (dstAdmin) uses mirror.admin.listener.name
to find the broker's own advertised endpoint and security config for
broker-to-broker RPCs (offset commits, start/stop mirror topics).
>RS27: The KIP says that the minimum supported source version is 2.1. Which
>means we need to support source clusters without topic ids, which seems ok
>and the KIP describes this. It also means we need to support source
>clusters which don't support truncation-on-fetch. The KIP says: "*Mirror
>fetcher threads use the explicit OffsetsForLeaderEpochRequest path for log
>truncation rather than truncation-on-fetch (diverging epoch in fetch
>responses) for older version source clusters (Fetch API v11 or older). *"
>I don't think Fetch API v11 is sufficient to support truncation-on-fetch.
>If I remember correctly, we had some issues that were fixed later and the
>minimum version is 2.7. The KIP describes the truncation-on-fetch mode in
>detail, e.g. for handling unclean leader election, but not the older style
>*OffsetsForLeaderEpochRequest *path. How does the fetcher thread determine
>the mode
It doesn't dynamically detect the source version. Mirror threads
always start with the explicit OffsetsForLeaderEpochRequest path. But
diverging epochs in fetch responses are still honored when present.
We have two paths depending on the source cluster version:
Proactive (diverging epoch, source >= 2.7): The fetcher includes
lastFetchedEpoch in every fetch request. The source compares this
against its own epoch lineage. If the epochs diverge (ULE changed the
leader lineage), the source returns a divergingEpoch in the response
immediately, even if the fetcher's offset is still valid. Detection
happens on the very next fetch, regardless of offset position.
Reactive (OFFSET_OUT_OF_RANGE, source < 2.7): The source ignores
lastFetchedEpoch (doesn't understand the field). The only signal is
when the fetcher requests an offset that no longer exists on the
source because the ULE truncated the log below the fetcher's position.
If the fetcher is at offset 100 and the ULE truncated the source to
offset 50, the next fetch returns OFFSET_OUT_OF_RANGE and the fetcher
corrects. But if the ULE truncated to offset 90 and new records have
already been produced past 100, the fetcher sees no error and keeps
appending from the diverged log. The divergence is never detected.
In short, we can only support ULE with source cluster on 2.7+. KIP updated.
> - can it handle upgrade/downgrade of the source cluster?
Not explicitly. The initial truncation path is fixed. If a source
cluster upgrades mid-mirroring, the fetcher won't switch to
truncation-on-fetch for initial truncation.
>How will
>unclean leader election etc. differ? It will be good to understand the
>additional complexity for supporting pre-2.7 source clusters.
Supporting pre-2.7 means the ULE recovery path described in the KIP
(which relies on diverging epochs in fetch responses) simply won't
work for those sources. I think it's fine to explicitly document that.
>RS28: The semantics of the config
>`mirror.support.unclean.leader.election` seems
>odd. We perform unclean truncation below high watermark even when the
>config is true. For deployments where unclean leader election is disabled
>and likely to trigger alerts, it will be useful to have a config that
>prevents unclean truncation and moves the mirror topic to failed state.
>Have we considered just using a config `
>mirror.unclean.leader.election.enable=false` to prevent unclean truncation?
>Then `mirror.unclean.leader.election.enable=true` would transition the
>topic to ULE_RECOVERY. Not sure if the in-between state which truncates,
>but leaves the partition in MIRRORING state is that useful.
Good point. With your proposal, the operator gets a clear choice:
accept data loss from ULE (mirror.unclean.leader.election.enable=true)
or halt mirroring and move to FAILED (=false). The tradeoff is that
both paths may require manual recovery. When ULE is enabled,
ULE_RECOVERY waits for all assigned replicas to converge with no
timeout. A single stuck or decommissioned replica that never rejoins
ISR blocks mirroring for that partition indefinitely.
Let me discuss it with the team and we will update.
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi team,
Thanks for the updated KIP. It has come a long way and is looking really
good. I have a few more questions.
RS20: Does `mirror.metadata.refresh.interval.ms` control how often consumer
offsets, ACLs etc. are sync'ed as well, in addition to topic metadata?
RS21: If we are supporting source versions without topic ids, how are we
detecting if a source topic was deleted and recreated? Will we end up with
mirror topics that contain records from both the old and new incarnation of
the topic?
RS22: What are the failure scenarios from which a mirror topic can be
recovered using RecoverMirrorTopics and what does recovery mean for log
consistency? Aren't there terminal failures which prevent recovery, e.g. if
source topic was deleted and recreated?
RS23: ConfigType currently has `CLUSTER_MIRRORS("cluster-mirrors")`. To
match existing entries, should we change to `
CLUSTER_MIRROR("cluster-mirrors")`?
RS24: "*Group ids with active members on the destination cluster are
retrieved with a ListGroups RPC and skipped with a warning log*"
Is this round trip request to group coordinator necessary? It isn't
watertight anyway since a group could add a member after the RPC before the
commit. Since group coordinators reject OffsetCommitRequest anyway if the
group has members, do we need the extra RPC, potentially to every broker
per commit cycle, to check state?
RS25: KIP says: "*During offset synchronization, the committed offset is
clamped to the valid range of the destination partition using the formula
max(dest log start offset, min(dest LEO, source committed offset))"*
Does this mean that every offset commit operation needs to get log offsets
of partitions from leader brokers?
RS26: Is "mirror.admin.listener.name" only used to send requests to other
brokers or is it used for broker->controller as well? From the discussion
on this thread, it sounded like it was both. And that doesn't seem right
since listeners can be different on brokers and controllers. For controller
connections, we could perhaps use the same configs as existing
broker->controller connections since we also rely on ClusterAction ACLs,
e.g. for BumpLeaderEpochs.
RS27: The KIP says that the minimum supported source version is 2.1. Which
means we need to support source clusters without topic ids, which seems ok
and the KIP describes this. It also means we need to support source
clusters which don't support truncation-on-fetch. The KIP says: "*Mirror
fetcher threads use the explicit OffsetsForLeaderEpochRequest path for log
truncation rather than truncation-on-fetch (diverging epoch in fetch
responses) for older version source clusters (Fetch API v11 or older). *"
I don't think Fetch API v11 is sufficient to support truncation-on-fetch.
If I remember correctly, we had some issues that were fixed later and the
minimum version is 2.7. The KIP describes the truncation-on-fetch mode in
detail, e.g. for handling unclean leader election, but not the older style
*OffsetsForLeaderEpochRequest *path. How does the fetcher thread determine
the mode - can it handle upgrade/downgrade of the source cluster? How will
unclean leader election etc. differ? It will be good to understand the
additional complexity for supporting pre-2.7 source clusters.
RS28: The semantics of the config
`mirror.support.unclean.leader.election` seems
odd. We perform unclean truncation below high watermark even when the
config is true. For deployments where unclean leader election is disabled
and likely to trigger alerts, it will be useful to have a config that
prevents unclean truncation and moves the mirror topic to failed state.
Have we considered just using a config `
mirror.unclean.leader.election.enable=false` to prevent unclean truncation?
Then `mirror.unclean.leader.election.enable=true` would transition the
topic to ULE_RECOVERY. Not sure if the in-between state which truncates,
but leaves the partition in MIRRORING state is that useful.
Regards,
Rajini
On Tue, Aug 18, 2026 at 3:09 PM Jun Rao via dev <de...@kafka.apache.org>
wrote:
> Hi, Luke,
>
> Thanks for the reply. The KIP LGTM now.
>
> Jun
>
> On Tue, Aug 18, 2026 at 1:08 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the comments.
> > Answering them below.
> >
> > > JR80. AdminClient.startMirrorTopics(): Is includePatterns() optional?
> It
> > seems that mirroring can't start without it.
> >
> > Ah, right. I’ve moved the includePatterns from the
> > StartMirrorTopicsOptions into the required parameter in
> > AdminClient.startMirrorTopics().
> >
> >
> > > JR91. Mirror configs: Some of the configs are prefixed with "mirror,"
> > while
> > others are not. Could we make them consistent? Since all those configs
> are
> > for Mirror, the mirror prefix seems redundant.
> > Indeed, we added the “mirror” prefix because we were afraid of confusing
> > users like “mirror.fetch.min.bytes” and “replica.fetch.min.bytes”. But
> like
> > you said, these configurations are for CLUSTER_MIRROR resource type, not
> > for brokers. Users should be able to distinguish between them easily.
> > Removed the mirror prefix from the mirror-level configs.
> >
> > Thank you,
> > Luke
> >
> >
> > On Tue, Aug 18, 2026 at 2:04 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Luke,
> >>
> >> Thanks for the reply.
> >>
> >> JR80. AdminClient.startMirrorTopics(): Is includePatterns() optional? It
> >> seems that mirroring can't start without it.
> >>
> >> JR91. Mirror configs: Some of the configs are prefixed with "mirror,"
> >> while
> >> others are not. Could we make them consistent? Since all those configs
> are
> >> for Mirror, the mirror prefix seems redundant.
> >>
> >> Jun
> >>
> >>
> >> On Mon, Aug 17, 2026 at 1:11 AM Luke Chen <sh...@gmail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thanks for the good comments.
> >> > Answering them below.
> >> >
> >> > > JR80. In that case, it seems that there is no need for
> >> > AdminClient.startMirrorTopics() to take topics. The adminClient only
> >> uses
> >> > it to register the included topics, which can be passed in as
> >> > includePatterns (specific names or regex). The broker will use
> >> > StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
> >> > through the AdminClient.
> >> >
> >> > Make sense. So when users invoke admin.startMirrorTopics, they only
> need
> >> > to set the topic patterns. And the receiving broker will resolve the
> >> > patterns into the actual topics using the listTopic response from the
> >> > source cluster, and get the topic id, numPartitions, … via the
> >> > describeTopics request from the source cluster if the topic is
> >> non-existent
> >> > in the destination cluster. KIP is updated.
> >> >
> >> >
> >> > > JR82. If “IncludeMirrorOffset” is disabled,
> >> > DescribeClusterMirrorsResponse.SourceOffset is still returned in the
> >> > response and what is its value in this case?
> >> >
> >> > It will be -1 because when the `IncludeMirrorOffset` is disabled, we
> >> won’t
> >> > query the mirror partition leader. I’ve updated the description to
> make
> >> it
> >> > clear. Thanks.
> >> >
> >> >
> >> > > JR88. Why is partition-load-time a Meter instead of a Gauge?
> >> >
> >> > You’re right. It is not the Meter type. It’s the SampledStat state
> like
> >> > “batch-size-avg” and “batch-size-max” in producer. I’ve updated the
> KIP.
> >> >
> >> >
> >> > > JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
> >> >
> >> > You’re right. It’s the rate type. KIP is updated
> >> >
> >> >
> >> > > JR90. MetadataRefreshError : Is that cumulative? If so, it should
> be a
> >> > Meter.
> >> > Yes, it makes more sense to make it as Meter type. I’ve updated these
> >> > error-related metrics to Meter type:
> >> > 1. MetadataRefreshError
> >> > 2. TopicConfigMetadataSyncError
> >> > 3. ConsumerGroupOffsetSyncError
> >> > 4. ShareGroupOffsetSyncError
> >> > 5. AclSyncError
> >> >
> >> >
> >> > Thank you,
> >> > Luke
> >> >
> >> >
> >> > On Sat, Aug 15, 2026 at 8:10 AM Jun Rao via dev <dev@kafka.apache.org
> >
> >> > wrote:
> >> >
> >> >> Hi, Luke,
> >> >>
> >> >> Thanks for the reply.
> >> >>
> >> >> JR80. In that case, it seems that there is no need for
> >> >> AdminClient.startMirrorTopics() to take topics. The adminClient only
> >> uses
> >> >> it to register the included topics, which can be passed in as
> >> >> includePatterns (specific names or regex). The broker will use
> >> >> StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
> >> >> through the AdminClient.
> >> >>
> >> >> JR82. If “IncludeMirrorOffset” is disabled,
> >> >> DescribeClusterMirrorsResponse.SourceOffset is still returned in the
> >> >> response and what is its value in this case?
> >> >>
> >> >> JR88. Why is partition-load-time a Meter instead of a Gauge?
> >> >>
> >> >> JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
> >> >>
> >> >> JR90. MetadataRefreshError : Is that cumulative? If so, it should be
> a
> >> >> Meter.
> >> >>
> >> >> Jun
> >> >>
> >> >> On Fri, Aug 14, 2026 at 4:32 AM Luke Chen <sh...@gmail.com> wrote:
> >> >>
> >> >> > Hi Jun,
> >> >> >
> >> >> > Thanks for the good comments,
> >> >> > Answering them below.
> >> >> >
> >> >> > > JR80. PauseMirrorTopicsOptions:
> >> "PauseMirrorTopics/ResumeMirrorTopics
> >> >> > supports regex in the ClusterMirrorComamnd, but we don’t need to
> add
> >> the
> >> >> > “patterns”."
> >> >> > So, in this case, we use topics to support both specific topics and
> >> >> regex.
> >> >> > However, startMirrorTopicsOptions uses topics for specific topics
> and
> >> >> > includePatterns for regex. Could we make them consistent?
> >> >> >
> >> >> >
> >> >> > The purpose of the “includePatterns” field is for the
> >> >> > “mirror.topics.include” update, so that if there is any newly
> created
> >> >> topic
> >> >> > in the source cluster that matches the regex, we can create it. And
> >> the
> >> >> > “Topics” field in the StartMirrorTopicsRequest is to specify the
> >> topicId
> >> >> > and numPartitions in the source cluster. So I think
> “includePatterns”
> >> >> and
> >> >> > “Topics” both are necessary in the StartMirrorTopicsRequest. On the
> >> >> other
> >> >> > hand, in PauseMirrorTopics, because it won’t change the
> >> >> > “mirror.topics.include”/”mirror.topics.exclude” configs, we don’t
> >> need
> >> >> to
> >> >> > provide the regex in the request.
> >> >> >
> >> >> >
> >> >> >
> >> >> > > JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value
> >> also
> >> >> -1
> >> >> > if
> >> >> > it's not included in the request?
> >> >> >
> >> >> > If “IncludeMirrorOffset” is disabled, we won’t query the mirror
> >> >> partition
> >> >> > leader, so we can’t get the sourceOffset.
> >> >> >
> >> >> > I assume you want to ask `if it’s still -1 if not actively
> mirroring,
> >> >> > while “IncludeMirrorOffset” is enabled`. That’s a good question. I
> >> think
> >> >> > our purpose of this command is to show the lag to users. So if the
> >> >> state is
> >> >> > not MIRRORING (ex: STOPPED/PAUSED), we think it’s not helpful to
> >> users
> >> >> to
> >> >> > show only the sourceOffset. Of course users can use
> >> >> `kafka-get-offsets.sh`
> >> >> > to get the offset if they really want to know.
> >> >> >
> >> >> >
> >> >> > > JR83. Start Mirror Topics:
> >> >> > > JR83.1 Should the controller verify the expected state for the
> >> topic?
> >> >> >
> >> >> > As replied in JR84, the desired state validation should happen in
> the
> >> >> > broker. And the controller validates the StateOffset. I’ve updated
> >> the
> >> >> > workflow to make it clear.
> >> >> >
> >> >> > > JR83.2 "d. truncates the log accordingly,"
> >> >> > Could you add a description of the truncation offset?
> >> >> >
> >> >> > OK, added one more sentence:
> >> >> > d. based on the last mirror epoch response in (b), the broker
> >> translates
> >> >> > the epoch into the offset by querying the leaderEpochCache
> >> >> > (endOffsetForEpoch), and truncates the log accordingly, and waits
> for
> >> >> all
> >> >> > ISR members…
> >> >> >
> >> >> >
> >> >> > > JR84. Stop Mirror Topics: "The controller verifies that the
> desired
> >> >> state
> >> >> > of all topics are in MIRRORING or PAUSED state."
> >> >> > Should the broker verfy the topic level state? I thought the
> >> controller
> >> >> > will use StateOffset in the request to make sure the validation on
> >> the
> >> >> > broker is stil valid. If the controller performs the verification,
> is
> >> >> > StateOffset still useful?
> >> >> >
> >> >> > Oh, you’re right. We’re doing duplicated work here. Yes, the broker
> >> >> > already verifies the “desired state” of all topics are in MIRRORING
> >> or
> >> >> > PAUSED state, as well as the “actual state” is in MIRRORING or
> PAUSED
> >> >> > state. If all validation passes, we’ll forward it to the
> controller.
> >> So
> >> >> the
> >> >> > controller only needs to validate the StateOffset to make sure the
> >> >> > validation on the broker is still valid. So, yes, we don’t need the
> >> >> desired
> >> >> > state validation on the controller side. I’ve updated the KIP to
> >> make it
> >> >> > clear.
> >> >> >
> >> >> > One more thing to add is that, I’ve thought about not adding
> >> >> StateOffset,
> >> >> > and relying on the desired state validation to make sure we’re safe
> >> to
> >> >> do
> >> >> > the deletion or other operations. The problem is that, even if the
> >> >> broker
> >> >> > sees the desired state for topic A is STOPPED, and then the
> >> controller
> >> >> also
> >> >> > sees the desired state is STOPPED, it still cannot make sure the
> >> >> > controller’s STOPPED state equals the broker’s STOPPED state. For
> >> >> example:
> >> >> > 1. Broker sees STOPPED, then forward to controller
> >> >> > 2. User changes the desired state to MIRRORING
> >> >> > 3. User changes the desired state to STOPPED
> >> >> > 4. Controller sees the desired state is STOPPED
> >> >> > So I think StateOffset is required.
> >> >> >
> >> >> > Thanks for catching it BTW.
> >> >> >
> >> >> >
> >> >> > > JR85. There are still a few references to includeMirrorLags. They
> >> >> should
> >> >> > be
> >> >> > IncludeMirrorOffset.
> >> >> >
> >> >> > Sorry, just updated.
> >> >> >
> >> >> >
> >> >> > > JR86. Metrics:
> >> >> > > JR86.1 Could you describe the type (gauge, meter, etc) of each
> >> metric?
> >> >> >
> >> >> > KIP is updated. I also group the metrics into new added metrics and
> >> >> other
> >> >> > inherited metrics.
> >> >> >
> >> >> > > JR86.2 Could you describe a bit more what MinFetchRate
> represents?
> >> >> >
> >> >> > It means the lowest fetch request rate among all of this mirror
> >> fetcher
> >> >> > manager's fetcher threads. It inherits from the
> >> AbstractFetcherManager’s
> >> >> > metric.
> >> >> >
> >> >> >
> >> >> > > JR87. Version Compatibility: "For migration or DR use cases where
> >> >> > failback
> >> >> > is not required, this proposal supports source brokers up to
> version
> >> 2.1
> >> >> > included"
> >> >> > This KIP needs topicId and KIP-516 was introduced later than 2.1.
> >> >> > Yes, when mirroring from a cluster without topic id support, we
> will
> >> >> > assign a random topic ID in the destination cluster and then fetch
> >> >> (without
> >> >> > topic id in the fetch request using the old version of Fetch API)
> as
> >> >> usual.
> >> >> > Of course this is for the one-way mirroring use case. It can’t
> start
> >> the
> >> >> > reverse mirror(failback) under this setup. You can check the
> >> >> “Compatibility
> >> >> > Matrix” section for more detail.
> >> >> >
> >> >> >
> >> >> > Thank you,
> >> >> > Luke
> >> >> >
> >> >> > On Fri, Aug 14, 2026 at 7:09 AM Jun Rao via dev <
> >> dev@kafka.apache.org>
> >> >> > wrote:
> >> >> >
> >> >> >> Hi, Luke,
> >> >> >>
> >> >> >> Thanks for the reply.
> >> >> >>
> >> >> >> JR80. PauseMirrorTopicsOptions:
> >> "PauseMirrorTopics/ResumeMirrorTopics
> >> >> >> supports regex in the ClusterMirrorComamnd, but we don’t need to
> add
> >> >> the
> >> >> >> “patterns”."
> >> >> >> So, in this case, we use topics to support both specific topics
> and
> >> >> regex.
> >> >> >> However, startMirrorTopicsOptions uses topics for specific topics
> >> and
> >> >> >> includePatterns for regex. Could we make them consistent?
> >> >> >>
> >> >> >> JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value
> also
> >> >> -1 if
> >> >> >> it's not included in the request?
> >> >> >>
> >> >> >> JR83. Start Mirror Topics:
> >> >> >> JR83.1 Should the controller verify the expected state for the
> >> topic?
> >> >> >> JR83.2 "d. truncates the log accordingly,"
> >> >> >> Could you add a description of the truncation offset?
> >> >> >>
> >> >> >> JR84. Stop Mirror Topics: "The controller verifies that the
> desired
> >> >> state
> >> >> >> of all topics are in MIRRORING or PAUSED state."
> >> >> >> Should the broker verfy the topic level state? I thought the
> >> controller
> >> >> >> will use StateOffset in the request to make sure the validation on
> >> the
> >> >> >> broker is stil valid. If the controller performs the verification,
> >> is
> >> >> >> StateOffset still useful?
> >> >> >>
> >> >> >> JR85. There are still a few references to includeMirrorLags. They
> >> >> should
> >> >> >> be
> >> >> >> IncludeMirrorOffset.
> >> >> >>
> >> >> >> JR86. Metrics:
> >> >> >> JR86.1 Could you describe the type (gauge, meter, etc) of each
> >> metric?
> >> >> >> JR86.2 Could you describe a bit more what MinFetchRate represents?
> >> >> >>
> >> >> >> JR87. Version Compatibility: "For migration or DR use cases where
> >> >> failback
> >> >> >> is not required, this proposal supports source brokers up to
> version
> >> >> 2.1
> >> >> >> included"
> >> >> >> This KIP needs topicId and KIP-516 was introduced later than 2.1.
> >> >> >>
> >> >> >> Jun
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Aug 13, 2026 at 5:14 AM Luke Chen <sh...@gmail.com>
> >> wrote:
> >> >> >>
> >> >> >> > Hi Jun,
> >> >> >> >
> >> >> >> > Thanks for the good comments.
> >> >> >> > Answering them below.
> >> >> >> >
> >> >> >> > > JR59.1 Could you add the RPC to the workflow? The broker first
> >> >> needs
> >> >> >> to
> >> >> >> > issue ListClusterMirrorsRequest with SourceClusterIdFilter to
> get
> >> the
> >> >> >> > mirror name and then issue DescribeClusterMirrors to get the
> LME.
> >> >> >> >
> >> >> >> > Yes, you’re right. The workflow is updated to make them clear.
> >> >> >> >
> >> >> >> > > JR71. mirror.support.unclean.leader.election : The name is
> >> >> confusing.
> >> >> >> > It's
> >> >> >> > also used in LOG_TRUNCATION, which is not related to unclean
> >> leader
> >> >> >> > election. Could we use a better name?
> >> >> >> >
> >> >> >> > I personally think the name should be fine because even though
> we
> >> >> use in
> >> >> >> > “LOG_TRUNCATION” state, the config is to allow us to know if
> users
> >> >> want
> >> >> >> to
> >> >> >> > support unclean leader election, if so, we wait for all replicas
> >> to
> >> >> >> catch
> >> >> >> > up. If we want to explicitly show the intention of the config,
> how
> >> >> about
> >> >> >> > “mirror.full.replica.convergence.required”? I’m open to other
> >> >> >> suggestions.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR72. DescribeClusterMirrors
> >> >> >> > > JR72.1 A user may specify just the topic name when describing
> a
> >> >> >> mirror.
> >> >> >> > Should we include the topic name in the request?
> >> >> >> >
> >> >> >> > Hmm.. we didn’t think about the use case. But yes, if users only
> >> care
> >> >> >> > about the state of a topic, they don’t need the info for other
> >> >> topics.
> >> >> >> I’ve
> >> >> >> > combined with JR72.2, added “TopicPartitions” filed into
> >> >> >> > DescribeClusterMirrorsRequest, and by default is null, which
> means
> >> >> all
> >> >> >> > topic partitions under the requested mirrors will be returned.
> If
> >> >> set,
> >> >> >> > we’ll query the mirrors and return the topic partitions that
> users
> >> >> >> > expected. KIP is updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR72.2 LastMirrorEpochLookups is a bit weird since the
> response
> >> >> >> returns
> >> >> >> > either mirror state or mirror lag and LastMirrorEpochLookups
> >> doesn't
> >> >> >> match
> >> >> >> > either.
> >> >> >> >
> >> >> >> > Yes, after the recent refactor, I think the
> LastMirrorEpochLookups
> >> >> can
> >> >> >> be
> >> >> >> > merged into one level up, combining with the topic name filter
> in
> >> >> above
> >> >> >> > comment, which will allow users to query down to the partition
> >> level.
> >> >> >> The
> >> >> >> > LastMirrorEpochLookups will query along with normal mirror state
> >> by
> >> >> the
> >> >> >> > readMirrorState from the coordinator. The difference is that
> after
> >> >> >> > receiving the response, it will filter out the mirrors that
> don’t
> >> >> match
> >> >> >> the
> >> >> >> > requested sourceClusterId, and then add the lastMirrorEpoch into
> >> >> >> > DescribeClusterMirrorResponse. KIP is updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> >> >
> >> >> >> > Right. Added.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR72.4 "includeMirrorLags: source offset, destination offset,
> >> lag"
> >> >> >> > DescribeClusterMirrorsRequest doesn't include those fields.
> Also,
> >> lag
> >> >> >> > should be omitted.
> >> >> >> >
> >> >> >> > Sorry, I removed the “SourceOffset” and “DestinationOffset”
> fields
> >> >> from
> >> >> >> > DescribeClusterMirrorsResponse because I added them into
> >> >> >> ReadMirrorOffsets
> >> >> >> > (ex-ReadMirrorLags) API. But I forgot we still need them in the
> >> >> >> > DescribeClusterMirrorsResponse so that the adminClient can
> receive
> >> >> the
> >> >> >> > results. KIP is updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR73. MirrorPartitionState record, LastMirrorEpochs record and
> >> >> >> > MirrorPartitionStateRequest: The values for type and name are
> >> >> incorrect.
> >> >> >> >
> >> >> >> > Ah, that’s my bad. Fixed.
> >> >> >> >
> >> >> >> > > JR74. ListClusterMirrorsRequest
> >> >> >> > > JR74.1 When is DesiredMirrorStateFilter used? There is no
> >> option to
> >> >> >> > specify
> >> >> >> > the desired state in the mirror-topic command.
> >> >> >> >
> >> >> >> > Yes, referred to kafka-consumer-groups.sh, I added the `--state`
> >> >> option
> >> >> >> to
> >> >> >> > allow users to set the state when listMirror.
> >> >> >> >
> >> >> >> > > JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED
> too?
> >> >> >> >
> >> >> >> > Yes, KIP updated.
> >> >> >> >
> >> >> >> > > JR75. ReadMirrorLags
> >> >> >> > > JR75.1 Should it be named ReadMirrorOffsets since the response
> >> >> doesn't
> >> >> >> > include lag?
> >> >> >> >
> >> >> >> > Indeed. Updated.
> >> >> >> >
> >> >> >> > > JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> >> >
> >> >> >> > Yes, added
> >> >> >> >
> >> >> >> > > JR75.3 Some places refer to this as ReadClusterMirrorLags. We
> >> need
> >> >> to
> >> >> >> be
> >> >> >> > consistent.
> >> >> >> >
> >> >> >> > Thanks for pointing it out. KIP is updated.
> >> >> >> >
> >> >> >> > > JR76. WriteMirrorStates: Should it return
> UNKNOWN_CLUSTER_MIRROR
> >> >> too?
> >> >> >> >
> >> >> >> > Yes, we should validate the mirror name before writing the data.
> >> KIP
> >> >> is
> >> >> >> > updated.
> >> >> >> >
> >> >> >> > > JR77. Delete Mirror: "3. The controller verifies that no
> topics
> >> >> are in
> >> >> >> > MIRRORING or PAUSED state."
> >> >> >> > Is this necessary since step 2 has verified all partitions are
> in
> >> >> >> STOPPED
> >> >> >> > state, which can only happen if the topics are in STOPPED state.
> >> >> >> >
> >> >> >> > It’s true, but there could be some delay between “actual state”
> >> and
> >> >> >> > “desired state”. In step (2), we verify all partitions’ “actual
> >> >> state”
> >> >> >> are
> >> >> >> > in STOPPED state. But the current desired state could be
> >> MIRRORING,
> >> >> it’s
> >> >> >> > just that the mirror partition leaders haven’t got the metadata
> >> >> update
> >> >> >> yet.
> >> >> >> > So, the controller still needs to verify the desired state is
> not
> >> in
> >> >> >> > MIRRORING or PAUSED. I found we didn’t specify that the
> >> controller is
> >> >> >> > actually verifying the “desired state”, not “actual state”. And
> >> this
> >> >> >> > controller’s desired state validation also applies in
> >> >> >> >
> >> >>
> >> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
> >> >> >> > I’ve updated the KIP to make it clear.
> >> >> >> >
> >> >> >> > > JR78. STOPPING: "The mirror fetcher is removed, the leader
> >> epoch is
> >> >> >> > bumped,
> >> >> >> > ABORT markers are appended for all ongoing transactions, last
> >> mirror
> >> >> >> epochs
> >> >> >> > are persisted, and a MIRROR_PID_RESET record appended."
> >> >> >> > This describes the state for STOPPED, not STOPPING.
> >> >> >> >
> >> >> >> > No, these are the tasks during STOPPING state, like what we
> >> >> described in
> >> >> >> > “Stop Mirror Topics” workflow. If we do these in STOPPED state,
> we
> >> >> can’t
> >> >> >> > safely move the readonly topic into writable because not all
> >> required
> >> >> >> tasks
> >> >> >> > are done. I think I know why it confuses you, it’s because all
> the
> >> >> tasks
> >> >> >> > are in the completed state. I’ve rephrased it to make it clear.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR79. RPCs like DescribeClusterMirrors can take time. Should
> we
> >> >> add a
> >> >> >> > timeout field?
> >> >> >> >
> >> >> >> > Make sense. Added into `DescribeClusterMirrors` request, as well
> >> as
> >> >> >> >
> >> >> >>
> >> >>
> >>
> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
> >> >> >> > because we will query the partition states from coordinator for
> >> state
> >> >> >> > validation in these APIs.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR80. PauseMirrorTopicsOptions: Should it support patterns?
> The
> >> >> >> command
> >> >> >> > line seems to suppor it.
> >> >> >> >
> >> >> >> > Sorry, it’s a little confusing here. The “pattern” in
> >> >> >> > startMirrorTopicsOptions or stopMirrorTopicsOptions is to update
> >> the
> >> >> >> > “mirror.topics.include” and “mirror.topics.exclude”
> configuration
> >> for
> >> >> >> topic
> >> >> >> > pattern discovery periodically. For the PauseMirrorTopics and
> >> >> >> > ResumeMirrorTopics in the clusterMirrorCommand, users can
> provide
> >> >> regex
> >> >> >> in
> >> >> >> > the “--topics” option, and the clusterMirrorCommand will
> >> listTopic in
> >> >> >> the
> >> >> >> > destination cluster and match with the patterns users provided,
> >> and
> >> >> then
> >> >> >> > feeds the matched topics into the
> >> >> PauseMirrorTopics/ResumeMirrorTopics
> >> >> >> > requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics
> >> >> >> supports
> >> >> >> > regex in the ClusterMirrorComamnd, but we don’t need to add the
> >> >> >> “patterns”.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR81. AdminClient
> >> >> >> > > JR81.1 ListClusterMirrors: We need to add the option to
> specify
> >> >> >> different
> >> >> >> > filters supported in the request.
> >> >> >> >
> >> >> >> > We already have the `--mirror` option. Added `--state` and
> >> >> >> > `--sourceClusterId` options to support the other filters.
> >> >> >> >
> >> >> >> > > JR81.2
> >> DescribeClusterMirrorsOptions.includeAuthorizedOperations:
> >> >> The
> >> >> >> > request doesn't support this option. Could we add other options
> >> >> >> supported
> >> >> >> > in the request?
> >> >> >> >
> >> >> >> > Currently, we have “includeAuthorizedOperations” in
> >> DescribeCluster,
> >> >> >> > DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
> >> >> >> > DescribeStreamsGroups requests. But none of them include the
> >> option
> >> >> in
> >> >> >> the
> >> >> >> > tool to allow users to enable it. I think the reason is that we
> >> don’t
> >> >> >> want
> >> >> >> > to show this to users on the tool output. But having this field,
> >> we
> >> >> >> allow
> >> >> >> > the advanced users to invoke admin API with
> >> >> includeAuthorizedOperations
> >> >> >> > enabled to get the results they want.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR81.3 LeaderStateDescription: Should it be named
> >> >> >> > MirrorPartitionStateDescription? Also, could we make the fields
> >> >> >> consistent
> >> >> >> > with those in the response?
> >> >> >> >
> >> >> >> > OK. KIP is updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > Thank you,
> >> >> >> > Luke
> >> >> >> >
> >> >> >> > On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <
> >> >> dev@kafka.apache.org>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Hi, Luke,
> >> >> >> >>
> >> >> >> >> Thanks for the reply.
> >> >> >> >>
> >> >> >> >> JR59.1 Could you add the RPC to the workflow? The broker first
> >> >> needs to
> >> >> >> >> issue ListClusterMirrorsRequest with SourceClusterIdFilter to
> get
> >> >> the
> >> >> >> >> mirror name and then issue DescribeClusterMirrors to get the
> LME.
> >> >> >> >>
> >> >> >> >> JR71. mirror.support.unclean.leader.election : The name is
> >> >> confusing.
> >> >> >> It's
> >> >> >> >> also used in LOG_TRUNCATION, which is not related to unclean
> >> leader
> >> >> >> >> election. Could we use a better name?
> >> >> >> >>
> >> >> >> >> JR72. DescribeClusterMirrors
> >> >> >> >> JR72.1 A user may specify just the topic name when describing a
> >> >> mirror.
> >> >> >> >> Should we include the topic name in the request?
> >> >> >> >> JR72.2 LastMirrorEpochLookups is a bit weird since the response
> >> >> returns
> >> >> >> >> either mirror state or mirror lag and LastMirrorEpochLookups
> >> doesn't
> >> >> >> match
> >> >> >> >> either.
> >> >> >> >> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> >> >> JR72.4 "includeMirrorLags: source offset, destination offset,
> >> lag"
> >> >> >> >> DescribeClusterMirrorsRequest doesn't include those fields.
> Also,
> >> >> lag
> >> >> >> >> should be omitted.
> >> >> >> >>
> >> >> >> >> JR73. MirrorPartitionState record, LastMirrorEpochs record and
> >> >> >> >> MirrorPartitionStateRequest: The values for type and name are
> >> >> >> incorrect.
> >> >> >> >>
> >> >> >> >> JR74. ListClusterMirrorsRequest
> >> >> >> >> JR74.1 When is DesiredMirrorStateFilter used? There is no
> option
> >> to
> >> >> >> >> specify
> >> >> >> >> the desired state in the mirror-topic command.
> >> >> >> >> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED
> too?
> >> >> >> >>
> >> >> >> >> JR75. ReadMirrorLags
> >> >> >> >> JR75.1 Should it be named ReadMirrorOffsets since the response
> >> >> doesn't
> >> >> >> >> include lag?
> >> >> >> >> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> >> >> JR75.3 Some places refer to this as ReadClusterMirrorLags. We
> >> need
> >> >> to
> >> >> >> be
> >> >> >> >> consistent.
> >> >> >> >>
> >> >> >> >> JR76. WriteMirrorStates: Should it return
> UNKNOWN_CLUSTER_MIRROR
> >> >> too?
> >> >> >> >>
> >> >> >> >> JR77. Delete Mirror: "3. The controller verifies that no topics
> >> are
> >> >> in
> >> >> >> >> MIRRORING or PAUSED state."
> >> >> >> >> Is this necessary since step 2 has verified all partitions are
> in
> >> >> >> STOPPED
> >> >> >> >> state, which can only happen if the topics are in STOPPED
> state.
> >> >> >> >>
> >> >> >> >> JR78. STOPPING: "The mirror fetcher is removed, the leader
> epoch
> >> is
> >> >> >> >> bumped,
> >> >> >> >> ABORT markers are appended for all ongoing transactions, last
> >> mirror
> >> >> >> >> epochs
> >> >> >> >> are persisted, and a MIRROR_PID_RESET record appended."
> >> >> >> >> This describes the state for STOPPED, not STOPPING.
> >> >> >> >>
> >> >> >> >> JR79. RPCs like DescribeClusterMirrors can take time. Should we
> >> add
> >> >> a
> >> >> >> >> timeout field?
> >> >> >> >>
> >> >> >> >> JR80. PauseMirrorTopicsOptions: Should it support patterns? The
> >> >> command
> >> >> >> >> line seems to suppor it.
> >> >> >> >>
> >> >> >> >> JR81. AdminClient
> >> >> >> >> JR81.1 ListClusterMirrors: We need to add the option to specify
> >> >> >> different
> >> >> >> >> filters supported in the request.
> >> >> >> >> JR81.2
> DescribeClusterMirrorsOptions.includeAuthorizedOperations:
> >> >> The
> >> >> >> >> request doesn't support this option. Could we add other options
> >> >> >> supported
> >> >> >> >> in the request?
> >> >> >> >> JR81.3 LeaderStateDescription: Should it be named
> >> >> >> >> MirrorPartitionStateDescription? Also, could we make the fields
> >> >> >> consistent
> >> >> >> >> with those in the response?
> >> >> >> >>
> >> >> >> >> Jun
> >> >> >> >>
> >> >> >> >> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com>
> >> >> wrote:
> >> >> >> >>
> >> >> >> >> > Hi Jun,
> >> >> >> >> >
> >> >> >> >> > Thanks for the good comments again.
> >> >> >> >> > Answering them below.
> >> >> >> >> >
> >> >> >> >> > > JR51. "ULE_LOG_TRUNCATION When
> >> >> >> mirror.support.unclean.leader.election
> >> >> >> >> is
> >> >> >> >> > enabled and the destination leader has log truncation, the
> >> state
> >> >> will
> >> >> >> >> move
> >> >> >> >> > from MIRRORING to this state."
> >> >> >> >> > This still suggests that divergeEpoch is only handled if
> >> >> >> >> > mirror.support.unclean.leader.election
> >> >> >> >> > is enabled.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > OK, KIP is updated to make it clear.
> >> >> >> >> >
> >> >> >> >> > > JR55. Should the mirror state and the mirror lags always be
> >> >> exposed
> >> >> >> >> > together? If so, we can remove IncludeMirrorState and
> >> >> >> IncludeMirrorLags
> >> >> >> >> > from DescribeClusterMirrorsRequest. If not, it's better to
> use
> >> two
> >> >> >> >> separate
> >> >> >> >> > RPCs.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > No, we sometimes only need MirrorState, and sometimes need
> >> both.
> >> >> So
> >> >> >> I’ll
> >> >> >> >> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
> >> >> >> >> > DescribeClusterMirrorsRequest. And we will use
> ReadMirrorStates
> >> >> and
> >> >> >> >> > ReadMirrorLags to get each info.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > DescribeClusterMirrorsRequest should be used in the
> Describe
> >> >> Mirror
> >> >> >> >> flow,
> >> >> >> >> > right? Currently it's missing. Also, I assume the admin
> client
> >> >> >> refers to
> >> >> >> >> > the one in the kafka-cluster-mirrors tool. But it seems that
> >> >> >> sometimes
> >> >> >> >> it
> >> >> >> >> > also refers to the admin client on the broker? Perhaps we can
> >> use
> >> >> >> 'the
> >> >> >> >> > tool' instead to make it clear.
> >> >> >> >> >
> >> >> >> >> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
> >> >> >> >> > And sorry for the confusion. I’d like to keep the admin
> client
> >> as
> >> >> the
> >> >> >> >> one
> >> >> >> >> > invoking from the tool (or user’s direct admin client API
> >> >> >> invocation).
> >> >> >> >> On
> >> >> >> >> > the broker side, we actually use InterBrokerSendThread to
> >> invoke
> >> >> >> these
> >> >> >> >> RPCs
> >> >> >> >> > or get directly from MetadataImage. I’ve updated the KIP to
> >> make
> >> >> it
> >> >> >> >> clear.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR58. BumpLeaderEpochs: Why does it need to be issued to
> the
> >> >> >> broker,
> >> >> >> >> > instead of just to the controller?
> >> >> >> >> >
> >> >> >> >> > Yes, we only need to issue to the controller. Updated to the
> >> >> >> >> > BumpLeaderEpochs API.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR59. Start Mirror Topics:
> >> >> >> >> > > JR59.1 "During this state, the broker queries the source
> >> cluster
> >> >> >> for
> >> >> >> >> last
> >> >> >> >> > mirror epochs".
> >> >> >> >> > Could you describe which RPC is issued to find the LME?
> >> >> >> >> >
> >> >> >> >> > We’re using describeClusterMirror API via admin client here.
> >> >> >> >> >
> >> >> >> >> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
> >> >> request
> >> >> >> >> > doesn't
> >> >> >> >> > contain a replication factor.
> >> >> >> >> >
> >> >> >> >> > We use “default.replication.factor” when topic creation, so
> the
> >> >> >> >> > INVALID_REPLICATION_FACTOR will be thrown like normal topic
> >> >> creation.
> >> >> >> >> Also,
> >> >> >> >> > because we require the replicas for the partition assignment
> >> to be
> >> >> >> all
> >> >> >> >> > active and not in fenced or controlled shutdown(JR35), we’ll
> >> throw
> >> >> >> >> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already
> >> used
> >> >> by a
> >> >> >> >> > different topic name"
> >> >> >> >> > Should we also return this error when the topic name is used
> >> as a
> >> >> >> >> regular
> >> >> >> >> > topic?
> >> >> >> >> >
> >> >> >> >> > There are 2 errors we will throw when topic id <-> topic name
> >> >> >> mismatch.
> >> >> >> >> > Suppose provided topic id: ID_A, topic name: NAME_A
> >> >> >> >> > 1. Destination cluster exists topic id: ID_A, but topic name
> >> is:
> >> >> >> NAME_B
> >> >> >> >> > TOPIC_ALREADY_EXISTS will be thrown
> >> >> >> >> > 2. Destination cluster exists topic id: ID_B, but topic name
> >> is:
> >> >> >> NAME_A
> >> >> >> >> > INCONSISTENT_TOPIC_ID will be thrown
> >> >> >> >> >
> >> >> >> >> > Hmm.. indeed, it looks like using the same
> TOPIC_ALREADY_EXISTS
> >> >> error
> >> >> >> >> for
> >> >> >> >> > these 2 cases makes more sense. KIP updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR60. CreateClusterMirrorRequest: When will it receive an
> >> >> >> >> > INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> >> >> >> > CLUSTER_MIRROR_ALREADY_EXISTS?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Sorry, createClusterMirrorRequest doesn’t throw
> >> >> >> >> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to
> validate
> >> >> the
> >> >> >> >> mirror
> >> >> >> >> > states when creating a cluster mirror. We only care if the
> >> “mirror
> >> >> >> name”
> >> >> >> >> > exists or not. KIP updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should
> we
> >> >> just
> >> >> >> >> remove
> >> >> >> >> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> >> >> >> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to
> >> describe
> >> >> >> the
> >> >> >> >> > specific state causing the error.
> >> >> >> >> >
> >> >> >> >> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the
> following
> >> >> >> errors:
> >> >> >> >> > a. MIRROR_TOPIC_ALREADY_PAUSED
> >> >> >> >> > b. MIRROR_TOPIC_NOT_PAUSED,
> >> >> >> >> > c. MIRROR_TOPIC_ALREADY_STOPPED,
> >> >> >> >> > d. CLUSTER_MIRROR_NOT_EMPTY
> >> >> >> >> > e. MIRROR_TOPIC_NOT_STOPPED
> >> >> >> >> >
> >> >> >> >> > KIP is updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in
> >> some
> >> >> >> >> requests
> >> >> >> >> > (e.g. StopMirrorTopics), but not others (e.g.
> >> StartMirrorTopics).
> >> >> >> Could
> >> >> >> >> we
> >> >> >> >> > make them consistent?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Good catch. The updated errors for the new added RPC are
> this:
> >> >> >> >> >
> >> >> >> >> > CreateClusterMirror: NOT_CONTROLLER
> >> >> >> >> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> >> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> >> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> >> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> >> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR63. StopMirrorTopics:
> >> >> >> >> > > JR63.1 "The broker validates that all target topic
> partitions
> >> >> are
> >> >> >> in
> >> >> >> >> > either
> >> >> >> >> > LOG_TRUNCATION or MIRRORING or FAILED state."
> >> >> >> >> > Is this accurate? LOG_TRUNCATION can't transition to STOP.
> >> PAUSED
> >> >> >> can.
> >> >> >> >> >
> >> >> >> >> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP
> is
> >> >> >> updated.
> >> >> >> >> >
> >> >> >> >> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED
> error
> >> >> since
> >> >> >> >> PAUSED
> >> >> >> >> > -> STOPPED is valid?
> >> >> >> >> >
> >> >> >> >> > Yes, it is not correct. KIP updated to remove this error.
> >> >> >> >> >
> >> >> >> >> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not
> in
> >> >> >> stopped
> >> >> >> >> > state"
> >> >> >> >> > This is weird. The reason to issue StopMirrorTopics is that
> >> it's
> >> >> not
> >> >> >> in
> >> >> >> >> > stopped state.
> >> >> >> >> >
> >> >> >> >> > Indeed. KIP is updated to only include
> >> >> INVALID_CLUSTER_MIRROR_STATES
> >> >> >> >> > error, and will be thrown when states are not in MIRRORING or
> >> >> PAUSED
> >> >> >> >> state.
> >> >> >> >> >
> >> >> >> >> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since
> >> it's
> >> >> >> >> derived
> >> >> >> >> > from SourceOffset and DestinationOffset.
> >> >> >> >> >
> >> >> >> >> > True. Removed.
> >> >> >> >> >
> >> >> >> >> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It
> >> seems
> >> >> >> that we
> >> >> >> >> > could just set the partition level state to STOPPED.
> >> >> >> >> >
> >> >> >> >> > Nice catch. We don’t use it at all. That is some old leftover
> >> >> field
> >> >> >> we
> >> >> >> >> > forgot to remove. KIP updated.
> >> >> >> >> >
> >> >> >> >> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need
> >> to be
> >> >> >> sent
> >> >> >> >> to
> >> >> >> >> > the controller?
> >> >> >> >> >
> >> >> >> >> > No, these 2 only send to the broker. KIP updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and
> >> >> >> StateEpoch?
> >> >> >> >> It
> >> >> >> >> > seems both are used for fencing and fencing based on the
> >> latter is
> >> >> >> >> > stronger.
> >> >> >> >> >
> >> >> >> >> > Yes, basically the StateEpoch can fence for most cases. The
> >> >> >> LeaderEpoch
> >> >> >> >> > fencing is used for cases like:
> >> >> >> >> >
> >> >> >> >> > 1. b1 is the current leader for foo-0, preparing write state
> >> from
> >> >> >> >> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
> >> >> >> >> > 2. leadership change to b2 for foo-0, LE changes to 2.
> >> >> >> >> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch:
> 1
> >> }.
> >> >> >> >> > 4. The coordinator receives the read from b2 in step (3), the
> >> >> highest
> >> >> >> >> > lastSeen LE for foo-0 updates to 2 in coordinator.
> >> >> >> >> > 5. The writes state from b1 in step (1) arrives the
> >> coordinator,
> >> >> but
> >> >> >> it
> >> >> >> >> > rejects b1 writes because the highest lastSeen LE is 2, which
> >> is
> >> >> > 1
> >> >> >> in
> >> >> >> >> the
> >> >> >> >> > request.
> >> >> >> >> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2,
> >> >> stateEpoch: 1
> >> >> >> and
> >> >> >> >> > succeeds.
> >> >> >> >> >
> >> >> >> >> > The leaderEpoch fence is to avoid any stale broker writes
> after
> >> >> the
> >> >> >> new
> >> >> >> >> > leader epoch read. This cannot be protected by the
> stateEpoch.
> >> The
> >> >> >> >> > stateEpoch can prevent the write with older stateEpoch. So in
> >> the
> >> >> >> >> situation
> >> >> >> >> > above, the stateEpoch cannot determine if we should accept b1
> >> >> writes
> >> >> >> or
> >> >> >> >> > not, but leaderEpoch can. By the way, we borrowed this
> concept
> >> >> from
> >> >> >> >> > KIP-932:
> >> >> >> >> >
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
> >> <
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rUm8b3EcsLLsbBwkWG6hAiL-_Kard9D9TP_fPHp1g519R0HO7gRzlR4X5v2m4r58FdqsJdyCTqKHu14$
> >
> >> >> <
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!t3M02ge9e-mRqsshxasG_K0qLi9sX21suoBd69nLUl_gWUvn9ZLLekJiQspa_q3gTVSzWyUQBQfQLHk$
> >> >
> >> >> >> <
> >> >>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rX3-NwDU3s0U7POQYH1R-DvASnCPTTtFRWxZTVaS-PQHg-C1TAcu9xMe3MQF4fIXGO67WKOqrlz-lc8$
> >> >> >
> >> >> >> >> <
> >> >> >>
> >> >>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!u8EkPGXm8_VKdltXy8lrIW14mR2kgYIoaBPD7LO2uAN6rjhrfCX37Pfa6r85QPpSwSPh4V6eW_bKlns$
> >> >> >> >
> >> >> >> >> > <
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR68. Could we make the usage of topicId and topicName in
> the
> >> >> RPCs
> >> >> >> >> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> >> >> >> > BumpLeaderEpochsReponse uses TopicName.
> >> >> >> >> >
> >> >> >> >> > Yes, we have made:
> >> >> >> >> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use
> >> TopicName.
> >> >> >> >> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse
> >> use
> >> >> >> >> > TopicName.
> >> >> >> >> > 3. MirrorTopicStateChangeRecord uses TopicName because we
> will
> >> do
> >> >> >> >> > validation before writing the metadata log.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR69. MIRROR_PID_RESET:
> >> >> >> >> > > JR69.1 Could you specify the value of the control record's
> >> type?
> >> >> >> >> >
> >> >> >> >> > It’s MIRROR_PID_RESET(7). KIP updated.
> >> >> >> >> >
> >> >> >> >> > > JR69.2 How is the SourceClusterId field used?
> >> >> >> >> >
> >> >> >> >> > I’m also surprised to see this field in the
> >> MirrorPidResetRecord.
> >> >> I
> >> >> >> >> traced
> >> >> >> >> > back to the internal PR we added for this field, and
> confirmed
> >> it
> >> >> is
> >> >> >> not
> >> >> >> >> > used since it was added. Sorry for the confusion and nice
> >> catch!
> >> >> KIP
> >> >> >> >> > updated.
> >> >> >> >> >
> >> >> >> >> > > JR70. MirrorPartitionState: type should be "data" not
> >> >> >> >> "coordinator-key",
> >> >> >> >> > right? Ditto for LastMirrorEpochs.
> >> >> >> >> > We store both MirrorPartitionState and LastMirrorEpochs
> records
> >> >> into
> >> >> >> >> > internal __mirror_state topic, and rely on Coordinator
> Runtime
> >> to
> >> >> >> >> > write/read the records, like how `OffsetCommit` and
> >> >> `GroupMetadata`
> >> >> >> >> work.
> >> >> >> >> > So I think we should keep them as
> >> >> >> "coordinator-key"/"coordinator-value"
> >> >> >> >> > type.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Thank you,
> >> >> >> >> > Luke
> >> >> >> >> >
> >> >> >> >> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <
> >> >> >> dev@kafka.apache.org>
> >> >> >> >> > wrote:
> >> >> >> >> >
> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >>
> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >>
> >> >> >> >> >> JR51. "ULE_LOG_TRUNCATION When
> >> >> >> mirror.support.unclean.leader.election
> >> >> >> >> is
> >> >> >> >> >> enabled and the destination leader has log truncation, the
> >> state
> >> >> >> will
> >> >> >> >> move
> >> >> >> >> >> from MIRRORING to this state."
> >> >> >> >> >> This still suggests that divergeEpoch is only handled if
> >> >> >> >> >> mirror.support.unclean.leader.election
> >> >> >> >> >> is enabled.
> >> >> >> >> >>
> >> >> >> >> >> JR55. Should the mirror state and the mirror lags always be
> >> >> exposed
> >> >> >> >> >> together? If so, we can remove IncludeMirrorState and
> >> >> >> IncludeMirrorLags
> >> >> >> >> >> from DescribeClusterMirrorsRequest. If not, it's better to
> use
> >> >> two
> >> >> >> >> >> separate
> >> >> >> >> >> RPCs.
> >> >> >> >> >>
> >> >> >> >> >> DescribeClusterMirrorsRequest should be used in the Describe
> >> >> Mirror
> >> >> >> >> flow,
> >> >> >> >> >> right? Currently it's missing. Also, I assume the admin
> client
> >> >> >> refers
> >> >> >> >> to
> >> >> >> >> >> the one in the kafka-cluster-mirrors tool. But it seems that
> >> >> >> sometimes
> >> >> >> >> it
> >> >> >> >> >> also refers to the admin client on the broker? Perhaps we
> can
> >> use
> >> >> >> 'the
> >> >> >> >> >> tool' instead to make it clear.
> >> >> >> >> >>
> >> >> >> >> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the
> >> >> broker,
> >> >> >> >> >> instead of just to the controller?
> >> >> >> >> >>
> >> >> >> >> >> JR59. Start Mirror Topics:
> >> >> >> >> >> JR59.1 "During this state, the broker queries the source
> >> cluster
> >> >> for
> >> >> >> >> last
> >> >> >> >> >> mirror epochs".
> >> >> >> >> >> Could you describe which RPC is issued to find the LME?
> >> >> >> >> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
> >> >> request
> >> >> >> >> >> doesn't
> >> >> >> >> >> contain a replication factor.
> >> >> >> >> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already
> used
> >> >> by a
> >> >> >> >> >> different topic name"
> >> >> >> >> >> Should we also return this error when the topic name is used
> >> as a
> >> >> >> >> regular
> >> >> >> >> >> topic?
> >> >> >> >> >>
> >> >> >> >> >> JR60. CreateClusterMirrorRequest: When will it receive an
> >> >> >> >> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> >> >> >> >> CLUSTER_MIRROR_ALREADY_EXISTS?
> >> >> >> >> >>
> >> >> >> >> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we
> >> just
> >> >> >> >> remove
> >> >> >> >> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> >> >> >> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to
> >> >> describe
> >> >> >> the
> >> >> >> >> >> specific state causing the error.
> >> >> >> >> >>
> >> >> >> >> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in
> >> some
> >> >> >> >> requests
> >> >> >> >> >> (e.g. StopMirrorTopics), but not others (e.g.
> >> StartMirrorTopics).
> >> >> >> >> Could we
> >> >> >> >> >> make them consistent?
> >> >> >> >> >>
> >> >> >> >> >> JR63. StopMirrorTopics:
> >> >> >> >> >> JR63.1 "The broker validates that all target topic
> partitions
> >> >> are in
> >> >> >> >> >> either
> >> >> >> >> >> LOG_TRUNCATION or MIRRORING or FAILED state."
> >> >> >> >> >> Is this accurate? LOG_TRUNCATION can't transition to STOP.
> >> PAUSED
> >> >> >> can.
> >> >> >> >> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error
> >> >> since
> >> >> >> >> PAUSED
> >> >> >> >> >> -> STOPPED is valid?
> >> >> >> >> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
> >> >> stopped
> >> >> >> >> >> state"
> >> >> >> >> >> This is weird. The reason to issue StopMirrorTopics is that
> >> it's
> >> >> >> not in
> >> >> >> >> >> stopped state.
> >> >> >> >> >>
> >> >> >> >> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since
> >> it's
> >> >> >> derived
> >> >> >> >> >> from SourceOffset and DestinationOffset.
> >> >> >> >> >>
> >> >> >> >> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It
> seems
> >> >> that
> >> >> >> we
> >> >> >> >> >> could just set the partition level state to STOPPED.
> >> >> >> >> >>
> >> >> >> >> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need
> to
> >> be
> >> >> >> sent
> >> >> >> >> to
> >> >> >> >> >> the controller?
> >> >> >> >> >>
> >> >> >> >> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and
> >> >> >> StateEpoch? It
> >> >> >> >> >> seems both are used for fencing and fencing based on the
> >> latter
> >> >> is
> >> >> >> >> >> stronger.
> >> >> >> >> >>
> >> >> >> >> >> JR68. Could we make the usage of topicId and topicName in
> the
> >> >> RPCs
> >> >> >> >> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> >> >> >> >> BumpLeaderEpochsReponse uses TopicName.
> >> >> >> >> >>
> >> >> >> >> >> JR69. MIRROR_PID_RESET:
> >> >> >> >> >> JR69.1 Could you specify the value of the control record's
> >> type?
> >> >> >> >> >> JR69.2 How is the SourceClusterId field used?
> >> >> >> >> >>
> >> >> >> >> >> JR70. MirrorPartitionState: type should be "data" not
> >> >> >> >> "coordinator-key",
> >> >> >> >> >> right? Ditto for LastMirrorEpochs.
> >> >> >> >> >>
> >> >> >> >> >> Jun
> >> >> >> >> >>
> >> >> >> >> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <
> showuon@gmail.com>
> >> >> >> wrote:
> >> >> >> >> >>
> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >
> >> >> >> >> >> > Thanks for the comments.
> >> >> >> >> >> > Answering them below.
> >> >> >> >> >> >
> >> >> >> >> >> > > JR36. Sounds good. Should we remove
> >> >> >> >> >> MirrorLeaderEpochExceededException?
> >> >> >> >> >> >
> >> >> >> >> >> > Sorry, it’s a confusing name. The
> >> >> >> MirrorLeaderEpochExceededException
> >> >> >> >> is
> >> >> >> >> >> > used and thrown when the fetched mirror batch leader
> epoch >
> >> >> local
> >> >> >> >> >> leader
> >> >> >> >> >> > epoch in destination cluster. And then we’ll stop
> mirroring
> >> and
> >> >> >> move
> >> >> >> >> the
> >> >> >> >> >> > partition to EPOCH_FENSING state to wait for local leader
> >> epoch
> >> >> >> >> bump. I
> >> >> >> >> >> > think since we don’t have the concept of
> `MirrorLeaderEpoch`
> >> >> >> anymore,
> >> >> >> >> >> the
> >> >> >> >> >> > `MirrorLeaderEpochExceededException` should not cause
> >> confusion
> >> >> >> now.
> >> >> >> >> >> I’ll
> >> >> >> >> >> > keep the name for the exception.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR51. Sounds good. Do we plan to always set
> >> lastFetchedEpoch
> >> >> in
> >> >> >> the
> >> >> >> >> >> > mirror
> >> >> >> >> >> > fetch request issued to the source cluster, independent of
> >> the
> >> >> >> value
> >> >> >> >> of
> >> >> >> >> >> > mirror.support.unclean.leader.election? It's needed to
> >> detect
> >> >> >> >> divergence
> >> >> >> >> >> > during failover. However, if it's also set when
> >> >> >> >> >> > mirror.support.unclean.leader.election=false,
> >> >> >> >> >> > an unclean leader election in the source cluster can halt
> >> the
> >> >> >> >> mirroring
> >> >> >> >> >> > since we don’t do anything with the returned divergeEpoch
> in
> >> >> that
> >> >> >> >> mode
> >> >> >> >> >> > and continue
> >> >> >> >> >> > the next fetch.
> >> >> >> >> >> >
> >> >> >> >> >> > > Do we plan to always set lastFetchedEpoch in the mirror
> >> >> >> >> >> > fetch request issued to the source cluster, independent of
> >> the
> >> >> >> value
> >> >> >> >> of
> >> >> >> >> >> > Mirror.support.unclean.leader.election?
> >> >> >> >> >> >
> >> >> >> >> >> > Yes, we always set lastFetchedEpoch in the mirror fetch
> >> >> request.
> >> >> >> >> >> >
> >> >> >> >> >> > > However, if it's also set when
> >> >> >> >> >> > mirror.support.unclean.leader.election=false,
> >> >> >> >> >> > an unclean leader election in the source cluster can halt
> >> the
> >> >> >> >> mirroring
> >> >> >> >> >> > since we don’t do anything with the returned divergeEpoch
> in
> >> >> that
> >> >> >> >> mode
> >> >> >> >> >> > and continue the next fetch.
> >> >> >> >> >> >
> >> >> >> >> >> > Sorry I didn’t make it clear. When
> >> >> >> >> >> > “mirror.support.unclean.leader.election=false” and a
> >> >> >> “divergeEpoch”
> >> >> >> >> is
> >> >> >> >> >> set
> >> >> >> >> >> > in fetch response, the leader node will handle the
> >> divergeEpoch
> >> >> >> and
> >> >> >> >> then
> >> >> >> >> >> > continue the next fetch as usual. The difference with
> >> >> >> >> >> > “mirror.support.unclean.leader.election=true” is that
> >> “should
> >> >> we
> >> >> >> wait
> >> >> >> >> >> for
> >> >> >> >> >> > ALL replicas caught up (complete truncation)”. Without the
> >> >> >> waiting,
> >> >> >> >> it’s
> >> >> >> >> >> > possible the inconsistent log issue in KAFKA-18723 could
> >> happen
> >> >> >> >> between
> >> >> >> >> >> > leader and follower nodes in the destination cluster
> >> because we
> >> >> >> don’t
> >> >> >> >> >> have
> >> >> >> >> >> > “MirrorLeaderEpoch” concept now to reject higher leader
> >> epoch
> >> >> >> >> batches.
> >> >> >> >> >> That
> >> >> >> >> >> > is, we won't halt the mirroring if "divergeEpoch" is set
> but
> >> >> >> >> >> > “mirror.support.unclean.leader.election=false”.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR55. The DescribeClusterMirrorsRequest still returns
> both
> >> >> the
> >> >> >> >> mirror
> >> >> >> >> >> > partition state and the mirror partition leader state. The
> >> >> >> problem is
> >> >> >> >> >> that
> >> >> >> >> >> > if a caller only needs one of the state, the other state
> >> >> becomes
> >> >> >> >> >> redundant.
> >> >> >> >> >> > For example, when starting the mirror, to obtain LME, we
> >> only
> >> >> need
> >> >> >> >> the
> >> >> >> >> >> > mirror partition state. Here is an alternative design: We
> >> >> expose
> >> >> >> the
> >> >> >> >> >> mirror
> >> >> >> >> >> > partition state through ReadMirrorState. The client will
> use
> >> >> the
> >> >> >> >> >> > FindCoordinatorRequest
> >> >> >> >> >> > to find the right mirror partition coordinator to issue
> that
> >> >> >> >> request. We
> >> >> >> >> >> > expose the mirror partition leader state through the
> >> >> >> >> >> ReadClusterMirrorLags
> >> >> >> >> >> > request. The client can use the metadata request to find
> the
> >> >> >> >> partition
> >> >> >> >> >> > leader to issue that request.
> >> >> >> >> >> >
> >> >> >> >> >> > > In the describing mirror flow, the admin client first
> >> issues
> >> >> the
> >> >> >> >> >> > ListMirrorRequest to find all the mirrors and the topics
> >> being
> >> >> >> >> >> mirrored. It
> >> >> >> >> >> > then issues the metadata request to find the partitions
> for
> >> >> those
> >> >> >> >> topics
> >> >> >> >> >> > and issues a FindCoordinatorRequest to find the mirror
> >> >> partition
> >> >> >> >> state
> >> >> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the
> >> mirror
> >> >> >> >> partition
> >> >> >> >> >> > state coordinator and ReadClusterMirrorLags to the
> partition
> >> >> >> leader.
> >> >> >> >> >> >
> >> >> >> >> >> > > In the starting mirror flow, the partition leader in the
> >> >> >> >> destination
> >> >> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name
> >> >> based
> >> >> >> on
> >> >> >> >> its
> >> >> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to
> find
> >> the
> >> >> >> >> mirror
> >> >> >> >> >> > partition state coordinators. Finally, it issues
> >> >> ReadMirrorState
> >> >> >> to
> >> >> >> >> the
> >> >> >> >> >> > mirror partition state coordinator to get the LME.
> >> >> >> >> >> >
> >> >> >> >> >> > > We can also improve the ListMirror request to accept a
> >> filter
> >> >> >> >> based on
> >> >> >> >> >> > mirror name, source cluster ID and topic name.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Answers:
> >> >> >> >> >> >
> >> >> >> >> >> > > The DescribeClusterMirrorsRequest still returns both the
> >> >> mirror
> >> >> >> >> >> > partition state and the mirror partition leader state. The
> >> >> >> problem is
> >> >> >> >> >> that
> >> >> >> >> >> > if a caller only needs one of the state, the other state
> >> >> becomes
> >> >> >> >> >> redundant.
> >> >> >> >> >> > For example, when starting the mirror, to obtain LME, we
> >> only
> >> >> need
> >> >> >> >> the
> >> >> >> >> >> > mirror partition state.
> >> >> >> >> >> >
> >> >> >> >> >> > Good point!
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > Here is an alternative design: We expose the mirror
> >> >> >> >> >> > partition state through ReadMirrorState. The client will
> use
> >> >> the
> >> >> >> >> >> > FindCoordinatorRequest
> >> >> >> >> >> > to find the right mirror partition coordinator to issue
> that
> >> >> >> >> request. We
> >> >> >> >> >> > expose the mirror partition leader state through the
> >> >> >> >> >> ReadClusterMirrorLags
> >> >> >> >> >> > request. The client can use the metadata request to find
> the
> >> >> >> >> partition
> >> >> >> >> >> > leader to issue that request.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Sounds good. But we try not to expose
> >> >> >> >> >> > ReadMirrorState/ReadClusterMirrorLags directly to the
> admin
> >> >> client
> >> >> >> >> >> because
> >> >> >> >> >> > users might get confused about these 2 requests, and they
> >> are
> >> >> also
> >> >> >> >> both
> >> >> >> >> >> > related to describeClusterMirror. I think we can add 2
> >> >> additional
> >> >> >> >> >> fields to
> >> >> >> >> >> > describeClusterMirrorRequest: “includeMirrorStates” and
> >> >> >> >> >> > “includeMirrorLags”. So the admin client can include
> either
> >> >> one or
> >> >> >> >> both
> >> >> >> >> >> of
> >> >> >> >> >> > them in the DescrbieClusterMirrorRequest based on the
> >> provided
> >> >> >> >> options.
> >> >> >> >> >> KIP
> >> >> >> >> >> > is updated.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > In the describing mirror flow, the admin client first
> >> issues
> >> >> the
> >> >> >> >> >> > ListMirrorRequest to find all the mirrors and the topics
> >> being
> >> >> >> >> >> mirrored. It
> >> >> >> >> >> > then issues the metadata request to find the partitions
> for
> >> >> those
> >> >> >> >> topics
> >> >> >> >> >> > and issues a FindCoordinatorRequest to find the mirror
> >> >> partition
> >> >> >> >> state
> >> >> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the
> >> mirror
> >> >> >> >> partition
> >> >> >> >> >> > state coordinator and ReadClusterMirrorLags to the
> partition
> >> >> >> leader.
> >> >> >> >> >> >
> >> >> >> >> >> > Makes sense. KIP is updated.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > In the starting mirror flow, the partition leader in the
> >> >> >> >> destination
> >> >> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name
> >> >> based
> >> >> >> on
> >> >> >> >> its
> >> >> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to
> find
> >> the
> >> >> >> >> mirror
> >> >> >> >> >> > partition state coordinators. Finally, it issues
> >> >> ReadMirrorState
> >> >> >> to
> >> >> >> >> the
> >> >> >> >> >> > mirror partition state coordinator to get the LME.
> >> >> >> >> >> >
> >> >> >> >> >> > Make sense. So we can send DescribeClusterMirrorRequest
> with
> >> >> >> >> >> > “includeMirrorStates” only.
> >> >> >> >> >> >
> >> >> >> >> >> > > We can also improve the ListMirror request to accept a
> >> filter
> >> >> >> >> based on
> >> >> >> >> >> > mirror name, source cluster ID and topic name.
> >> >> >> >> >> >
> >> >> >> >> >> > Good suggestion, just like what ListGroupsRequest does.
> >> >> >> >> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
> >> >> >> >> >> > "DesiredMirrorStateFilter" in the
> ListClusterMirrorsRequest.
> >> >> KIP
> >> >> >> >> >> updated.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR56. Could you list the ACL needed for each new RPC?
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Yes, it’s already listed in “Security Control” section.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR57. In the command workflow, it sometimes says "The
> user
> >> >> >> sends"
> >> >> >> >> and
> >> >> >> >> >> > some
> >> >> >> >> >> > other times says "The admin client sends". It's better to
> >> use
> >> >> "The
> >> >> >> >> admin
> >> >> >> >> >> > client" consistently in all those places.
> >> >> >> >> >> >
> >> >> >> >> >> > Agree. KIP updated.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Thank you,
> >> >> >> >> >> > Luke
> >> >> >> >> >> >
> >> >> >> >> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <
> >> >> >> dev@kafka.apache.org
> >> >> >> >> >
> >> >> >> >> >> > wrote:
> >> >> >> >> >> >
> >> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >> >>
> >> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR36. Sounds good. Should we remove
> >> >> >> >> MirrorLeaderEpochExceededException?
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR51. Sounds good. Do we plan to always set
> >> lastFetchedEpoch
> >> >> in
> >> >> >> the
> >> >> >> >> >> mirror
> >> >> >> >> >> >> fetch request issued to the source cluster, independent
> of
> >> the
> >> >> >> >> value of
> >> >> >> >> >> >> mirror.support.unclean.leader.election? It's needed to
> >> detect
> >> >> >> >> >> divergence
> >> >> >> >> >> >> during failover. However, if it's also set when
> >> >> >> >> >> >> mirror.support.unclean.leader.election=false,
> >> >> >> >> >> >> an unclean leader election in the source cluster can halt
> >> the
> >> >> >> >> mirroring
> >> >> >> >> >> >> since we don’t do anything with the returned divergeEpoch
> >> in
> >> >> that
> >> >> >> >> mode
> >> >> >> >> >> >> and continue
> >> >> >> >> >> >> the next fetch.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR55. The DescribeClusterMirrorsRequest still returns
> both
> >> the
> >> >> >> >> mirror
> >> >> >> >> >> >> partition state and the mirror partition leader state.
> The
> >> >> >> problem
> >> >> >> >> is
> >> >> >> >> >> that
> >> >> >> >> >> >> if a caller only needs one of the state, the other state
> >> >> becomes
> >> >> >> >> >> >> redundant.
> >> >> >> >> >> >> For example, when starting the mirror, to obtain LME, we
> >> only
> >> >> >> need
> >> >> >> >> the
> >> >> >> >> >> >> mirror partition state. Here is an alternative design: We
> >> >> expose
> >> >> >> the
> >> >> >> >> >> >> mirror
> >> >> >> >> >> >> partition state through ReadMirrorState. The client will
> >> use
> >> >> the
> >> >> >> >> >> >> FindCoordinatorRequest
> >> >> >> >> >> >> to find the right mirror partition coordinator to issue
> >> that
> >> >> >> >> request.
> >> >> >> >> >> We
> >> >> >> >> >> >> expose the mirror partition leader state through the
> >> >> >> >> >> ReadClusterMirrorLags
> >> >> >> >> >> >> request. The client can use the metadata request to find
> >> the
> >> >> >> >> partition
> >> >> >> >> >> >> leader to issue that request.
> >> >> >> >> >> >>
> >> >> >> >> >> >> In the describing mirror flow, the admin client first
> >> issues
> >> >> the
> >> >> >> >> >> >> ListMirrorRequest to find all the mirrors and the topics
> >> being
> >> >> >> >> >> mirrored.
> >> >> >> >> >> >> It
> >> >> >> >> >> >> then issues the metadata request to find the partitions
> for
> >> >> those
> >> >> >> >> >> topics
> >> >> >> >> >> >> and issues a FindCoordinatorRequest to find the mirror
> >> >> partition
> >> >> >> >> state
> >> >> >> >> >> >> coordinators. Finally, it issues ReadMirrorState to the
> >> mirror
> >> >> >> >> >> partition
> >> >> >> >> >> >> state coordinator and ReadClusterMirrorLags to the
> >> partition
> >> >> >> leader.
> >> >> >> >> >> >>
> >> >> >> >> >> >> In the starting mirror flow, the partition leader in the
> >> >> >> destination
> >> >> >> >> >> >> cluster issues a ListMirrorRequest to find the mirror
> name
> >> >> based
> >> >> >> on
> >> >> >> >> its
> >> >> >> >> >> >> cluster id. It then issues a FindCoordinatorRequest to
> find
> >> >> the
> >> >> >> >> mirror
> >> >> >> >> >> >> partition state coordinators. Finally, it issues
> >> >> ReadMirrorState
> >> >> >> to
> >> >> >> >> the
> >> >> >> >> >> >> mirror partition state coordinator to get the LME.
> >> >> >> >> >> >>
> >> >> >> >> >> >> We can also improve the ListMirror request to accept a
> >> filter
> >> >> >> based
> >> >> >> >> on
> >> >> >> >> >> >> mirror name, source cluster ID and topic name.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR56. Could you list the ACL needed for each new RPC?
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR57. In the command workflow, it sometimes says "The
> user
> >> >> sends"
> >> >> >> >> and
> >> >> >> >> >> some
> >> >> >> >> >> >> other times says "The admin client sends". It's better to
> >> use
> >> >> >> "The
> >> >> >> >> >> admin
> >> >> >> >> >> >> client" consistently in all those places.
> >> >> >> >> >> >>
> >> >> >> >> >> >> Jun
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <
> >> showuon@gmail.com>
> >> >> >> wrote:
> >> >> >> >> >> >>
> >> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Thanks for your comments.
> >> >> >> >> >> >> > Answering them below.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch”
> >> is
> >> >> >> >> >> specifically
> >> >> >> >> >> >> > designed for the unclean leader election support in
> >> Cluster
> >> >> >> >> Mirroring
> >> >> >> >> >> >> > because the unclean leader election will break the
> >> >> assumption
> >> >> >> that
> >> >> >> >> >> the
> >> >> >> >> >> >> > committed data won’t be truncated."
> >> >> >> >> >> >> > If the source cluster has an unclean leader election,
> the
> >> >> >> >> destination
> >> >> >> >> >> >> > cluster will detect the inconsistency and truncate the
> >> log
> >> >> at
> >> >> >> the
> >> >> >> >> >> >> > destination leader. We can just
> >> >> >> >> >> >> > extend mirror.support.unclean.leader.election to cover
> >> this
> >> >> >> case,
> >> >> >> >> >> i.e.,
> >> >> >> >> >> >> we
> >> >> >> >> >> >> > wait until all replicas' log end offsets are reduced to
> >> the
> >> >> >> target
> >> >> >> >> >> >> offset
> >> >> >> >> >> >> > before resuming mirroring. This avoids the
> inconsistency.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > If the destination cluster has an unclean leader
> >> election,
> >> >> it
> >> >> >> will
> >> >> >> >> >> >> > re-fetch the same data from the source. There won't be
> >> any
> >> >> >> >> >> inconsistency
> >> >> >> >> >> >> > either.
> >> >> >> >> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Yes, that works, too. I agree this is better because
> the
> >> >> >> unclean
> >> >> >> >> >> leader
> >> >> >> >> >> >> > election is rare, we don't want to occupy a field just
> >> for
> >> >> it.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > So to achieve this, we have to:
> >> >> >> >> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
> >> >> >> >> >> >> > 2. When the destination cluster leader completes a log
> >> >> >> truncation
> >> >> >> >> >> after
> >> >> >> >> >> >> a
> >> >> >> >> >> >> > fetch response due to divergeEpoch, we know it must be
> >> >> caused
> >> >> >> by
> >> >> >> >> >> unclean
> >> >> >> >> >> >> > leader election in the source cluster because we only
> >> fetch
> >> >> >> >> committed
> >> >> >> >> >> >> > records. If
> >> “mirror.support.unclean.leader.election=false”,
> >> >> we
> >> >> >> >> don’t
> >> >> >> >> >> do
> >> >> >> >> >> >> > anything and continue the next fetch. If
> >> >> >> >> >> >> > “mirror.support.unclean.leader.election=true”, we
> remove
> >> the
> >> >> >> >> mirror
> >> >> >> >> >> >> fetcher
> >> >> >> >> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
> >> >> >> >> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL
> >> replicas
> >> >> to
> >> >> >> be
> >> >> >> >> >> caught
> >> >> >> >> >> >> up
> >> >> >> >> >> >> > into ISR and reduced to the expected LEO.
> >> >> >> >> >> >> > 4. Move to “MIRRORING” state and create the mirror
> >> fetcher
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in
> >> Fetch
> >> >> >> API.
> >> >> >> >> >> >> > KIP is updated.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR51. Make sense. To truncate based on LME, we need
> to
> >> >> issue
> >> >> >> >> >> another
> >> >> >> >> >> >> RPC
> >> >> >> >> >> >> > to translate the LME to an offset. It would be useful
> to
> >> >> >> include
> >> >> >> >> >> that in
> >> >> >> >> >> >> > the workflow description.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > About the additional RPC to translate the LME to the
> >> offset
> >> >> >> (i.e.
> >> >> >> >> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on
> >> this
> >> >> >> to do
> >> >> >> >> >> the
> >> >> >> >> >> >> > offset translation. We use the local leader epoch cache
> >> in
> >> >> the
> >> >> >> >> >> >> destination
> >> >> >> >> >> >> > cluster to translate the epoch to save the RPC call.
> This
> >> >> works
> >> >> >> >> >> because
> >> >> >> >> >> >> of
> >> >> >> >> >> >> > an invariant the LME truncation protocol maintains: for
> >> any
> >> >> >> leader
> >> >> >> >> >> >> epoch ≤
> >> >> >> >> >> >> > LME, that epoch's data belongs to exactly one cluster,
> >> >> either
> >> >> >> it
> >> >> >> >> >> >> originated
> >> >> >> >> >> >> > there, or it was mirrored. This holds even across
> >> repeated
> >> >> >> mirror
> >> >> >> >> ->
> >> >> >> >> >> >> > reverse-mirror -> mirror cycles, since each direction's
> >> >> >> >> >> LOG_TRUNCATION
> >> >> >> >> >> >> > phase re-establishes the invariant before that cluster
> >> >> starts
> >> >> >> >> >> mirroring.
> >> >> >> >> >> >> > With this invariant, the replication protocol can find
> >> out
> >> >> the
> >> >> >> >> >> diverge
> >> >> >> >> >> >> > point and truncate the log correctly when mirroring.
> For
> >> >> >> example:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> >> >> > Offset 2: A (LE: 0)
> >> >> >> >> >> >> > Offset 3: A (LE: 1)
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Cluster A crashes while Cluster B mirrors to offset 1.
> >> >> Failover
> >> >> >> >> to B
> >> >> >> >> >> and
> >> >> >> >> >> >> > store the LME=0 in cluster B. When cluster A recovers
> and
> >> >> >> starts
> >> >> >> >> the
> >> >> >> >> >> >> > reverse mirroring, it’ll query the LME from B and get
> >> back
> >> >> >> LME: 0.
> >> >> >> >> >> Then
> >> >> >> >> >> >> > cluster A gets the end offset for epoch 0 from
> >> >> >> leaderEpochCache,
> >> >> >> >> and
> >> >> >> >> >> >> gets
> >> >> >> >> >> >> > offset 2. So offset 3 is truncated. The log is like
> this
> >> >> now:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res
> (LE:
> >> 10)
> >> >> >> >> >> >> > Offset 3: B
> >> (LE:
> >> >> >> 10)
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > When mirror starts, cluster A sends the fetch request
> >> with {
> >> >> >> >> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the
> >> >> >> divergingEpoch
> >> >> >> >> >> check
> >> >> >> >> >> >> in
> >> >> >> >> >> >> > the cluster B queries the endOffset for leader epoch 0,
> >> and
> >> >> >> found
> >> >> >> >> the
> >> >> >> >> >> >> end
> >> >> >> >> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so
> >> >> returns
> >> >> >> >> fetch
> >> >> >> >> >> >> > response with divergingEpoch info to cluster A for log
> >> >> >> >> truncation. In
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > end, offset 2 in cluster A will be truncated and then
> log
> >> >> >> >> converges
> >> >> >> >> >> with
> >> >> >> >> >> >> > cluster B.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a
> >> bit
> >> >> >> >> strange
> >> >> >> >> >> >> > because it mixes querying the mirror partition state
> with
> >> >> >> querying
> >> >> >> >> >> the
> >> >> >> >> >> >> > mirror partition leader. Another option is to use
> >> >> >> >> >> >> ReadMirrorStatesRequest
> >> >> >> >> >> >> > to query the mirror partition state and a new request
> to
> >> >> query
> >> >> >> the
> >> >> >> >> >> >> mirror
> >> >> >> >> >> >> > partition leader. If we want the adminClient to query
> the
> >> >> right
> >> >> >> >> >> >> ClusterMirrorCoordinator
> >> >> >> >> >> >> > directly, we can extend the FindCoordinatorRequest to
> >> >> support
> >> >> >> >> that.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Good suggestion. Yes, we need to fan out to all brokers
> >> >> >> because we
> >> >> >> >> >> have
> >> >> >> >> >> >> to
> >> >> >> >> >> >> > collect info from mirror partition leaders and mirror
> >> >> >> >> coordinators.
> >> >> >> >> >> If
> >> >> >> >> >> >> we
> >> >> >> >> >> >> > create a new API to contact mirror partition leaders,
> it
> >> >> will
> >> >> >> make
> >> >> >> >> >> the
> >> >> >> >> >> >> API
> >> >> >> >> >> >> > cleaner and clearer. So we will do this:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag
> fields
> >> >> from
> >> >> >> >> >> >> > DescribeClusterMirror request/response
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > b. Add a new API: ReadClusterMirrorLags API to request
> >> for
> >> >> >> >> >> SourceOffset,
> >> >> >> >> >> >> > DestinationOffset, and Lag info.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > The workflow for admin client describeClusterMirror
> will
> >> be
> >> >> >> like
> >> >> >> >> >> this:
> >> >> >> >> >> >> > 1. adminClient sends describeClusterMirror to the
> >> >> >> leastLoadedNode
> >> >> >> >> >> broker
> >> >> >> >> >> >> > 2. The received broker collects the current mirrored
> >> >> >> partitions,
> >> >> >> >> and
> >> >> >> >> >> >> > groups them by coordinator node and by partition leader
> >> >> node.
> >> >> >> >> >> >> > 3. The broker sends ReadMirrorState request to the
> >> >> coordinator
> >> >> >> >> nodes
> >> >> >> >> >> to
> >> >> >> >> >> >> > get the mirror state related info
> >> >> >> >> >> >> > 4. The broker sends ReadClusterMirrorLags request to
> the
> >> >> mirror
> >> >> >> >> >> >> partition
> >> >> >> >> >> >> > leader nodes to get the mirror lag, source offset,
> >> >> destination
> >> >> >> >> offset
> >> >> >> >> >> >> > (which updates on every fetch response).
> >> >> >> >> >> >> > 5. The broker merge all the response into
> >> >> describeClusterMirror
> >> >> >> >> >> response
> >> >> >> >> >> >> > and send to admin client
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > KIP is updated. Thanks for the suggestion.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR55. For __mirror_state topic, could we describe
> >> whether
> >> >> it
> >> >> >> is
> >> >> >> >> >> >> > compacted?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Yes, it is a compacted topic. I made it much clearer in
> >> the
> >> >> >> >> >> >> > “ClusterMirrorCoordinator” section.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Thank you,
> >> >> >> >> >> >> > Luke
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
> >> >> >> >> dev@kafka.apache.org
> >> >> >> >> >> >
> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch”
> is
> >> >> >> >> >> specifically
> >> >> >> >> >> >> >> designed for the unclean leader election support in
> >> Cluster
> >> >> >> >> >> Mirroring
> >> >> >> >> >> >> >> because the unclean leader election will break the
> >> >> assumption
> >> >> >> >> that
> >> >> >> >> >> the
> >> >> >> >> >> >> >> committed data won’t be truncated."
> >> >> >> >> >> >> >> If the source cluster has an unclean leader election,
> >> the
> >> >> >> >> >> destination
> >> >> >> >> >> >> >> cluster will detect the inconsistency and truncate the
> >> log
> >> >> at
> >> >> >> the
> >> >> >> >> >> >> >> destination leader. We can just
> >> >> >> >> >> >> >> extend mirror.support.unclean.leader.election to cover
> >> this
> >> >> >> case,
> >> >> >> >> >> >> i.e., we
> >> >> >> >> >> >> >> wait until all replicas' log end offsets are reduced
> to
> >> the
> >> >> >> >> target
> >> >> >> >> >> >> offset
> >> >> >> >> >> >> >> before resuming mirroring. This avoids the
> >> inconsistency.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> If the destination cluster has an unclean leader
> >> election,
> >> >> it
> >> >> >> >> will
> >> >> >> >> >> >> >> re-fetch
> >> >> >> >> >> >> >> the same data from the source. There won't be any
> >> >> >> inconsistency
> >> >> >> >> >> either.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR51. Make sense. To truncate based on LME, we need to
> >> >> issue
> >> >> >> >> another
> >> >> >> >> >> >> RPC
> >> >> >> >> >> >> >> to
> >> >> >> >> >> >> >> translate the LME to an offset. It would be useful to
> >> >> include
> >> >> >> >> that
> >> >> >> >> >> in
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> workflow description.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a
> >> bit
> >> >> >> strange
> >> >> >> >> >> >> because
> >> >> >> >> >> >> >> it mixes querying the mirror partition state with
> >> querying
> >> >> the
> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> partition leader. Another option is to use
> >> >> >> >> ReadMirrorStatesRequest
> >> >> >> >> >> to
> >> >> >> >> >> >> >> query
> >> >> >> >> >> >> >> the mirror partition state and a new request to query
> >> the
> >> >> >> mirror
> >> >> >> >> >> >> partition
> >> >> >> >> >> >> >> leader. If we want the adminClient to query the right
> >> >> >> >> >> >> >> ClusterMirrorCoordinator
> >> >> >> >> >> >> >> directly, we can extend the FindCoordinatorRequest to
> >> >> support
> >> >> >> >> that.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR55. For __mirror_state topic, could we describe
> >> whether
> >> >> it
> >> >> >> is
> >> >> >> >> >> >> compacted?
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> Jun
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <
> >> >> showuon@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Thanks for the comments.
> >> >> >> >> >> >> >> > Answering them below:
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR31.3 It makes sense for the new partition leader
> >> to
> >> >> >> query
> >> >> >> >> the
> >> >> >> >> >> >> >> current
> >> >> >> >> >> >> >> > mirror state from the local or remote
> >> >> MirrorMetadataManager
> >> >> >> >> that
> >> >> >> >> >> owns
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > partition's state. However, the wording in the KIP
> is
> >> >> >> >> confusing.
> >> >> >> >> >> >> >> > MirrorMetadataManager
> >> >> >> >> >> >> >> > mirror partition state cache should be populated
> only
> >> by
> >> >> one
> >> >> >> >> way,
> >> >> >> >> >> >> which
> >> >> >> >> >> >> >> is
> >> >> >> >> >> >> >> > replaying __mirror_state partitions this broker
> >> leads. By
> >> >> >> >> saying
> >> >> >> >> >> >> "and by
> >> >> >> >> >> >> >> > fetching state from remote coordinators via
> >> >> ReadMirrorStates
> >> >> >> >> >> RPCs.",
> >> >> >> >> >> >> >> it's
> >> >> >> >> >> >> >> > as if the mirror partition state cache on each
> broker
> >> >> stores
> >> >> >> >> the
> >> >> >> >> >> >> state
> >> >> >> >> >> >> >> for
> >> >> >> >> >> >> >> > all mirror partitions.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > You’re right. KIP is updated to make it clear.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR36. I understand the intention now, but I still
> >> don't
> >> >> >> >> think we
> >> >> >> >> >> >> need
> >> >> >> >> >> >> >> > MirrorLeaderEpoch. Let's
> >> >> >> >> >> >> >> > assume the common case: no unclean leader election
> in
> >> the
> >> >> >> >> source
> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> >> > during mirroring. Mirroring only fetches the
> committed
> >> >> data
> >> >> >> >> from
> >> >> >> >> >> the
> >> >> >> >> >> >> >> > source, which never changes if obtained a second
> >> time. In
> >> >> >> your
> >> >> >> >> >> >> example
> >> >> >> >> >> >> >> > image, follower broker 1 in the destination cluster
> >> can
> >> >> >> never
> >> >> >> >> >> obtain
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > record B(0) because it's never committed in the
> source
> >> >> and
> >> >> >> >> thus is
> >> >> >> >> >> >> never
> >> >> >> >> >> >> >> > exposed to the destination leader. It's possible for
> >> >> broker
> >> >> >> 1
> >> >> >> >> to
> >> >> >> >> >> >> first
> >> >> >> >> >> >> >> get
> >> >> >> >> >> >> >> > D(1) which is truncated later when broker 0 follows
> a
> >> new
> >> >> >> >> leader.
> >> >> >> >> >> But
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > new leader will refetch the same D(1) from the
> source
> >> >> >> cluster.
> >> >> >> >> So,
> >> >> >> >> >> >> it's
> >> >> >> >> >> >> >> ok
> >> >> >> >> >> >> >> > for broker 1 to keep D(1).
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Yes, you are correct. Because mirroring only fetches
> >> the
> >> >> >> >> committed
> >> >> >> >> >> >> data
> >> >> >> >> >> >> >> > from the source, the MirrorLeaderEpoch is
> unnecessary
> >> for
> >> >> >> >> normal
> >> >> >> >> >> >> >> mirroring.
> >> >> >> >> >> >> >> > Thanks for the clarification. So I think we can say,
> >> the
> >> >> >> >> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the
> >> >> unclean
> >> >> >> >> >> leader
> >> >> >> >> >> >> >> > election support in Cluster Mirroring because the
> >> unclean
> >> >> >> >> leader
> >> >> >> >> >> >> >> election
> >> >> >> >> >> >> >> > will break the assumption that the committed data
> >> won’t
> >> >> be
> >> >> >> >> >> truncated.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Given that, we could gate the MirrorLeaderEpoch
> field
> >> by
> >> >> >> only
> >> >> >> >> >> >> setting it
> >> >> >> >> >> >> >> > in the Fetch request when
> >> >> >> >> >> >> "mirror.support.unclean.leader.election=true"
> >> >> >> >> >> >> >> > (false by default), and skip it entirely for normal
> >> >> >> mirroring.
> >> >> >> >> >> What
> >> >> >> >> >> >> do
> >> >> >> >> >> >> >> you
> >> >> >> >> >> >> >> > think?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR51. Sounds good. Why do we choose to store the
> >> last
> >> >> >> mirror
> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> > instead
> >> >> >> >> >> >> >> > of a last mirror offset?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Yes, originally we stored the “last mirror offset”
> >> >> instead
> >> >> >> of
> >> >> >> >> >> “last
> >> >> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> > epoch”. But later, we found it doesn’t work for
> >> unclean
> >> >> >> leader
> >> >> >> >> >> >> election
> >> >> >> >> >> >> >> > cases. (yes, `unclean leader election` again :) )
> For
> >> >> >> example:
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE:
> 0)
> >> >> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE:
> 0)
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Cluster B is mirroring from A. And A crashes and
> >> >> failover to
> >> >> >> >> B. B
> >> >> >> >> >> >> stores
> >> >> >> >> >> >> >> > the “last mirror offset” as 1.
> >> >> >> >> >> >> >> > When cluster A recovers, somehow the unclean leader
> >> >> election
> >> >> >> >> >> happens,
> >> >> >> >> >> >> >> log
> >> >> >> >> >> >> >> > is empty in the new leader, and then some more
> >> leadership
> >> >> >> >> changes
> >> >> >> >> >> to
> >> >> >> >> >> >> >> make
> >> >> >> >> >> >> >> > the leader epoch become 10. Then it starts the
> reverse
> >> >> >> mirror
> >> >> >> >> >> from B.
> >> >> >> >> >> >> >> The
> >> >> >> >> >> >> >> > log looks like this:
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> >> >> >> >> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> >> >> >> >> >> >> >> > Offset
> 2:
> >> >> >> pid_res
> >> >> >> >> (LE:
> >> >> >> >> >> >> 10)
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > When A starts mirroring from B, log is truncated to
> >> last
> >> >> >> mirror
> >> >> >> >> >> >> offset:
> >> >> >> >> >> >> >> 1
> >> >> >> >> >> >> >> > (no-op), and then fetch from cluster B. The 1st
> fetch
> >> >> >> request
> >> >> >> >> is
> >> >> >> >> >> like
> >> >> >> >> >> >> >> this:
> >> >> >> >> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10,
> >> >> fetchOffset:
> >> >> >> >> 2, …
> >> >> >> >> >> }.
> >> >> >> >> >> >> The
> >> >> >> >> >> >> >> > divergingEpoch check in the cluster B queries the
> >> >> endOffset
> >> >> >> for
> >> >> >> >> >> >> leader
> >> >> >> >> >> >> >> > epoch 10 (in Partition#readRecords), and found the
> end
> >> >> >> offset
> >> >> >> >> for
> >> >> >> >> >> LE
> >> >> >> >> >> >> 10
> >> >> >> >> >> >> >> is
> >> >> >> >> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch
> >> >> >> validation
> >> >> >> >> >> >> passes
> >> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> > returns offset 2 to cluster A. In short, relying on
> >> “last
> >> >> >> >> mirror
> >> >> >> >> >> >> offset”
> >> >> >> >> >> >> >> > cannot resolve the log diverge issue in unclean
> leader
> >> >> >> election
> >> >> >> >> >> case.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > If instead, we truncate until “last mirror epoch”:
> 0,
> >> the
> >> >> >> >> offset 0
> >> >> >> >> >> >> and 1
> >> >> >> >> >> >> >> > in cluster A will be truncated as expected and then
> >> log
> >> >> >> >> converge
> >> >> >> >> >> with
> >> >> >> >> >> >> >> > cluster B correctly in the end.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Also, about JR53.2:
> >> >> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request
> to
> >> all
> >> >> >> >> brokers
> >> >> >> >> >> and
> >> >> >> >> >> >> >> > merges
> >> >> >> >> >> >> >> > the responses client-side."
> >> >> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
> >> >> >> partition.
> >> >> >> >> So
> >> >> >> >> >> >> sending
> >> >> >> >> >> >> >> > the request to every broker is wasteful. It's
> probably
> >> >> >> better
> >> >> >> >> to
> >> >> >> >> >> send
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > request to a single broker, which can then contact
> the
> >> >> right
> >> >> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > While working on this, I found we need to add one
> more
> >> >> >> tagged
> >> >> >> >> >> field
> >> >> >> >> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest
> >> schema
> >> >> >> because
> >> >> >> >> >> >> >> otherwise
> >> >> >> >> >> >> >> > the receiving broker doesn’t know if this is the
> >> request
> >> >> >> from
> >> >> >> >> >> client
> >> >> >> >> >> >> >> and it
> >> >> >> >> >> >> >> > needs to contact other brokers, or this is just a
> >> >> forwarded
> >> >> >> >> >> request.
> >> >> >> >> >> >> The
> >> >> >> >> >> >> >> > workflow is like this:
> >> >> >> >> >> >> >> > 1. the adminClient sends the
> >> >> DescribeClusterMirrorsRequest
> >> >> >> to a
> >> >> >> >> >> >> >> > leastLoaded broker, which doesn’t contain the
> >> "Forwarded"
> >> >> >> >> field.
> >> >> >> >> >> >> >> > 2. The broker receives the request and sends
> >> >> >> >> >> >> >> DescribeClusterMirrorsRequest
> >> >> >> >> >> >> >> > with "Forwarded=true" to the responsible brokers and
> >> >> waits
> >> >> >> for
> >> >> >> >> the
> >> >> >> >> >> >> >> response
> >> >> >> >> >> >> >> > async.
> >> >> >> >> >> >> >> > 3. When all responses are received, merge the
> results
> >> and
> >> >> >> >> respond
> >> >> >> >> >> to
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > client.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here
> >> because
> >> >> it
> >> >> >> >> >> doesn’t
> >> >> >> >> >> >> >> > contain the lag info in the coordinator.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > The alternative solution is the adminClient sends
> >> >> >> >> >> >> >> > DescribeClusterMirrorsRequest to the “responsible
> >> brokers
> >> >> >> only”
> >> >> >> >> >> (via
> >> >> >> >> >> >> >> > describeTopics), instead of ALL brokers, and merge
> the
> >> >> >> results
> >> >> >> >> in
> >> >> >> >> >> the
> >> >> >> >> >> >> >> > client side. This solution doesn’t need to add a new
> >> >> field
> >> >> >> in
> >> >> >> >> the
> >> >> >> >> >> >> >> > DescribeClusterMirrorsRequest and still can avoid
> >> >> >> unnecessary
> >> >> >> >> >> >> requests
> >> >> >> >> >> >> >> > sent.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > My preference is to keep the merge logic on the
> admin
> >> >> client
> >> >> >> >> side,
> >> >> >> >> >> >> which
> >> >> >> >> >> >> >> > saves the overhead in the broker side.
> >> >> >> >> >> >> >> > What do you think?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Thank you,
> >> >> >> >> >> >> >> > Luke
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
> >> >> >> >> >> dev@kafka.apache.org
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR31.3 It makes sense for the new partition leader
> to
> >> >> query
> >> >> >> >> the
> >> >> >> >> >> >> current
> >> >> >> >> >> >> >> >> mirror state from the local or remote
> >> >> MirrorMetadataManager
> >> >> >> >> that
> >> >> >> >> >> >> owns
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> partition's state. However, the wording in the KIP
> is
> >> >> >> >> confusing.
> >> >> >> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> >> >> mirror partition state cache should be populated
> >> only by
> >> >> >> one
> >> >> >> >> way,
> >> >> >> >> >> >> >> which is
> >> >> >> >> >> >> >> >> replaying __mirror_state partitions this broker
> >> leads.
> >> >> By
> >> >> >> >> saying
> >> >> >> >> >> >> "and
> >> >> >> >> >> >> >> by
> >> >> >> >> >> >> >> >> fetching state from remote coordinators via
> >> >> >> ReadMirrorStates
> >> >> >> >> >> RPCs.",
> >> >> >> >> >> >> >> it's
> >> >> >> >> >> >> >> >> as if the mirror partition state cache on each
> broker
> >> >> >> stores
> >> >> >> >> the
> >> >> >> >> >> >> state
> >> >> >> >> >> >> >> for
> >> >> >> >> >> >> >> >> all mirror partitions.
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR36. I understand the intention now, but I still
> >> don't
> >> >> >> think
> >> >> >> >> we
> >> >> >> >> >> >> need
> >> >> >> >> >> >> >> >> MirrorLeaderEpoch. Let's
> >> >> >> >> >> >> >> >> assume the common case: no unclean leader election
> in
> >> >> the
> >> >> >> >> source
> >> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> >> >> during mirroring. Mirroring only fetches the
> >> committed
> >> >> data
> >> >> >> >> from
> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> source, which never changes if obtained a second
> >> time.
> >> >> In
> >> >> >> your
> >> >> >> >> >> >> example
> >> >> >> >> >> >> >> >> image, follower broker 1 in the destination cluster
> >> can
> >> >> >> never
> >> >> >> >> >> obtain
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> record B(0) because it's never committed in the
> >> source
> >> >> and
> >> >> >> >> thus
> >> >> >> >> >> is
> >> >> >> >> >> >> >> never
> >> >> >> >> >> >> >> >> exposed to the destination leader. It's possible
> for
> >> >> >> broker 1
> >> >> >> >> to
> >> >> >> >> >> >> first
> >> >> >> >> >> >> >> get
> >> >> >> >> >> >> >> >> D(1) which is truncated later when broker 0
> follows a
> >> >> new
> >> >> >> >> leader.
> >> >> >> >> >> >> But
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> new leader will refetch the same D(1) from the
> source
> >> >> >> cluster.
> >> >> >> >> >> So,
> >> >> >> >> >> >> >> it's ok
> >> >> >> >> >> >> >> >> for broker 1 to keep D(1).
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR51. Sounds good. Why do we choose to store the
> last
> >> >> >> mirror
> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> instead
> >> >> >> >> >> >> >> >> of a last mirror offset?
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> Jun
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <
> >> >> >> showuon@gmail.com>
> >> >> >> >> >> wrote:
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Thanks for the great comments again.
> >> >> >> >> >> >> >> >> > Answering your question below.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror
> >> >> partition
> >> >> >> >> cache
> >> >> >> >> >> >> lives
> >> >> >> >> >> >> >> in
> >> >> >> >> >> >> >> >> > MirrorMetadataManager. It is populated in two
> >> ways: by
> >> >> >> >> >> replaying
> >> >> >> >> >> >> >> >> > __mirror_state partitions this broker leads, and
> by
> >> >> >> fetching
> >> >> >> >> >> state
> >> >> >> >> >> >> >> >> > from remote coordinators via ReadMirrorStates
> >> RPCs."
> >> >> >> >> >> >> >> >> > Hmm, a given broker only caches the mirror
> >> partition
> >> >> >> state
> >> >> >> >> for
> >> >> >> >> >> >> >> >> partitions
> >> >> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted
> on
> >> >> this
> >> >> >> >> broker,
> >> >> >> >> >> >> >> right?
> >> >> >> >> >> >> >> >> Why
> >> >> >> >> >> >> >> >> > does it need to fetch the state from other remote
> >> >> >> >> coordinators?
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > a given broker only caches the mirror partition
> >> >> state
> >> >> >> for
> >> >> >> >> >> >> >> partitions
> >> >> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted
> on
> >> >> this
> >> >> >> >> broker,
> >> >> >> >> >> >> >> right?
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Right.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > Why does it need to fetch the state from other
> >> >> remote
> >> >> >> >> >> >> coordinators?
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Because the data partition leader needs to know
> the
> >> >> >> current
> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> state
> >> >> >> >> >> >> >> >> > of the partitions it leads. On a leadership
> change,
> >> >> the
> >> >> >> >> >> metadata
> >> >> >> >> >> >> >> delta
> >> >> >> >> >> >> >> >> is
> >> >> >> >> >> >> >> >> > published to the MirrorMetadataManager, which
> >> queries
> >> >> the
> >> >> >> >> >> current
> >> >> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> >> > state for its led partitions from the coordinator
> >> >> (local
> >> >> >> or
> >> >> >> >> >> >> remote),
> >> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> > processes tasks based on that state. This is the
> >> >> reason
> >> >> >> why
> >> >> >> >> it
> >> >> >> >> >> >> >> fetches
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > state from other remote coordinator. This part of
> >> >> cache
> >> >> >> >> will be
> >> >> >> >> >> >> >> cleared
> >> >> >> >> >> >> >> >> > after the leadership change.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR36. I am still not quite sure why we need
> >> >> >> >> >> MirrorLeaderEpoch in
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > Fetch
> >> >> >> >> >> >> >> >> > request. In the current Fetch request, in
> addition
> >> to
> >> >> >> >> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> >> >> >> > it has another field LastFetchedEpoch, which
> >> captures
> >> >> the
> >> >> >> >> last
> >> >> >> >> >> >> leader
> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> > in the data and is used to resolve the
> >> inconsistency
> >> >> >> between
> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> follower
> >> >> >> >> >> >> >> >> > and the leader. Does it achieve the same purpose
> as
> >> >> >> >> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be
> used
> >> to
> >> >> do
> >> >> >> the
> >> >> >> >> >> >> >> validation
> >> >> >> >> >> >> >> >> to
> >> >> >> >> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any
> >> >> fetched
> >> >> >> >> >> batches
> >> >> >> >> >> >> with
> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> > higher than the LastFetchedEpoch, it means the
> >> >> follower
> >> >> >> can
> >> >> >> >> >> never
> >> >> >> >> >> >> >> accept
> >> >> >> >> >> >> >> >> > new appended batches with higher epoch in the
> >> leader
> >> >> >> node.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2
> >> >> >> purposes in
> >> >> >> >> >> the
> >> >> >> >> >> >> >> fetch
> >> >> >> >> >> >> >> >> > API:
> >> >> >> >> >> >> >> >> > 1. The receiver (leader node) uses it to validate
> >> it
> >> >> >> matches
> >> >> >> >> >> the
> >> >> >> >> >> >> >> leader
> >> >> >> >> >> >> >> >> > epoch in the leader node
> >> >> >> >> >> >> >> >> > 2. The sender (follower node) uses it to validate
> >> the
> >> >> >> epoch
> >> >> >> >> in
> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> fetched
> >> >> >> >> >> >> >> >> > batches are all <= this value.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still
> >> >> serving
> >> >> >> for
> >> >> >> >> >> >> >> followers
> >> >> >> >> >> >> >> >> in
> >> >> >> >> >> >> >> >> > the destination cluster. For purpose (2), my
> >> >> >> understanding
> >> >> >> >> of
> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > validation for KAFKA-18723 is like: the follower
> >> knows
> >> >> >> the
> >> >> >> >> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the
> >> leader
> >> >> node
> >> >> >> >> >> should
> >> >> >> >> >> >> >> >> contain in
> >> >> >> >> >> >> >> >> > the log batches when fetch request is sent. Any
> log
> >> >> batch
> >> >> >> >> >> beyond
> >> >> >> >> >> >> this
> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> > in the fetch response means the follower’s
> >> metadata is
> >> >> >> >> stale.
> >> >> >> >> >> The
> >> >> >> >> >> >> >> >> batches
> >> >> >> >> >> >> >> >> > need to be rejected and follower needs to update
> >> >> metadata
> >> >> >> >> for
> >> >> >> >> >> next
> >> >> >> >> >> >> >> >> fetch.
> >> >> >> >> >> >> >> >> > In the Cluster Mirroring’s world, the
> >> >> MirrorLeaderEpoch
> >> >> >> >> serves
> >> >> >> >> >> >> this
> >> >> >> >> >> >> >> >> purpose
> >> >> >> >> >> >> >> >> > to let the followers know the highest epoch in
> the
> >> >> leader
> >> >> >> >> node
> >> >> >> >> >> >> log in
> >> >> >> >> >> >> >> >> each
> >> >> >> >> >> >> >> >> > fetch response. Once the info is stale in the
> >> follower
> >> >> >> >> side, it
> >> >> >> >> >> >> >> updates
> >> >> >> >> >> >> >> >> it
> >> >> >> >> >> >> >> >> > from fetch response, and send next fetch request
> >> with
> >> >> >> >> correct
> >> >> >> >> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent
> >> data is
> >> >> >> >> possible
> >> >> >> >> >> >> like
> >> >> >> >> >> >> >> >> > KAFKA-18723.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > I tried not to add new field in Fetch API, but
> >> this is
> >> >> >> the
> >> >> >> >> best
> >> >> >> >> >> >> >> >> solution I
> >> >> >> >> >> >> >> >> > can think of.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR48. "The NumPartitions field is needed so the
> >> >> >> controller
> >> >> >> >> >> can
> >> >> >> >> >> >> >> create
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > destination topic if it does not already exist.
> >> When a
> >> >> >> >> topic is
> >> >> >> >> >> >> being
> >> >> >> >> >> >> >> >> > mirrored for the first time, the destination
> >> cluster
> >> >> has
> >> >> >> no
> >> >> >> >> >> >> knowledge
> >> >> >> >> >> >> >> >> > of the source topic's partition count, so the
> >> request
> >> >> >> must
> >> >> >> >> >> carry
> >> >> >> >> >> >> it."
> >> >> >> >> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is
> >> >> started,
> >> >> >> the
> >> >> >> >> >> source
> >> >> >> >> >> >> >> topic
> >> >> >> >> >> >> >> >> > may not exist yet. When the source topic is
> created
> >> >> >> later,
> >> >> >> >> we
> >> >> >> >> >> can
> >> >> >> >> >> >> >> start
> >> >> >> >> >> >> >> >> > mirroring without requiring the user to provide
> the
> >> >> >> number
> >> >> >> >> of
> >> >> >> >> >> >> >> >> partitions.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > When the source topic is created later, we can
> >> start
> >> >> >> >> >> mirroring
> >> >> >> >> >> >> >> without
> >> >> >> >> >> >> >> >> > requiring the user to provide the number of
> >> >> partitions.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > This is because this happens in periodical sync
> up
> >> in
> >> >> the
> >> >> >> >> >> >> destination
> >> >> >> >> >> >> >> >> > cluster, and the numPartitions and other topic
> >> info in
> >> >> >> the
> >> >> >> >> >> source
> >> >> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> >> >> > is already retrieved during the sync up process.
> >> >> Also, if
> >> >> >> >> users
> >> >> >> >> >> >> are
> >> >> >> >> >> >> >> >> > creating mirror topics via admin API, this info
> can
> >> >> also
> >> >> >> be
> >> >> >> >> >> >> retrieved
> >> >> >> >> >> >> >> >> > inside the admin client if not provided.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly
> >> need
> >> >> the
> >> >> >> >> last
> >> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> > for truncation when starting a mirror. When a
> >> mirror
> >> >> is
> >> >> >> >> >> stopped at
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > source, we already bump up the leader epoch and
> >> write
> >> >> >> some
> >> >> >> >> >> >> additional
> >> >> >> >> >> >> >> >> > records for cleanup. It seems that we can just
> use
> >> the
> >> >> >> >> >> >> DivergingEpoch
> >> >> >> >> >> >> >> >> > returned in the initial mirror Fetch request to
> >> detect
> >> >> >> the
> >> >> >> >> >> >> diverging
> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> > and use that for truncation at the mirror
> >> destination.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > That’s a good question. I thought about this
> before
> >> >> but
> >> >> >> >> >> >> >> unfortunately it
> >> >> >> >> >> >> >> >> > doesn’t work. The reason we need the last mirror
> >> epoch
> >> >> >> >> record
> >> >> >> >> >> is
> >> >> >> >> >> >> >> because
> >> >> >> >> >> >> >> >> > the destination cluster might not completely sync
> >> with
> >> >> >> the
> >> >> >> >> >> source
> >> >> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> >> >> > before STOPPED. And this un-mirrored data might
> >> >> confuse
> >> >> >> the
> >> >> >> >> >> >> >> replication
> >> >> >> >> >> >> >> >> > protocol when reverse mirror. For example,
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
> >> >> >> >> >> >> >> >> > foo-0 leader log foo-0 leader
> log
> >> >> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A
> >> (LE:
> >> >> 0)
> >> >> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A
> >> (LE:
> >> >> 0)
> >> >> >> >> >> >> >> >> > Offset 2: from A (LE: 10)
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > While cluster B mirrors until offset 1, the
> >> cluster A
> >> >> >> >> crashes
> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> failover
> >> >> >> >> >> >> >> >> > to cluster B. B bumps leader epoch to 10,
> appends a
> >> >> >> >> >> >> MIRROR_PID_RESET
> >> >> >> >> >> >> >> >> with
> >> >> >> >> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror
> >> from
> >> >> >> >> cluster B,
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> log
> >> >> >> >> >> >> >> >> > will be like this:
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B
> (source)
> >> >> >> >> >> >> >> >> > foo-0 leader log: foo-0 leader
> >> log:
> >> >> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A
> >> (LE:
> >> >> 0)
> >> >> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A
> >> (LE:
> >> >> 0)
> >> >> >> >> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET
> >> (LE:
> >> >> >> 10)
> >> >> >> >> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B
> (LE:
> >> >> 10)
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > As you can see, cluster A (new destination
> cluster)
> >> >> >> doesn’t
> >> >> >> >> >> know
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > offset 2 is not belonging to the cluster B, which
> >> >> causes
> >> >> >> >> >> >> inconsistent
> >> >> >> >> >> >> >> >> logs.
> >> >> >> >> >> >> >> >> > So like what we described in the KIP, the last
> >> mirror
> >> >> >> epoch
> >> >> >> >> is
> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > synchronization point between source and
> >> destination.
> >> >> Any
> >> >> >> >> data
> >> >> >> >> >> >> beyond
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > point is treated as unknown to the source cluster
> >> and
> >> >> >> >> should be
> >> >> >> >> >> >> >> >> truncated.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR52. Stop Mirror:
> >> >> >> >> >> >> >> >> > > JR52.1 "3. If patterns are provided, the
> >> controller
> >> >> >> >> removes
> >> >> >> >> >> >> >> matching
> >> >> >> >> >> >> >> >> > entries from mirror.topics.include or adds them
> to
> >> >> >> >> >> >> >> >> mirror.topics.exclude on
> >> >> >> >> >> >> >> >> > the CLUSTER_MIRROR resource."
> >> >> >> >> >> >> >> >> > Why is it 'or' and not 'and'?
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh
> cycles,
> >> >> >> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> >> >> > discovers new source topics matching the
> persisted
> >> >> >> >> >> include/exclude
> >> >> >> >> >> >> >> >> patterns
> >> >> >> >> >> >> >> >> > and repeats steps 3 through 7 for each."
> >> >> >> >> >> >> >> >> > The broker will send a create topic request to
> the
> >> >> >> >> controller.
> >> >> >> >> >> How
> >> >> >> >> >> >> >> does
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > controller associate it with the mirror name to
> >> >> create a
> >> >> >> >> >> >> >> >> > `MirrorTopicStateChangeRecord`?
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > No, we don’t send a create topic request to the
> >> >> >> controller
> >> >> >> >> >> here.
> >> >> >> >> >> >> >> >> Actually
> >> >> >> >> >> >> >> >> > we send a StartMirrorTopics request to the
> >> controller,
> >> >> >> which
> >> >> >> >> >> >> contains
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > mirror name and topic metadata like
> NumPartitions,
> >> >> >> topicId…
> >> >> >> >> >> etc.
> >> >> >> >> >> >> So
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > controller can create the non-existed topic as
> >> >> explained
> >> >> >> in
> >> >> >> >> >> JR48,
> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR53. Describe Mirrors:
> >> >> >> >> >> >> >> >> > > JR53.1 "1. The user sends a
> >> DescribeClusterMirrors
> >> >> >> request
> >> >> >> >> >> with
> >> >> >> >> >> >> >> >> optional
> >> >> >> >> >> >> >> >> > mirror names (empty means all mirrors)."
> >> >> >> >> >> >> >> >> > In metadata request, we use a null arrary to
> >> represent
> >> >> >> all
> >> >> >> >> >> >> topics. It
> >> >> >> >> >> >> >> >> would
> >> >> >> >> >> >> >> >> > be useful to be consistent here.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > OK, KIP updated.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the
> request
> >> to
> >> >> all
> >> >> >> >> >> brokers
> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> > merges
> >> >> >> >> >> >> >> >> > the responses client-side."
> >> >> >> >> >> >> >> >> > This is not ideal. Not every broker hosts a
> mirror
> >> >> >> >> partition.
> >> >> >> >> >> So
> >> >> >> >> >> >> >> sending
> >> >> >> >> >> >> >> >> > the request to every broker is wasteful. It's
> >> probably
> >> >> >> >> better
> >> >> >> >> >> to
> >> >> >> >> >> >> send
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > request to a single broker, which can then
> contact
> >> the
> >> >> >> right
> >> >> >> >> >> >> >> >> > MirrorFetcherManager and
> ClusterMirrorCoordinator.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Agree, KIP updated.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > > JR54.
> MirrorTopicStateChangeRecord.DesiredState:
> >> Are
> >> >> >> >> >> MIRRORING,
> >> >> >> >> >> >> >> >> PAUSED,
> >> >> >> >> >> >> >> >> > STOPPED the only validate states? The state table
> >> has
> >> >> >> other
> >> >> >> >> >> states
> >> >> >> >> >> >> >> too.
> >> >> >> >> >> >> >> >> > Ditto for MirrorPartitionState.
> >> >> >> >> >> >> >> >> > The DesiredState field in
> >> MirrorTopicStateChangeRecord
> >> >> >> is to
> >> >> >> >> >> >> define
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > target state the user wants. And the desired
> state
> >> >> users
> >> >> >> can
> >> >> >> >> >> >> assign
> >> >> >> >> >> >> >> is
> >> >> >> >> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a
> >> >> desired
> >> >> >> >> state
> >> >> >> >> >> as
> >> >> >> >> >> >> >> >> STOPPING
> >> >> >> >> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
> >> >> >> >> intermediate
> >> >> >> >> >> >> >> states.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > About MirrorPartitionState record, it’ll record
> any
> >> >> >> state in
> >> >> >> >> >> the
> >> >> >> >> >> >> >> state
> >> >> >> >> >> >> >> >> > table that the current partition is. This is the
> >> real
> >> >> >> state
> >> >> >> >> for
> >> >> >> >> >> >> each
> >> >> >> >> >> >> >> >> > partition. It’s like in the kubernetes world,
> when
> >> we
> >> >> >> change
> >> >> >> >> >> the
> >> >> >> >> >> >> >> replica
> >> >> >> >> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired
> >> >> >> state), it
> >> >> >> >> >> won’t
> >> >> >> >> >> >> >> >> increase
> >> >> >> >> >> >> >> >> > to 5 immediately. The current replica
> >> >> >> (MirrorPartitionState
> >> >> >> >> >> >> record)
> >> >> >> >> >> >> >> is
> >> >> >> >> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > Thank you,
> >> >> >> >> >> >> >> >> > Luke
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
> >> >> >> >> >> >> dev@kafka.apache.org
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >> Hi, Fede,
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> Thanks for the reply. A few more comments.
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror
> >> partition
> >> >> >> cache
> >> >> >> >> >> >> lives in
> >> >> >> >> >> >> >> >> >> MirrorMetadataManager. It is populated in two
> >> ways:
> >> >> by
> >> >> >> >> >> replaying
> >> >> >> >> >> >> >> >> >> __mirror_state partitions this broker leads, and
> >> by
> >> >> >> >> fetching
> >> >> >> >> >> >> state
> >> >> >> >> >> >> >> >> >> from remote coordinators via ReadMirrorStates
> >> RPCs."
> >> >> >> >> >> >> >> >> >> Hmm, a given broker only caches the mirror
> >> partition
> >> >> >> state
> >> >> >> >> for
> >> >> >> >> >> >> >> >> partitions
> >> >> >> >> >> >> >> >> >> hashed into the __mirror_state partition hosted
> on
> >> >> this
> >> >> >> >> >> broker,
> >> >> >> >> >> >> >> right?
> >> >> >> >> >> >> >> >> Why
> >> >> >> >> >> >> >> >> >> does it need to fetch the state from other
> remote
> >> >> >> >> >> coordinators?
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> JR36. I am still not quite sure why we need
> >> >> >> >> MirrorLeaderEpoch
> >> >> >> >> >> in
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> Fetch
> >> >> >> >> >> >> >> >> >> request. In the current Fetch request, in
> >> addition to
> >> >> >> >> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> >> >> >> >> it has another field LastFetchedEpoch, which
> >> captures
> >> >> >> the
> >> >> >> >> last
> >> >> >> >> >> >> >> leader
> >> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> >> in the data and is used to resolve the
> >> inconsistency
> >> >> >> >> between
> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> follower
> >> >> >> >> >> >> >> >> >> and the leader. Does it achieve the same purpose
> >> as
> >> >> >> >> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> JR48. "The NumPartitions field is needed so the
> >> >> >> controller
> >> >> >> >> can
> >> >> >> >> >> >> >> create
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> >> destination topic if it does not already exist.
> >> When
> >> >> a
> >> >> >> >> topic
> >> >> >> >> >> is
> >> >> >> >> >> >> >> being
> >> >> >> >> >> >> >> >> >> mirrored for the first time, the destination
> >> cluster
> >> >> >> has no
> >> >> >> >> >> >> >> knowledge
> >> >> >> >> >> >> >> >> >> of the source topic's partition count, so the
> >> request
> >> >> >> must
> >> >> >> >> >> carry
> >> >> >> >> >> >> >> it."
> >> >> >> >> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is
> >> >> started,
> >> >> >> the
> >> >> >> >> >> >> source
> >> >> >> >> >> >> >> >> topic
> >> >> >> >> >> >> >> >> >> may not exist yet. When the source topic is
> >> created
> >> >> >> later,
> >> >> >> >> we
> >> >> >> >> >> can
> >> >> >> >> >> >> >> start
> >> >> >> >> >> >> >> >> >> mirroring without requiring the user to provide
> >> the
> >> >> >> number
> >> >> >> >> of
> >> >> >> >> >> >> >> >> partitions.
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly
> >> need
> >> >> the
> >> >> >> >> last
> >> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> >> for truncation when starting a mirror. When a
> >> mirror
> >> >> is
> >> >> >> >> >> stopped
> >> >> >> >> >> >> at
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> >> source, we already bump up the leader epoch and
> >> write
> >> >> >> some
> >> >> >> >> >> >> >> additional
> >> >> >> >> >> >> >> >> >> records for cleanup. It seems that we can just
> use
> >> >> the
> >> >> >> >> >> >> >> DivergingEpoch
> >> >> >> >> >> >> >> >> >> returned in the initial mirror Fetch request to
> >> >> detect
> >> >> >> the
> >> >> >> >> >> >> diverging
> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> >> and use that for truncation at the mirror
> >> >> destination.
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> JR52. Stop Mirror:
> >> >> >> >> >> >> >> >> >> JR52.1 "3. If patterns are provided, the
> >> controller
> >> >> >> removes
> >> >> >> >> >> >> matching
> >> >> >> >> >> >> >> >> >> entries from mirror.topics.include or adds them
> to
> >> >> >> >> >> >> >> >> mirror.topics.exclude
> >> >> >> >> >> >> >> >> >> on
> >> >> >> >> >> >> >> >> >> the CLUSTER_MIRROR resource."
> >> >> >> >> >> >> >> >> >> Why is it 'or' and not 'and'?
> >> >> >> >> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh
> cycles,
> >> >> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> >> >> >> discovers new source topics matching the
> persisted
> >> >> >> >> >> >> include/exclude
> >> >> >> >> >> >> >> >> >> patterns
> >> >> >> >> >> >> >> >> >> and repeats steps 3 through 7 for each."
> >> >> >> >> >> >> >> >> >> The broker will send a create topic request to
> the
> >> >> >> >> controller.
> >> >> >> >> >> >> How
> >> >> >> >> >> >> >> does
> >> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> >> controller associate it with the mirror name to
> >> >> create a
> >> >> >> >> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> JR53. Describe Mirrors:
> >> >> >> >> >> >> >> >> >> JR53.1 "1. The user sends a
> DescribeClusterMirrors
> >> >> >> request
> >> >> >> >> >> with
> >> >> >> >> >> >> >> >> optional
> >> >> >> >> >> >> >> >> >> mirror names (empty means all mirrors)."
> >> >> >> >> >> >> >> >> >> In metadata request, we use a null arrary to
> >> >> represent
> >> >> >> all
> >> >> >> >> >> >> topics.
> >> >> >> >> >> >> >> It
> >> >> >> >> >> >> >> >> >> would
> >> >> >> >> >> >> >> >> >> be useful to be consistent here.
> >> >> >> >> >> >> >> >> >> JR53.2 "2. The admin client fans out the request
> >> to
> >> >> all
> >> >> >> >> >> brokers
> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> merges
> >> >> >> >> >> >> >> >> >> the responses client-side."
> >> >> >> >> >> >> >> >> >> This is not ideal. Not every broker hosts a
> mirror
> >> >> >> >> partition.
> >> >> >> >> >> So
> >> >> >> >> >> >> >> >> sending
> >> >> >> >> >> >> >> >> >> the request to every broker is wasteful. It's
> >> >> probably
> >> >> >> >> better
> >> >> >> >> >> to
> >> >> >> >> >> >> >> send
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> >> request to a single broker, which can then
> contact
> >> >> the
> >> >> >> >> right
> >> >> >> >> >> >> >> >> >> MirrorFetcherManager and
> ClusterMirrorCoordinator.
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState:
> >> Are
> >> >> >> >> >> MIRRORING,
> >> >> >> >> >> >> >> PAUSED,
> >> >> >> >> >> >> >> >> >> STOPPED the only validate states? The state
> table
> >> has
> >> >> >> other
> >> >> >> >> >> >> states
> >> >> >> >> >> >> >> too.
> >> >> >> >> >> >> >> >> >> Ditto for MirrorPartitionState.
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> Jun
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri
> <
> >> >> >> >> >> >> >> fedevaleri@gmail.com>
> >> >> >> >> >> >> >> >> >> wrote:
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >> > Hi Andrew, yes, validation is called at the
> >> start
> >> >> of
> >> >> >> >> >> >> createTopic.
> >> >> >> >> >> >> >> If
> >> >> >> >> >> >> >> >> >> > the topic ID is invalid, the request fails
> with
> >> >> >> >> >> >> INVALID_REQUEST.
> >> >> >> >> >> >> >> If
> >> >> >> >> >> >> >> >> >> > the topic ID is already used by a different
> >> topic
> >> >> >> name,
> >> >> >> >> it
> >> >> >> >> >> >> fails
> >> >> >> >> >> >> >> with
> >> >> >> >> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the
> >> >> >> individual
> >> >> >> >> >> topic
> >> >> >> >> >> >> in
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> >> > batch fails; other topics in the same
> >> CreateTopics
> >> >> >> >> request
> >> >> >> >> >> >> >> continue
> >> >> >> >> >> >> >> >> >> > processing normally. Added this information to
> >> the
> >> >> >> KIP.
> >> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew
> >> Schofield <
> >> >> >> >> >> >> >> >> >> schofielaj@apache.org>
> >> >> >> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> >> >> > > Hi Fede,
> >> >> >> >> >> >> >> >> >> > > One more small question.
> >> >> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> >> >> > > AS37: If the controller fails validation for
> >> the
> >> >> >> >> >> >> >> MirrorInfo.TopicId
> >> >> >> >> >> >> >> >> >> > added to the CreateTopic request, what does it
> >> do?
> >> >> I
> >> >> >> >> suspect
> >> >> >> >> >> >> that
> >> >> >> >> >> >> >> it
> >> >> >> >> >> >> >> >> >> fails
> >> >> >> >> >> >> >> >> >> > the request with a particular error code.
> >> >> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> >> >> > > Thanks,
> >> >> >> >> >> >> >> >> >> > > Andrew
> >> >> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri
> wrote:
> >> >> >> >> >> >> >> >> >> > > > Hi all,
> >> >> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> >> >> > > > We updated the KIP to include Coordinator
> >> >> Runtime
> >> >> >> >> >> >> >> configurations
> >> >> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> >> > > > the coordinator state transition
> validation
> >> >> >> >> paragraph.
> >> >> >> >> >> The
> >> >> >> >> >> >> >> latter
> >> >> >> >> >> >> >> >> >> > > > covers the per-partition epoch fencing
> >> >> mechanism
> >> >> >> >> (leader
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> >> > > > state epoch) that complements the existing
> >> >> >> per-topic
> >> >> >> >> >> >> >> StateOffset
> >> >> >> >> >> >> >> >> >> > > > fencing for lifecycle operations.
> >> >> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> >> >> > > > Thanks,
> >> >> >> >> >> >> >> >> >> > > > Fede
> >> >> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >
> >> >> >> >> >> >>
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply. The KIP LGTM now.
Jun
On Tue, Aug 18, 2026 at 1:08 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the comments.
> Answering them below.
>
> > JR80. AdminClient.startMirrorTopics(): Is includePatterns() optional? It
> seems that mirroring can't start without it.
>
> Ah, right. I’ve moved the includePatterns from the
> StartMirrorTopicsOptions into the required parameter in
> AdminClient.startMirrorTopics().
>
>
> > JR91. Mirror configs: Some of the configs are prefixed with "mirror,"
> while
> others are not. Could we make them consistent? Since all those configs are
> for Mirror, the mirror prefix seems redundant.
> Indeed, we added the “mirror” prefix because we were afraid of confusing
> users like “mirror.fetch.min.bytes” and “replica.fetch.min.bytes”. But like
> you said, these configurations are for CLUSTER_MIRROR resource type, not
> for brokers. Users should be able to distinguish between them easily.
> Removed the mirror prefix from the mirror-level configs.
>
> Thank you,
> Luke
>
>
> On Tue, Aug 18, 2026 at 2:04 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Luke,
>>
>> Thanks for the reply.
>>
>> JR80. AdminClient.startMirrorTopics(): Is includePatterns() optional? It
>> seems that mirroring can't start without it.
>>
>> JR91. Mirror configs: Some of the configs are prefixed with "mirror,"
>> while
>> others are not. Could we make them consistent? Since all those configs are
>> for Mirror, the mirror prefix seems redundant.
>>
>> Jun
>>
>>
>> On Mon, Aug 17, 2026 at 1:11 AM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for the good comments.
>> > Answering them below.
>> >
>> > > JR80. In that case, it seems that there is no need for
>> > AdminClient.startMirrorTopics() to take topics. The adminClient only
>> uses
>> > it to register the included topics, which can be passed in as
>> > includePatterns (specific names or regex). The broker will use
>> > StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
>> > through the AdminClient.
>> >
>> > Make sense. So when users invoke admin.startMirrorTopics, they only need
>> > to set the topic patterns. And the receiving broker will resolve the
>> > patterns into the actual topics using the listTopic response from the
>> > source cluster, and get the topic id, numPartitions, … via the
>> > describeTopics request from the source cluster if the topic is
>> non-existent
>> > in the destination cluster. KIP is updated.
>> >
>> >
>> > > JR82. If “IncludeMirrorOffset” is disabled,
>> > DescribeClusterMirrorsResponse.SourceOffset is still returned in the
>> > response and what is its value in this case?
>> >
>> > It will be -1 because when the `IncludeMirrorOffset` is disabled, we
>> won’t
>> > query the mirror partition leader. I’ve updated the description to make
>> it
>> > clear. Thanks.
>> >
>> >
>> > > JR88. Why is partition-load-time a Meter instead of a Gauge?
>> >
>> > You’re right. It is not the Meter type. It’s the SampledStat state like
>> > “batch-size-avg” and “batch-size-max” in producer. I’ve updated the KIP.
>> >
>> >
>> > > JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
>> >
>> > You’re right. It’s the rate type. KIP is updated
>> >
>> >
>> > > JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
>> > Meter.
>> > Yes, it makes more sense to make it as Meter type. I’ve updated these
>> > error-related metrics to Meter type:
>> > 1. MetadataRefreshError
>> > 2. TopicConfigMetadataSyncError
>> > 3. ConsumerGroupOffsetSyncError
>> > 4. ShareGroupOffsetSyncError
>> > 5. AclSyncError
>> >
>> >
>> > Thank you,
>> > Luke
>> >
>> >
>> > On Sat, Aug 15, 2026 at 8:10 AM Jun Rao via dev <de...@kafka.apache.org>
>> > wrote:
>> >
>> >> Hi, Luke,
>> >>
>> >> Thanks for the reply.
>> >>
>> >> JR80. In that case, it seems that there is no need for
>> >> AdminClient.startMirrorTopics() to take topics. The adminClient only
>> uses
>> >> it to register the included topics, which can be passed in as
>> >> includePatterns (specific names or regex). The broker will use
>> >> StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
>> >> through the AdminClient.
>> >>
>> >> JR82. If “IncludeMirrorOffset” is disabled,
>> >> DescribeClusterMirrorsResponse.SourceOffset is still returned in the
>> >> response and what is its value in this case?
>> >>
>> >> JR88. Why is partition-load-time a Meter instead of a Gauge?
>> >>
>> >> JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
>> >>
>> >> JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
>> >> Meter.
>> >>
>> >> Jun
>> >>
>> >> On Fri, Aug 14, 2026 at 4:32 AM Luke Chen <sh...@gmail.com> wrote:
>> >>
>> >> > Hi Jun,
>> >> >
>> >> > Thanks for the good comments,
>> >> > Answering them below.
>> >> >
>> >> > > JR80. PauseMirrorTopicsOptions:
>> "PauseMirrorTopics/ResumeMirrorTopics
>> >> > supports regex in the ClusterMirrorComamnd, but we don’t need to add
>> the
>> >> > “patterns”."
>> >> > So, in this case, we use topics to support both specific topics and
>> >> regex.
>> >> > However, startMirrorTopicsOptions uses topics for specific topics and
>> >> > includePatterns for regex. Could we make them consistent?
>> >> >
>> >> >
>> >> > The purpose of the “includePatterns” field is for the
>> >> > “mirror.topics.include” update, so that if there is any newly created
>> >> topic
>> >> > in the source cluster that matches the regex, we can create it. And
>> the
>> >> > “Topics” field in the StartMirrorTopicsRequest is to specify the
>> topicId
>> >> > and numPartitions in the source cluster. So I think “includePatterns”
>> >> and
>> >> > “Topics” both are necessary in the StartMirrorTopicsRequest. On the
>> >> other
>> >> > hand, in PauseMirrorTopics, because it won’t change the
>> >> > “mirror.topics.include”/”mirror.topics.exclude” configs, we don’t
>> need
>> >> to
>> >> > provide the regex in the request.
>> >> >
>> >> >
>> >> >
>> >> > > JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value
>> also
>> >> -1
>> >> > if
>> >> > it's not included in the request?
>> >> >
>> >> > If “IncludeMirrorOffset” is disabled, we won’t query the mirror
>> >> partition
>> >> > leader, so we can’t get the sourceOffset.
>> >> >
>> >> > I assume you want to ask `if it’s still -1 if not actively mirroring,
>> >> > while “IncludeMirrorOffset” is enabled`. That’s a good question. I
>> think
>> >> > our purpose of this command is to show the lag to users. So if the
>> >> state is
>> >> > not MIRRORING (ex: STOPPED/PAUSED), we think it’s not helpful to
>> users
>> >> to
>> >> > show only the sourceOffset. Of course users can use
>> >> `kafka-get-offsets.sh`
>> >> > to get the offset if they really want to know.
>> >> >
>> >> >
>> >> > > JR83. Start Mirror Topics:
>> >> > > JR83.1 Should the controller verify the expected state for the
>> topic?
>> >> >
>> >> > As replied in JR84, the desired state validation should happen in the
>> >> > broker. And the controller validates the StateOffset. I’ve updated
>> the
>> >> > workflow to make it clear.
>> >> >
>> >> > > JR83.2 "d. truncates the log accordingly,"
>> >> > Could you add a description of the truncation offset?
>> >> >
>> >> > OK, added one more sentence:
>> >> > d. based on the last mirror epoch response in (b), the broker
>> translates
>> >> > the epoch into the offset by querying the leaderEpochCache
>> >> > (endOffsetForEpoch), and truncates the log accordingly, and waits for
>> >> all
>> >> > ISR members…
>> >> >
>> >> >
>> >> > > JR84. Stop Mirror Topics: "The controller verifies that the desired
>> >> state
>> >> > of all topics are in MIRRORING or PAUSED state."
>> >> > Should the broker verfy the topic level state? I thought the
>> controller
>> >> > will use StateOffset in the request to make sure the validation on
>> the
>> >> > broker is stil valid. If the controller performs the verification, is
>> >> > StateOffset still useful?
>> >> >
>> >> > Oh, you’re right. We’re doing duplicated work here. Yes, the broker
>> >> > already verifies the “desired state” of all topics are in MIRRORING
>> or
>> >> > PAUSED state, as well as the “actual state” is in MIRRORING or PAUSED
>> >> > state. If all validation passes, we’ll forward it to the controller.
>> So
>> >> the
>> >> > controller only needs to validate the StateOffset to make sure the
>> >> > validation on the broker is still valid. So, yes, we don’t need the
>> >> desired
>> >> > state validation on the controller side. I’ve updated the KIP to
>> make it
>> >> > clear.
>> >> >
>> >> > One more thing to add is that, I’ve thought about not adding
>> >> StateOffset,
>> >> > and relying on the desired state validation to make sure we’re safe
>> to
>> >> do
>> >> > the deletion or other operations. The problem is that, even if the
>> >> broker
>> >> > sees the desired state for topic A is STOPPED, and then the
>> controller
>> >> also
>> >> > sees the desired state is STOPPED, it still cannot make sure the
>> >> > controller’s STOPPED state equals the broker’s STOPPED state. For
>> >> example:
>> >> > 1. Broker sees STOPPED, then forward to controller
>> >> > 2. User changes the desired state to MIRRORING
>> >> > 3. User changes the desired state to STOPPED
>> >> > 4. Controller sees the desired state is STOPPED
>> >> > So I think StateOffset is required.
>> >> >
>> >> > Thanks for catching it BTW.
>> >> >
>> >> >
>> >> > > JR85. There are still a few references to includeMirrorLags. They
>> >> should
>> >> > be
>> >> > IncludeMirrorOffset.
>> >> >
>> >> > Sorry, just updated.
>> >> >
>> >> >
>> >> > > JR86. Metrics:
>> >> > > JR86.1 Could you describe the type (gauge, meter, etc) of each
>> metric?
>> >> >
>> >> > KIP is updated. I also group the metrics into new added metrics and
>> >> other
>> >> > inherited metrics.
>> >> >
>> >> > > JR86.2 Could you describe a bit more what MinFetchRate represents?
>> >> >
>> >> > It means the lowest fetch request rate among all of this mirror
>> fetcher
>> >> > manager's fetcher threads. It inherits from the
>> AbstractFetcherManager’s
>> >> > metric.
>> >> >
>> >> >
>> >> > > JR87. Version Compatibility: "For migration or DR use cases where
>> >> > failback
>> >> > is not required, this proposal supports source brokers up to version
>> 2.1
>> >> > included"
>> >> > This KIP needs topicId and KIP-516 was introduced later than 2.1.
>> >> > Yes, when mirroring from a cluster without topic id support, we will
>> >> > assign a random topic ID in the destination cluster and then fetch
>> >> (without
>> >> > topic id in the fetch request using the old version of Fetch API) as
>> >> usual.
>> >> > Of course this is for the one-way mirroring use case. It can’t start
>> the
>> >> > reverse mirror(failback) under this setup. You can check the
>> >> “Compatibility
>> >> > Matrix” section for more detail.
>> >> >
>> >> >
>> >> > Thank you,
>> >> > Luke
>> >> >
>> >> > On Fri, Aug 14, 2026 at 7:09 AM Jun Rao via dev <
>> dev@kafka.apache.org>
>> >> > wrote:
>> >> >
>> >> >> Hi, Luke,
>> >> >>
>> >> >> Thanks for the reply.
>> >> >>
>> >> >> JR80. PauseMirrorTopicsOptions:
>> "PauseMirrorTopics/ResumeMirrorTopics
>> >> >> supports regex in the ClusterMirrorComamnd, but we don’t need to add
>> >> the
>> >> >> “patterns”."
>> >> >> So, in this case, we use topics to support both specific topics and
>> >> regex.
>> >> >> However, startMirrorTopicsOptions uses topics for specific topics
>> and
>> >> >> includePatterns for regex. Could we make them consistent?
>> >> >>
>> >> >> JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also
>> >> -1 if
>> >> >> it's not included in the request?
>> >> >>
>> >> >> JR83. Start Mirror Topics:
>> >> >> JR83.1 Should the controller verify the expected state for the
>> topic?
>> >> >> JR83.2 "d. truncates the log accordingly,"
>> >> >> Could you add a description of the truncation offset?
>> >> >>
>> >> >> JR84. Stop Mirror Topics: "The controller verifies that the desired
>> >> state
>> >> >> of all topics are in MIRRORING or PAUSED state."
>> >> >> Should the broker verfy the topic level state? I thought the
>> controller
>> >> >> will use StateOffset in the request to make sure the validation on
>> the
>> >> >> broker is stil valid. If the controller performs the verification,
>> is
>> >> >> StateOffset still useful?
>> >> >>
>> >> >> JR85. There are still a few references to includeMirrorLags. They
>> >> should
>> >> >> be
>> >> >> IncludeMirrorOffset.
>> >> >>
>> >> >> JR86. Metrics:
>> >> >> JR86.1 Could you describe the type (gauge, meter, etc) of each
>> metric?
>> >> >> JR86.2 Could you describe a bit more what MinFetchRate represents?
>> >> >>
>> >> >> JR87. Version Compatibility: "For migration or DR use cases where
>> >> failback
>> >> >> is not required, this proposal supports source brokers up to version
>> >> 2.1
>> >> >> included"
>> >> >> This KIP needs topicId and KIP-516 was introduced later than 2.1.
>> >> >>
>> >> >> Jun
>> >> >>
>> >> >>
>> >> >> On Thu, Aug 13, 2026 at 5:14 AM Luke Chen <sh...@gmail.com>
>> wrote:
>> >> >>
>> >> >> > Hi Jun,
>> >> >> >
>> >> >> > Thanks for the good comments.
>> >> >> > Answering them below.
>> >> >> >
>> >> >> > > JR59.1 Could you add the RPC to the workflow? The broker first
>> >> needs
>> >> >> to
>> >> >> > issue ListClusterMirrorsRequest with SourceClusterIdFilter to get
>> the
>> >> >> > mirror name and then issue DescribeClusterMirrors to get the LME.
>> >> >> >
>> >> >> > Yes, you’re right. The workflow is updated to make them clear.
>> >> >> >
>> >> >> > > JR71. mirror.support.unclean.leader.election : The name is
>> >> confusing.
>> >> >> > It's
>> >> >> > also used in LOG_TRUNCATION, which is not related to unclean
>> leader
>> >> >> > election. Could we use a better name?
>> >> >> >
>> >> >> > I personally think the name should be fine because even though we
>> >> use in
>> >> >> > “LOG_TRUNCATION” state, the config is to allow us to know if users
>> >> want
>> >> >> to
>> >> >> > support unclean leader election, if so, we wait for all replicas
>> to
>> >> >> catch
>> >> >> > up. If we want to explicitly show the intention of the config, how
>> >> about
>> >> >> > “mirror.full.replica.convergence.required”? I’m open to other
>> >> >> suggestions.
>> >> >> >
>> >> >> >
>> >> >> > > JR72. DescribeClusterMirrors
>> >> >> > > JR72.1 A user may specify just the topic name when describing a
>> >> >> mirror.
>> >> >> > Should we include the topic name in the request?
>> >> >> >
>> >> >> > Hmm.. we didn’t think about the use case. But yes, if users only
>> care
>> >> >> > about the state of a topic, they don’t need the info for other
>> >> topics.
>> >> >> I’ve
>> >> >> > combined with JR72.2, added “TopicPartitions” filed into
>> >> >> > DescribeClusterMirrorsRequest, and by default is null, which means
>> >> all
>> >> >> > topic partitions under the requested mirrors will be returned. If
>> >> set,
>> >> >> > we’ll query the mirrors and return the topic partitions that users
>> >> >> > expected. KIP is updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR72.2 LastMirrorEpochLookups is a bit weird since the response
>> >> >> returns
>> >> >> > either mirror state or mirror lag and LastMirrorEpochLookups
>> doesn't
>> >> >> match
>> >> >> > either.
>> >> >> >
>> >> >> > Yes, after the recent refactor, I think the LastMirrorEpochLookups
>> >> can
>> >> >> be
>> >> >> > merged into one level up, combining with the topic name filter in
>> >> above
>> >> >> > comment, which will allow users to query down to the partition
>> level.
>> >> >> The
>> >> >> > LastMirrorEpochLookups will query along with normal mirror state
>> by
>> >> the
>> >> >> > readMirrorState from the coordinator. The difference is that after
>> >> >> > receiving the response, it will filter out the mirrors that don’t
>> >> match
>> >> >> the
>> >> >> > requested sourceClusterId, and then add the lastMirrorEpoch into
>> >> >> > DescribeClusterMirrorResponse. KIP is updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> >> >
>> >> >> > Right. Added.
>> >> >> >
>> >> >> >
>> >> >> > > JR72.4 "includeMirrorLags: source offset, destination offset,
>> lag"
>> >> >> > DescribeClusterMirrorsRequest doesn't include those fields. Also,
>> lag
>> >> >> > should be omitted.
>> >> >> >
>> >> >> > Sorry, I removed the “SourceOffset” and “DestinationOffset” fields
>> >> from
>> >> >> > DescribeClusterMirrorsResponse because I added them into
>> >> >> ReadMirrorOffsets
>> >> >> > (ex-ReadMirrorLags) API. But I forgot we still need them in the
>> >> >> > DescribeClusterMirrorsResponse so that the adminClient can receive
>> >> the
>> >> >> > results. KIP is updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR73. MirrorPartitionState record, LastMirrorEpochs record and
>> >> >> > MirrorPartitionStateRequest: The values for type and name are
>> >> incorrect.
>> >> >> >
>> >> >> > Ah, that’s my bad. Fixed.
>> >> >> >
>> >> >> > > JR74. ListClusterMirrorsRequest
>> >> >> > > JR74.1 When is DesiredMirrorStateFilter used? There is no
>> option to
>> >> >> > specify
>> >> >> > the desired state in the mirror-topic command.
>> >> >> >
>> >> >> > Yes, referred to kafka-consumer-groups.sh, I added the `--state`
>> >> option
>> >> >> to
>> >> >> > allow users to set the state when listMirror.
>> >> >> >
>> >> >> > > JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>> >> >> >
>> >> >> > Yes, KIP updated.
>> >> >> >
>> >> >> > > JR75. ReadMirrorLags
>> >> >> > > JR75.1 Should it be named ReadMirrorOffsets since the response
>> >> doesn't
>> >> >> > include lag?
>> >> >> >
>> >> >> > Indeed. Updated.
>> >> >> >
>> >> >> > > JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> >> >
>> >> >> > Yes, added
>> >> >> >
>> >> >> > > JR75.3 Some places refer to this as ReadClusterMirrorLags. We
>> need
>> >> to
>> >> >> be
>> >> >> > consistent.
>> >> >> >
>> >> >> > Thanks for pointing it out. KIP is updated.
>> >> >> >
>> >> >> > > JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR
>> >> too?
>> >> >> >
>> >> >> > Yes, we should validate the mirror name before writing the data.
>> KIP
>> >> is
>> >> >> > updated.
>> >> >> >
>> >> >> > > JR77. Delete Mirror: "3. The controller verifies that no topics
>> >> are in
>> >> >> > MIRRORING or PAUSED state."
>> >> >> > Is this necessary since step 2 has verified all partitions are in
>> >> >> STOPPED
>> >> >> > state, which can only happen if the topics are in STOPPED state.
>> >> >> >
>> >> >> > It’s true, but there could be some delay between “actual state”
>> and
>> >> >> > “desired state”. In step (2), we verify all partitions’ “actual
>> >> state”
>> >> >> are
>> >> >> > in STOPPED state. But the current desired state could be
>> MIRRORING,
>> >> it’s
>> >> >> > just that the mirror partition leaders haven’t got the metadata
>> >> update
>> >> >> yet.
>> >> >> > So, the controller still needs to verify the desired state is not
>> in
>> >> >> > MIRRORING or PAUSED. I found we didn’t specify that the
>> controller is
>> >> >> > actually verifying the “desired state”, not “actual state”. And
>> this
>> >> >> > controller’s desired state validation also applies in
>> >> >> >
>> >>
>> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
>> >> >> > I’ve updated the KIP to make it clear.
>> >> >> >
>> >> >> > > JR78. STOPPING: "The mirror fetcher is removed, the leader
>> epoch is
>> >> >> > bumped,
>> >> >> > ABORT markers are appended for all ongoing transactions, last
>> mirror
>> >> >> epochs
>> >> >> > are persisted, and a MIRROR_PID_RESET record appended."
>> >> >> > This describes the state for STOPPED, not STOPPING.
>> >> >> >
>> >> >> > No, these are the tasks during STOPPING state, like what we
>> >> described in
>> >> >> > “Stop Mirror Topics” workflow. If we do these in STOPPED state, we
>> >> can’t
>> >> >> > safely move the readonly topic into writable because not all
>> required
>> >> >> tasks
>> >> >> > are done. I think I know why it confuses you, it’s because all the
>> >> tasks
>> >> >> > are in the completed state. I’ve rephrased it to make it clear.
>> >> >> >
>> >> >> >
>> >> >> > > JR79. RPCs like DescribeClusterMirrors can take time. Should we
>> >> add a
>> >> >> > timeout field?
>> >> >> >
>> >> >> > Make sense. Added into `DescribeClusterMirrors` request, as well
>> as
>> >> >> >
>> >> >>
>> >>
>> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
>> >> >> > because we will query the partition states from coordinator for
>> state
>> >> >> > validation in these APIs.
>> >> >> >
>> >> >> >
>> >> >> > > JR80. PauseMirrorTopicsOptions: Should it support patterns? The
>> >> >> command
>> >> >> > line seems to suppor it.
>> >> >> >
>> >> >> > Sorry, it’s a little confusing here. The “pattern” in
>> >> >> > startMirrorTopicsOptions or stopMirrorTopicsOptions is to update
>> the
>> >> >> > “mirror.topics.include” and “mirror.topics.exclude” configuration
>> for
>> >> >> topic
>> >> >> > pattern discovery periodically. For the PauseMirrorTopics and
>> >> >> > ResumeMirrorTopics in the clusterMirrorCommand, users can provide
>> >> regex
>> >> >> in
>> >> >> > the “--topics” option, and the clusterMirrorCommand will
>> listTopic in
>> >> >> the
>> >> >> > destination cluster and match with the patterns users provided,
>> and
>> >> then
>> >> >> > feeds the matched topics into the
>> >> PauseMirrorTopics/ResumeMirrorTopics
>> >> >> > requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics
>> >> >> supports
>> >> >> > regex in the ClusterMirrorComamnd, but we don’t need to add the
>> >> >> “patterns”.
>> >> >> >
>> >> >> >
>> >> >> > > JR81. AdminClient
>> >> >> > > JR81.1 ListClusterMirrors: We need to add the option to specify
>> >> >> different
>> >> >> > filters supported in the request.
>> >> >> >
>> >> >> > We already have the `--mirror` option. Added `--state` and
>> >> >> > `--sourceClusterId` options to support the other filters.
>> >> >> >
>> >> >> > > JR81.2
>> DescribeClusterMirrorsOptions.includeAuthorizedOperations:
>> >> The
>> >> >> > request doesn't support this option. Could we add other options
>> >> >> supported
>> >> >> > in the request?
>> >> >> >
>> >> >> > Currently, we have “includeAuthorizedOperations” in
>> DescribeCluster,
>> >> >> > DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
>> >> >> > DescribeStreamsGroups requests. But none of them include the
>> option
>> >> in
>> >> >> the
>> >> >> > tool to allow users to enable it. I think the reason is that we
>> don’t
>> >> >> want
>> >> >> > to show this to users on the tool output. But having this field,
>> we
>> >> >> allow
>> >> >> > the advanced users to invoke admin API with
>> >> includeAuthorizedOperations
>> >> >> > enabled to get the results they want.
>> >> >> >
>> >> >> >
>> >> >> > > JR81.3 LeaderStateDescription: Should it be named
>> >> >> > MirrorPartitionStateDescription? Also, could we make the fields
>> >> >> consistent
>> >> >> > with those in the response?
>> >> >> >
>> >> >> > OK. KIP is updated.
>> >> >> >
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Luke
>> >> >> >
>> >> >> > On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <
>> >> dev@kafka.apache.org>
>> >> >> > wrote:
>> >> >> >
>> >> >> >> Hi, Luke,
>> >> >> >>
>> >> >> >> Thanks for the reply.
>> >> >> >>
>> >> >> >> JR59.1 Could you add the RPC to the workflow? The broker first
>> >> needs to
>> >> >> >> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get
>> >> the
>> >> >> >> mirror name and then issue DescribeClusterMirrors to get the LME.
>> >> >> >>
>> >> >> >> JR71. mirror.support.unclean.leader.election : The name is
>> >> confusing.
>> >> >> It's
>> >> >> >> also used in LOG_TRUNCATION, which is not related to unclean
>> leader
>> >> >> >> election. Could we use a better name?
>> >> >> >>
>> >> >> >> JR72. DescribeClusterMirrors
>> >> >> >> JR72.1 A user may specify just the topic name when describing a
>> >> mirror.
>> >> >> >> Should we include the topic name in the request?
>> >> >> >> JR72.2 LastMirrorEpochLookups is a bit weird since the response
>> >> returns
>> >> >> >> either mirror state or mirror lag and LastMirrorEpochLookups
>> doesn't
>> >> >> match
>> >> >> >> either.
>> >> >> >> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> >> >> JR72.4 "includeMirrorLags: source offset, destination offset,
>> lag"
>> >> >> >> DescribeClusterMirrorsRequest doesn't include those fields. Also,
>> >> lag
>> >> >> >> should be omitted.
>> >> >> >>
>> >> >> >> JR73. MirrorPartitionState record, LastMirrorEpochs record and
>> >> >> >> MirrorPartitionStateRequest: The values for type and name are
>> >> >> incorrect.
>> >> >> >>
>> >> >> >> JR74. ListClusterMirrorsRequest
>> >> >> >> JR74.1 When is DesiredMirrorStateFilter used? There is no option
>> to
>> >> >> >> specify
>> >> >> >> the desired state in the mirror-topic command.
>> >> >> >> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>> >> >> >>
>> >> >> >> JR75. ReadMirrorLags
>> >> >> >> JR75.1 Should it be named ReadMirrorOffsets since the response
>> >> doesn't
>> >> >> >> include lag?
>> >> >> >> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> >> >> JR75.3 Some places refer to this as ReadClusterMirrorLags. We
>> need
>> >> to
>> >> >> be
>> >> >> >> consistent.
>> >> >> >>
>> >> >> >> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR
>> >> too?
>> >> >> >>
>> >> >> >> JR77. Delete Mirror: "3. The controller verifies that no topics
>> are
>> >> in
>> >> >> >> MIRRORING or PAUSED state."
>> >> >> >> Is this necessary since step 2 has verified all partitions are in
>> >> >> STOPPED
>> >> >> >> state, which can only happen if the topics are in STOPPED state.
>> >> >> >>
>> >> >> >> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch
>> is
>> >> >> >> bumped,
>> >> >> >> ABORT markers are appended for all ongoing transactions, last
>> mirror
>> >> >> >> epochs
>> >> >> >> are persisted, and a MIRROR_PID_RESET record appended."
>> >> >> >> This describes the state for STOPPED, not STOPPING.
>> >> >> >>
>> >> >> >> JR79. RPCs like DescribeClusterMirrors can take time. Should we
>> add
>> >> a
>> >> >> >> timeout field?
>> >> >> >>
>> >> >> >> JR80. PauseMirrorTopicsOptions: Should it support patterns? The
>> >> command
>> >> >> >> line seems to suppor it.
>> >> >> >>
>> >> >> >> JR81. AdminClient
>> >> >> >> JR81.1 ListClusterMirrors: We need to add the option to specify
>> >> >> different
>> >> >> >> filters supported in the request.
>> >> >> >> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations:
>> >> The
>> >> >> >> request doesn't support this option. Could we add other options
>> >> >> supported
>> >> >> >> in the request?
>> >> >> >> JR81.3 LeaderStateDescription: Should it be named
>> >> >> >> MirrorPartitionStateDescription? Also, could we make the fields
>> >> >> consistent
>> >> >> >> with those in the response?
>> >> >> >>
>> >> >> >> Jun
>> >> >> >>
>> >> >> >> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com>
>> >> wrote:
>> >> >> >>
>> >> >> >> > Hi Jun,
>> >> >> >> >
>> >> >> >> > Thanks for the good comments again.
>> >> >> >> > Answering them below.
>> >> >> >> >
>> >> >> >> > > JR51. "ULE_LOG_TRUNCATION When
>> >> >> mirror.support.unclean.leader.election
>> >> >> >> is
>> >> >> >> > enabled and the destination leader has log truncation, the
>> state
>> >> will
>> >> >> >> move
>> >> >> >> > from MIRRORING to this state."
>> >> >> >> > This still suggests that divergeEpoch is only handled if
>> >> >> >> > mirror.support.unclean.leader.election
>> >> >> >> > is enabled.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > OK, KIP is updated to make it clear.
>> >> >> >> >
>> >> >> >> > > JR55. Should the mirror state and the mirror lags always be
>> >> exposed
>> >> >> >> > together? If so, we can remove IncludeMirrorState and
>> >> >> IncludeMirrorLags
>> >> >> >> > from DescribeClusterMirrorsRequest. If not, it's better to use
>> two
>> >> >> >> separate
>> >> >> >> > RPCs.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > No, we sometimes only need MirrorState, and sometimes need
>> both.
>> >> So
>> >> >> I’ll
>> >> >> >> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
>> >> >> >> > DescribeClusterMirrorsRequest. And we will use ReadMirrorStates
>> >> and
>> >> >> >> > ReadMirrorLags to get each info.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > DescribeClusterMirrorsRequest should be used in the Describe
>> >> Mirror
>> >> >> >> flow,
>> >> >> >> > right? Currently it's missing. Also, I assume the admin client
>> >> >> refers to
>> >> >> >> > the one in the kafka-cluster-mirrors tool. But it seems that
>> >> >> sometimes
>> >> >> >> it
>> >> >> >> > also refers to the admin client on the broker? Perhaps we can
>> use
>> >> >> 'the
>> >> >> >> > tool' instead to make it clear.
>> >> >> >> >
>> >> >> >> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
>> >> >> >> > And sorry for the confusion. I’d like to keep the admin client
>> as
>> >> the
>> >> >> >> one
>> >> >> >> > invoking from the tool (or user’s direct admin client API
>> >> >> invocation).
>> >> >> >> On
>> >> >> >> > the broker side, we actually use InterBrokerSendThread to
>> invoke
>> >> >> these
>> >> >> >> RPCs
>> >> >> >> > or get directly from MetadataImage. I’ve updated the KIP to
>> make
>> >> it
>> >> >> >> clear.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR58. BumpLeaderEpochs: Why does it need to be issued to the
>> >> >> broker,
>> >> >> >> > instead of just to the controller?
>> >> >> >> >
>> >> >> >> > Yes, we only need to issue to the controller. Updated to the
>> >> >> >> > BumpLeaderEpochs API.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR59. Start Mirror Topics:
>> >> >> >> > > JR59.1 "During this state, the broker queries the source
>> cluster
>> >> >> for
>> >> >> >> last
>> >> >> >> > mirror epochs".
>> >> >> >> > Could you describe which RPC is issued to find the LME?
>> >> >> >> >
>> >> >> >> > We’re using describeClusterMirror API via admin client here.
>> >> >> >> >
>> >> >> >> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
>> >> request
>> >> >> >> > doesn't
>> >> >> >> > contain a replication factor.
>> >> >> >> >
>> >> >> >> > We use “default.replication.factor” when topic creation, so the
>> >> >> >> > INVALID_REPLICATION_FACTOR will be thrown like normal topic
>> >> creation.
>> >> >> >> Also,
>> >> >> >> > because we require the replicas for the partition assignment
>> to be
>> >> >> all
>> >> >> >> > active and not in fenced or controlled shutdown(JR35), we’ll
>> throw
>> >> >> >> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already
>> used
>> >> by a
>> >> >> >> > different topic name"
>> >> >> >> > Should we also return this error when the topic name is used
>> as a
>> >> >> >> regular
>> >> >> >> > topic?
>> >> >> >> >
>> >> >> >> > There are 2 errors we will throw when topic id <-> topic name
>> >> >> mismatch.
>> >> >> >> > Suppose provided topic id: ID_A, topic name: NAME_A
>> >> >> >> > 1. Destination cluster exists topic id: ID_A, but topic name
>> is:
>> >> >> NAME_B
>> >> >> >> > TOPIC_ALREADY_EXISTS will be thrown
>> >> >> >> > 2. Destination cluster exists topic id: ID_B, but topic name
>> is:
>> >> >> NAME_A
>> >> >> >> > INCONSISTENT_TOPIC_ID will be thrown
>> >> >> >> >
>> >> >> >> > Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS
>> >> error
>> >> >> >> for
>> >> >> >> > these 2 cases makes more sense. KIP updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR60. CreateClusterMirrorRequest: When will it receive an
>> >> >> >> > INVALID_CLUSTER_MIRROR_STATES error not covered by
>> >> >> >> > CLUSTER_MIRROR_ALREADY_EXISTS?
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Sorry, createClusterMirrorRequest doesn’t throw
>> >> >> >> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate
>> >> the
>> >> >> >> mirror
>> >> >> >> > states when creating a cluster mirror. We only care if the
>> “mirror
>> >> >> name”
>> >> >> >> > exists or not. KIP updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we
>> >> just
>> >> >> >> remove
>> >> >> >> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> >> >> >> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to
>> describe
>> >> >> the
>> >> >> >> > specific state causing the error.
>> >> >> >> >
>> >> >> >> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following
>> >> >> errors:
>> >> >> >> > a. MIRROR_TOPIC_ALREADY_PAUSED
>> >> >> >> > b. MIRROR_TOPIC_NOT_PAUSED,
>> >> >> >> > c. MIRROR_TOPIC_ALREADY_STOPPED,
>> >> >> >> > d. CLUSTER_MIRROR_NOT_EMPTY
>> >> >> >> > e. MIRROR_TOPIC_NOT_STOPPED
>> >> >> >> >
>> >> >> >> > KIP is updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in
>> some
>> >> >> >> requests
>> >> >> >> > (e.g. StopMirrorTopics), but not others (e.g.
>> StartMirrorTopics).
>> >> >> Could
>> >> >> >> we
>> >> >> >> > make them consistent?
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Good catch. The updated errors for the new added RPC are this:
>> >> >> >> >
>> >> >> >> > CreateClusterMirror: NOT_CONTROLLER
>> >> >> >> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> >> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> >> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> >> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> >> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR63. StopMirrorTopics:
>> >> >> >> > > JR63.1 "The broker validates that all target topic partitions
>> >> are
>> >> >> in
>> >> >> >> > either
>> >> >> >> > LOG_TRUNCATION or MIRRORING or FAILED state."
>> >> >> >> > Is this accurate? LOG_TRUNCATION can't transition to STOP.
>> PAUSED
>> >> >> can.
>> >> >> >> >
>> >> >> >> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is
>> >> >> updated.
>> >> >> >> >
>> >> >> >> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error
>> >> since
>> >> >> >> PAUSED
>> >> >> >> > -> STOPPED is valid?
>> >> >> >> >
>> >> >> >> > Yes, it is not correct. KIP updated to remove this error.
>> >> >> >> >
>> >> >> >> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
>> >> >> stopped
>> >> >> >> > state"
>> >> >> >> > This is weird. The reason to issue StopMirrorTopics is that
>> it's
>> >> not
>> >> >> in
>> >> >> >> > stopped state.
>> >> >> >> >
>> >> >> >> > Indeed. KIP is updated to only include
>> >> INVALID_CLUSTER_MIRROR_STATES
>> >> >> >> > error, and will be thrown when states are not in MIRRORING or
>> >> PAUSED
>> >> >> >> state.
>> >> >> >> >
>> >> >> >> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since
>> it's
>> >> >> >> derived
>> >> >> >> > from SourceOffset and DestinationOffset.
>> >> >> >> >
>> >> >> >> > True. Removed.
>> >> >> >> >
>> >> >> >> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It
>> seems
>> >> >> that we
>> >> >> >> > could just set the partition level state to STOPPED.
>> >> >> >> >
>> >> >> >> > Nice catch. We don’t use it at all. That is some old leftover
>> >> field
>> >> >> we
>> >> >> >> > forgot to remove. KIP updated.
>> >> >> >> >
>> >> >> >> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need
>> to be
>> >> >> sent
>> >> >> >> to
>> >> >> >> > the controller?
>> >> >> >> >
>> >> >> >> > No, these 2 only send to the broker. KIP updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and
>> >> >> StateEpoch?
>> >> >> >> It
>> >> >> >> > seems both are used for fencing and fencing based on the
>> latter is
>> >> >> >> > stronger.
>> >> >> >> >
>> >> >> >> > Yes, basically the StateEpoch can fence for most cases. The
>> >> >> LeaderEpoch
>> >> >> >> > fencing is used for cases like:
>> >> >> >> >
>> >> >> >> > 1. b1 is the current leader for foo-0, preparing write state
>> from
>> >> >> >> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
>> >> >> >> > 2. leadership change to b2 for foo-0, LE changes to 2.
>> >> >> >> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1
>> }.
>> >> >> >> > 4. The coordinator receives the read from b2 in step (3), the
>> >> highest
>> >> >> >> > lastSeen LE for foo-0 updates to 2 in coordinator.
>> >> >> >> > 5. The writes state from b1 in step (1) arrives the
>> coordinator,
>> >> but
>> >> >> it
>> >> >> >> > rejects b1 writes because the highest lastSeen LE is 2, which
>> is
>> >> > 1
>> >> >> in
>> >> >> >> the
>> >> >> >> > request.
>> >> >> >> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2,
>> >> stateEpoch: 1
>> >> >> and
>> >> >> >> > succeeds.
>> >> >> >> >
>> >> >> >> > The leaderEpoch fence is to avoid any stale broker writes after
>> >> the
>> >> >> new
>> >> >> >> > leader epoch read. This cannot be protected by the stateEpoch.
>> The
>> >> >> >> > stateEpoch can prevent the write with older stateEpoch. So in
>> the
>> >> >> >> situation
>> >> >> >> > above, the stateEpoch cannot determine if we should accept b1
>> >> writes
>> >> >> or
>> >> >> >> > not, but leaderEpoch can. By the way, we borrowed this concept
>> >> from
>> >> >> >> > KIP-932:
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
>> <https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rUm8b3EcsLLsbBwkWG6hAiL-_Kard9D9TP_fPHp1g519R0HO7gRzlR4X5v2m4r58FdqsJdyCTqKHu14$>
>> >> <
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!t3M02ge9e-mRqsshxasG_K0qLi9sX21suoBd69nLUl_gWUvn9ZLLekJiQspa_q3gTVSzWyUQBQfQLHk$
>> >
>> >> >> <
>> >>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rX3-NwDU3s0U7POQYH1R-DvASnCPTTtFRWxZTVaS-PQHg-C1TAcu9xMe3MQF4fIXGO67WKOqrlz-lc8$
>> >> >
>> >> >> >> <
>> >> >>
>> >>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!u8EkPGXm8_VKdltXy8lrIW14mR2kgYIoaBPD7LO2uAN6rjhrfCX37Pfa6r85QPpSwSPh4V6eW_bKlns$
>> >> >> >
>> >> >> >> > <
>> >> >> >>
>> >> >>
>> >>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR68. Could we make the usage of topicId and topicName in the
>> >> RPCs
>> >> >> >> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> >> >> >> > BumpLeaderEpochsReponse uses TopicName.
>> >> >> >> >
>> >> >> >> > Yes, we have made:
>> >> >> >> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use
>> TopicName.
>> >> >> >> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse
>> use
>> >> >> >> > TopicName.
>> >> >> >> > 3. MirrorTopicStateChangeRecord uses TopicName because we will
>> do
>> >> >> >> > validation before writing the metadata log.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR69. MIRROR_PID_RESET:
>> >> >> >> > > JR69.1 Could you specify the value of the control record's
>> type?
>> >> >> >> >
>> >> >> >> > It’s MIRROR_PID_RESET(7). KIP updated.
>> >> >> >> >
>> >> >> >> > > JR69.2 How is the SourceClusterId field used?
>> >> >> >> >
>> >> >> >> > I’m also surprised to see this field in the
>> MirrorPidResetRecord.
>> >> I
>> >> >> >> traced
>> >> >> >> > back to the internal PR we added for this field, and confirmed
>> it
>> >> is
>> >> >> not
>> >> >> >> > used since it was added. Sorry for the confusion and nice
>> catch!
>> >> KIP
>> >> >> >> > updated.
>> >> >> >> >
>> >> >> >> > > JR70. MirrorPartitionState: type should be "data" not
>> >> >> >> "coordinator-key",
>> >> >> >> > right? Ditto for LastMirrorEpochs.
>> >> >> >> > We store both MirrorPartitionState and LastMirrorEpochs records
>> >> into
>> >> >> >> > internal __mirror_state topic, and rely on Coordinator Runtime
>> to
>> >> >> >> > write/read the records, like how `OffsetCommit` and
>> >> `GroupMetadata`
>> >> >> >> work.
>> >> >> >> > So I think we should keep them as
>> >> >> "coordinator-key"/"coordinator-value"
>> >> >> >> > type.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Thank you,
>> >> >> >> > Luke
>> >> >> >> >
>> >> >> >> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <
>> >> >> dev@kafka.apache.org>
>> >> >> >> > wrote:
>> >> >> >> >
>> >> >> >> >> Hi, Luke,
>> >> >> >> >>
>> >> >> >> >> Thanks for the reply.
>> >> >> >> >>
>> >> >> >> >> JR51. "ULE_LOG_TRUNCATION When
>> >> >> mirror.support.unclean.leader.election
>> >> >> >> is
>> >> >> >> >> enabled and the destination leader has log truncation, the
>> state
>> >> >> will
>> >> >> >> move
>> >> >> >> >> from MIRRORING to this state."
>> >> >> >> >> This still suggests that divergeEpoch is only handled if
>> >> >> >> >> mirror.support.unclean.leader.election
>> >> >> >> >> is enabled.
>> >> >> >> >>
>> >> >> >> >> JR55. Should the mirror state and the mirror lags always be
>> >> exposed
>> >> >> >> >> together? If so, we can remove IncludeMirrorState and
>> >> >> IncludeMirrorLags
>> >> >> >> >> from DescribeClusterMirrorsRequest. If not, it's better to use
>> >> two
>> >> >> >> >> separate
>> >> >> >> >> RPCs.
>> >> >> >> >>
>> >> >> >> >> DescribeClusterMirrorsRequest should be used in the Describe
>> >> Mirror
>> >> >> >> flow,
>> >> >> >> >> right? Currently it's missing. Also, I assume the admin client
>> >> >> refers
>> >> >> >> to
>> >> >> >> >> the one in the kafka-cluster-mirrors tool. But it seems that
>> >> >> sometimes
>> >> >> >> it
>> >> >> >> >> also refers to the admin client on the broker? Perhaps we can
>> use
>> >> >> 'the
>> >> >> >> >> tool' instead to make it clear.
>> >> >> >> >>
>> >> >> >> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the
>> >> broker,
>> >> >> >> >> instead of just to the controller?
>> >> >> >> >>
>> >> >> >> >> JR59. Start Mirror Topics:
>> >> >> >> >> JR59.1 "During this state, the broker queries the source
>> cluster
>> >> for
>> >> >> >> last
>> >> >> >> >> mirror epochs".
>> >> >> >> >> Could you describe which RPC is issued to find the LME?
>> >> >> >> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
>> >> request
>> >> >> >> >> doesn't
>> >> >> >> >> contain a replication factor.
>> >> >> >> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used
>> >> by a
>> >> >> >> >> different topic name"
>> >> >> >> >> Should we also return this error when the topic name is used
>> as a
>> >> >> >> regular
>> >> >> >> >> topic?
>> >> >> >> >>
>> >> >> >> >> JR60. CreateClusterMirrorRequest: When will it receive an
>> >> >> >> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
>> >> >> >> >> CLUSTER_MIRROR_ALREADY_EXISTS?
>> >> >> >> >>
>> >> >> >> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we
>> just
>> >> >> >> remove
>> >> >> >> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> >> >> >> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to
>> >> describe
>> >> >> the
>> >> >> >> >> specific state causing the error.
>> >> >> >> >>
>> >> >> >> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in
>> some
>> >> >> >> requests
>> >> >> >> >> (e.g. StopMirrorTopics), but not others (e.g.
>> StartMirrorTopics).
>> >> >> >> Could we
>> >> >> >> >> make them consistent?
>> >> >> >> >>
>> >> >> >> >> JR63. StopMirrorTopics:
>> >> >> >> >> JR63.1 "The broker validates that all target topic partitions
>> >> are in
>> >> >> >> >> either
>> >> >> >> >> LOG_TRUNCATION or MIRRORING or FAILED state."
>> >> >> >> >> Is this accurate? LOG_TRUNCATION can't transition to STOP.
>> PAUSED
>> >> >> can.
>> >> >> >> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error
>> >> since
>> >> >> >> PAUSED
>> >> >> >> >> -> STOPPED is valid?
>> >> >> >> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
>> >> stopped
>> >> >> >> >> state"
>> >> >> >> >> This is weird. The reason to issue StopMirrorTopics is that
>> it's
>> >> >> not in
>> >> >> >> >> stopped state.
>> >> >> >> >>
>> >> >> >> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since
>> it's
>> >> >> derived
>> >> >> >> >> from SourceOffset and DestinationOffset.
>> >> >> >> >>
>> >> >> >> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems
>> >> that
>> >> >> we
>> >> >> >> >> could just set the partition level state to STOPPED.
>> >> >> >> >>
>> >> >> >> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to
>> be
>> >> >> sent
>> >> >> >> to
>> >> >> >> >> the controller?
>> >> >> >> >>
>> >> >> >> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and
>> >> >> StateEpoch? It
>> >> >> >> >> seems both are used for fencing and fencing based on the
>> latter
>> >> is
>> >> >> >> >> stronger.
>> >> >> >> >>
>> >> >> >> >> JR68. Could we make the usage of topicId and topicName in the
>> >> RPCs
>> >> >> >> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> >> >> >> >> BumpLeaderEpochsReponse uses TopicName.
>> >> >> >> >>
>> >> >> >> >> JR69. MIRROR_PID_RESET:
>> >> >> >> >> JR69.1 Could you specify the value of the control record's
>> type?
>> >> >> >> >> JR69.2 How is the SourceClusterId field used?
>> >> >> >> >>
>> >> >> >> >> JR70. MirrorPartitionState: type should be "data" not
>> >> >> >> "coordinator-key",
>> >> >> >> >> right? Ditto for LastMirrorEpochs.
>> >> >> >> >>
>> >> >> >> >> Jun
>> >> >> >> >>
>> >> >> >> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com>
>> >> >> wrote:
>> >> >> >> >>
>> >> >> >> >> > Hi Jun,
>> >> >> >> >> >
>> >> >> >> >> > Thanks for the comments.
>> >> >> >> >> > Answering them below.
>> >> >> >> >> >
>> >> >> >> >> > > JR36. Sounds good. Should we remove
>> >> >> >> >> MirrorLeaderEpochExceededException?
>> >> >> >> >> >
>> >> >> >> >> > Sorry, it’s a confusing name. The
>> >> >> MirrorLeaderEpochExceededException
>> >> >> >> is
>> >> >> >> >> > used and thrown when the fetched mirror batch leader epoch >
>> >> local
>> >> >> >> >> leader
>> >> >> >> >> > epoch in destination cluster. And then we’ll stop mirroring
>> and
>> >> >> move
>> >> >> >> the
>> >> >> >> >> > partition to EPOCH_FENSING state to wait for local leader
>> epoch
>> >> >> >> bump. I
>> >> >> >> >> > think since we don’t have the concept of `MirrorLeaderEpoch`
>> >> >> anymore,
>> >> >> >> >> the
>> >> >> >> >> > `MirrorLeaderEpochExceededException` should not cause
>> confusion
>> >> >> now.
>> >> >> >> >> I’ll
>> >> >> >> >> > keep the name for the exception.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR51. Sounds good. Do we plan to always set
>> lastFetchedEpoch
>> >> in
>> >> >> the
>> >> >> >> >> > mirror
>> >> >> >> >> > fetch request issued to the source cluster, independent of
>> the
>> >> >> value
>> >> >> >> of
>> >> >> >> >> > mirror.support.unclean.leader.election? It's needed to
>> detect
>> >> >> >> divergence
>> >> >> >> >> > during failover. However, if it's also set when
>> >> >> >> >> > mirror.support.unclean.leader.election=false,
>> >> >> >> >> > an unclean leader election in the source cluster can halt
>> the
>> >> >> >> mirroring
>> >> >> >> >> > since we don’t do anything with the returned divergeEpoch in
>> >> that
>> >> >> >> mode
>> >> >> >> >> > and continue
>> >> >> >> >> > the next fetch.
>> >> >> >> >> >
>> >> >> >> >> > > Do we plan to always set lastFetchedEpoch in the mirror
>> >> >> >> >> > fetch request issued to the source cluster, independent of
>> the
>> >> >> value
>> >> >> >> of
>> >> >> >> >> > Mirror.support.unclean.leader.election?
>> >> >> >> >> >
>> >> >> >> >> > Yes, we always set lastFetchedEpoch in the mirror fetch
>> >> request.
>> >> >> >> >> >
>> >> >> >> >> > > However, if it's also set when
>> >> >> >> >> > mirror.support.unclean.leader.election=false,
>> >> >> >> >> > an unclean leader election in the source cluster can halt
>> the
>> >> >> >> mirroring
>> >> >> >> >> > since we don’t do anything with the returned divergeEpoch in
>> >> that
>> >> >> >> mode
>> >> >> >> >> > and continue the next fetch.
>> >> >> >> >> >
>> >> >> >> >> > Sorry I didn’t make it clear. When
>> >> >> >> >> > “mirror.support.unclean.leader.election=false” and a
>> >> >> “divergeEpoch”
>> >> >> >> is
>> >> >> >> >> set
>> >> >> >> >> > in fetch response, the leader node will handle the
>> divergeEpoch
>> >> >> and
>> >> >> >> then
>> >> >> >> >> > continue the next fetch as usual. The difference with
>> >> >> >> >> > “mirror.support.unclean.leader.election=true” is that
>> “should
>> >> we
>> >> >> wait
>> >> >> >> >> for
>> >> >> >> >> > ALL replicas caught up (complete truncation)”. Without the
>> >> >> waiting,
>> >> >> >> it’s
>> >> >> >> >> > possible the inconsistent log issue in KAFKA-18723 could
>> happen
>> >> >> >> between
>> >> >> >> >> > leader and follower nodes in the destination cluster
>> because we
>> >> >> don’t
>> >> >> >> >> have
>> >> >> >> >> > “MirrorLeaderEpoch” concept now to reject higher leader
>> epoch
>> >> >> >> batches.
>> >> >> >> >> That
>> >> >> >> >> > is, we won't halt the mirroring if "divergeEpoch" is set but
>> >> >> >> >> > “mirror.support.unclean.leader.election=false”.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR55. The DescribeClusterMirrorsRequest still returns both
>> >> the
>> >> >> >> mirror
>> >> >> >> >> > partition state and the mirror partition leader state. The
>> >> >> problem is
>> >> >> >> >> that
>> >> >> >> >> > if a caller only needs one of the state, the other state
>> >> becomes
>> >> >> >> >> redundant.
>> >> >> >> >> > For example, when starting the mirror, to obtain LME, we
>> only
>> >> need
>> >> >> >> the
>> >> >> >> >> > mirror partition state. Here is an alternative design: We
>> >> expose
>> >> >> the
>> >> >> >> >> mirror
>> >> >> >> >> > partition state through ReadMirrorState. The client will use
>> >> the
>> >> >> >> >> > FindCoordinatorRequest
>> >> >> >> >> > to find the right mirror partition coordinator to issue that
>> >> >> >> request. We
>> >> >> >> >> > expose the mirror partition leader state through the
>> >> >> >> >> ReadClusterMirrorLags
>> >> >> >> >> > request. The client can use the metadata request to find the
>> >> >> >> partition
>> >> >> >> >> > leader to issue that request.
>> >> >> >> >> >
>> >> >> >> >> > > In the describing mirror flow, the admin client first
>> issues
>> >> the
>> >> >> >> >> > ListMirrorRequest to find all the mirrors and the topics
>> being
>> >> >> >> >> mirrored. It
>> >> >> >> >> > then issues the metadata request to find the partitions for
>> >> those
>> >> >> >> topics
>> >> >> >> >> > and issues a FindCoordinatorRequest to find the mirror
>> >> partition
>> >> >> >> state
>> >> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the
>> mirror
>> >> >> >> partition
>> >> >> >> >> > state coordinator and ReadClusterMirrorLags to the partition
>> >> >> leader.
>> >> >> >> >> >
>> >> >> >> >> > > In the starting mirror flow, the partition leader in the
>> >> >> >> destination
>> >> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name
>> >> based
>> >> >> on
>> >> >> >> its
>> >> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to find
>> the
>> >> >> >> mirror
>> >> >> >> >> > partition state coordinators. Finally, it issues
>> >> ReadMirrorState
>> >> >> to
>> >> >> >> the
>> >> >> >> >> > mirror partition state coordinator to get the LME.
>> >> >> >> >> >
>> >> >> >> >> > > We can also improve the ListMirror request to accept a
>> filter
>> >> >> >> based on
>> >> >> >> >> > mirror name, source cluster ID and topic name.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Answers:
>> >> >> >> >> >
>> >> >> >> >> > > The DescribeClusterMirrorsRequest still returns both the
>> >> mirror
>> >> >> >> >> > partition state and the mirror partition leader state. The
>> >> >> problem is
>> >> >> >> >> that
>> >> >> >> >> > if a caller only needs one of the state, the other state
>> >> becomes
>> >> >> >> >> redundant.
>> >> >> >> >> > For example, when starting the mirror, to obtain LME, we
>> only
>> >> need
>> >> >> >> the
>> >> >> >> >> > mirror partition state.
>> >> >> >> >> >
>> >> >> >> >> > Good point!
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > Here is an alternative design: We expose the mirror
>> >> >> >> >> > partition state through ReadMirrorState. The client will use
>> >> the
>> >> >> >> >> > FindCoordinatorRequest
>> >> >> >> >> > to find the right mirror partition coordinator to issue that
>> >> >> >> request. We
>> >> >> >> >> > expose the mirror partition leader state through the
>> >> >> >> >> ReadClusterMirrorLags
>> >> >> >> >> > request. The client can use the metadata request to find the
>> >> >> >> partition
>> >> >> >> >> > leader to issue that request.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Sounds good. But we try not to expose
>> >> >> >> >> > ReadMirrorState/ReadClusterMirrorLags directly to the admin
>> >> client
>> >> >> >> >> because
>> >> >> >> >> > users might get confused about these 2 requests, and they
>> are
>> >> also
>> >> >> >> both
>> >> >> >> >> > related to describeClusterMirror. I think we can add 2
>> >> additional
>> >> >> >> >> fields to
>> >> >> >> >> > describeClusterMirrorRequest: “includeMirrorStates” and
>> >> >> >> >> > “includeMirrorLags”. So the admin client can include either
>> >> one or
>> >> >> >> both
>> >> >> >> >> of
>> >> >> >> >> > them in the DescrbieClusterMirrorRequest based on the
>> provided
>> >> >> >> options.
>> >> >> >> >> KIP
>> >> >> >> >> > is updated.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > In the describing mirror flow, the admin client first
>> issues
>> >> the
>> >> >> >> >> > ListMirrorRequest to find all the mirrors and the topics
>> being
>> >> >> >> >> mirrored. It
>> >> >> >> >> > then issues the metadata request to find the partitions for
>> >> those
>> >> >> >> topics
>> >> >> >> >> > and issues a FindCoordinatorRequest to find the mirror
>> >> partition
>> >> >> >> state
>> >> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the
>> mirror
>> >> >> >> partition
>> >> >> >> >> > state coordinator and ReadClusterMirrorLags to the partition
>> >> >> leader.
>> >> >> >> >> >
>> >> >> >> >> > Makes sense. KIP is updated.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > In the starting mirror flow, the partition leader in the
>> >> >> >> destination
>> >> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name
>> >> based
>> >> >> on
>> >> >> >> its
>> >> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to find
>> the
>> >> >> >> mirror
>> >> >> >> >> > partition state coordinators. Finally, it issues
>> >> ReadMirrorState
>> >> >> to
>> >> >> >> the
>> >> >> >> >> > mirror partition state coordinator to get the LME.
>> >> >> >> >> >
>> >> >> >> >> > Make sense. So we can send DescribeClusterMirrorRequest with
>> >> >> >> >> > “includeMirrorStates” only.
>> >> >> >> >> >
>> >> >> >> >> > > We can also improve the ListMirror request to accept a
>> filter
>> >> >> >> based on
>> >> >> >> >> > mirror name, source cluster ID and topic name.
>> >> >> >> >> >
>> >> >> >> >> > Good suggestion, just like what ListGroupsRequest does.
>> >> >> >> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
>> >> >> >> >> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest.
>> >> KIP
>> >> >> >> >> updated.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR56. Could you list the ACL needed for each new RPC?
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Yes, it’s already listed in “Security Control” section.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR57. In the command workflow, it sometimes says "The user
>> >> >> sends"
>> >> >> >> and
>> >> >> >> >> > some
>> >> >> >> >> > other times says "The admin client sends". It's better to
>> use
>> >> "The
>> >> >> >> admin
>> >> >> >> >> > client" consistently in all those places.
>> >> >> >> >> >
>> >> >> >> >> > Agree. KIP updated.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Thank you,
>> >> >> >> >> > Luke
>> >> >> >> >> >
>> >> >> >> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <
>> >> >> dev@kafka.apache.org
>> >> >> >> >
>> >> >> >> >> > wrote:
>> >> >> >> >> >
>> >> >> >> >> >> Hi, Luke,
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks for the reply.
>> >> >> >> >> >>
>> >> >> >> >> >> JR36. Sounds good. Should we remove
>> >> >> >> MirrorLeaderEpochExceededException?
>> >> >> >> >> >>
>> >> >> >> >> >> JR51. Sounds good. Do we plan to always set
>> lastFetchedEpoch
>> >> in
>> >> >> the
>> >> >> >> >> mirror
>> >> >> >> >> >> fetch request issued to the source cluster, independent of
>> the
>> >> >> >> value of
>> >> >> >> >> >> mirror.support.unclean.leader.election? It's needed to
>> detect
>> >> >> >> >> divergence
>> >> >> >> >> >> during failover. However, if it's also set when
>> >> >> >> >> >> mirror.support.unclean.leader.election=false,
>> >> >> >> >> >> an unclean leader election in the source cluster can halt
>> the
>> >> >> >> mirroring
>> >> >> >> >> >> since we don’t do anything with the returned divergeEpoch
>> in
>> >> that
>> >> >> >> mode
>> >> >> >> >> >> and continue
>> >> >> >> >> >> the next fetch.
>> >> >> >> >> >>
>> >> >> >> >> >> JR55. The DescribeClusterMirrorsRequest still returns both
>> the
>> >> >> >> mirror
>> >> >> >> >> >> partition state and the mirror partition leader state. The
>> >> >> problem
>> >> >> >> is
>> >> >> >> >> that
>> >> >> >> >> >> if a caller only needs one of the state, the other state
>> >> becomes
>> >> >> >> >> >> redundant.
>> >> >> >> >> >> For example, when starting the mirror, to obtain LME, we
>> only
>> >> >> need
>> >> >> >> the
>> >> >> >> >> >> mirror partition state. Here is an alternative design: We
>> >> expose
>> >> >> the
>> >> >> >> >> >> mirror
>> >> >> >> >> >> partition state through ReadMirrorState. The client will
>> use
>> >> the
>> >> >> >> >> >> FindCoordinatorRequest
>> >> >> >> >> >> to find the right mirror partition coordinator to issue
>> that
>> >> >> >> request.
>> >> >> >> >> We
>> >> >> >> >> >> expose the mirror partition leader state through the
>> >> >> >> >> ReadClusterMirrorLags
>> >> >> >> >> >> request. The client can use the metadata request to find
>> the
>> >> >> >> partition
>> >> >> >> >> >> leader to issue that request.
>> >> >> >> >> >>
>> >> >> >> >> >> In the describing mirror flow, the admin client first
>> issues
>> >> the
>> >> >> >> >> >> ListMirrorRequest to find all the mirrors and the topics
>> being
>> >> >> >> >> mirrored.
>> >> >> >> >> >> It
>> >> >> >> >> >> then issues the metadata request to find the partitions for
>> >> those
>> >> >> >> >> topics
>> >> >> >> >> >> and issues a FindCoordinatorRequest to find the mirror
>> >> partition
>> >> >> >> state
>> >> >> >> >> >> coordinators. Finally, it issues ReadMirrorState to the
>> mirror
>> >> >> >> >> partition
>> >> >> >> >> >> state coordinator and ReadClusterMirrorLags to the
>> partition
>> >> >> leader.
>> >> >> >> >> >>
>> >> >> >> >> >> In the starting mirror flow, the partition leader in the
>> >> >> destination
>> >> >> >> >> >> cluster issues a ListMirrorRequest to find the mirror name
>> >> based
>> >> >> on
>> >> >> >> its
>> >> >> >> >> >> cluster id. It then issues a FindCoordinatorRequest to find
>> >> the
>> >> >> >> mirror
>> >> >> >> >> >> partition state coordinators. Finally, it issues
>> >> ReadMirrorState
>> >> >> to
>> >> >> >> the
>> >> >> >> >> >> mirror partition state coordinator to get the LME.
>> >> >> >> >> >>
>> >> >> >> >> >> We can also improve the ListMirror request to accept a
>> filter
>> >> >> based
>> >> >> >> on
>> >> >> >> >> >> mirror name, source cluster ID and topic name.
>> >> >> >> >> >>
>> >> >> >> >> >> JR56. Could you list the ACL needed for each new RPC?
>> >> >> >> >> >>
>> >> >> >> >> >> JR57. In the command workflow, it sometimes says "The user
>> >> sends"
>> >> >> >> and
>> >> >> >> >> some
>> >> >> >> >> >> other times says "The admin client sends". It's better to
>> use
>> >> >> "The
>> >> >> >> >> admin
>> >> >> >> >> >> client" consistently in all those places.
>> >> >> >> >> >>
>> >> >> >> >> >> Jun
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <
>> showuon@gmail.com>
>> >> >> wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> > Hi Jun,
>> >> >> >> >> >> >
>> >> >> >> >> >> > Thanks for your comments.
>> >> >> >> >> >> > Answering them below.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch”
>> is
>> >> >> >> >> specifically
>> >> >> >> >> >> > designed for the unclean leader election support in
>> Cluster
>> >> >> >> Mirroring
>> >> >> >> >> >> > because the unclean leader election will break the
>> >> assumption
>> >> >> that
>> >> >> >> >> the
>> >> >> >> >> >> > committed data won’t be truncated."
>> >> >> >> >> >> > If the source cluster has an unclean leader election, the
>> >> >> >> destination
>> >> >> >> >> >> > cluster will detect the inconsistency and truncate the
>> log
>> >> at
>> >> >> the
>> >> >> >> >> >> > destination leader. We can just
>> >> >> >> >> >> > extend mirror.support.unclean.leader.election to cover
>> this
>> >> >> case,
>> >> >> >> >> i.e.,
>> >> >> >> >> >> we
>> >> >> >> >> >> > wait until all replicas' log end offsets are reduced to
>> the
>> >> >> target
>> >> >> >> >> >> offset
>> >> >> >> >> >> > before resuming mirroring. This avoids the inconsistency.
>> >> >> >> >> >> >
>> >> >> >> >> >> > If the destination cluster has an unclean leader
>> election,
>> >> it
>> >> >> will
>> >> >> >> >> >> > re-fetch the same data from the source. There won't be
>> any
>> >> >> >> >> inconsistency
>> >> >> >> >> >> > either.
>> >> >> >> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > Yes, that works, too. I agree this is better because the
>> >> >> unclean
>> >> >> >> >> leader
>> >> >> >> >> >> > election is rare, we don't want to occupy a field just
>> for
>> >> it.
>> >> >> >> >> >> >
>> >> >> >> >> >> > So to achieve this, we have to:
>> >> >> >> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
>> >> >> >> >> >> > 2. When the destination cluster leader completes a log
>> >> >> truncation
>> >> >> >> >> after
>> >> >> >> >> >> a
>> >> >> >> >> >> > fetch response due to divergeEpoch, we know it must be
>> >> caused
>> >> >> by
>> >> >> >> >> unclean
>> >> >> >> >> >> > leader election in the source cluster because we only
>> fetch
>> >> >> >> committed
>> >> >> >> >> >> > records. If
>> “mirror.support.unclean.leader.election=false”,
>> >> we
>> >> >> >> don’t
>> >> >> >> >> do
>> >> >> >> >> >> > anything and continue the next fetch. If
>> >> >> >> >> >> > “mirror.support.unclean.leader.election=true”, we remove
>> the
>> >> >> >> mirror
>> >> >> >> >> >> fetcher
>> >> >> >> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
>> >> >> >> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL
>> replicas
>> >> to
>> >> >> be
>> >> >> >> >> caught
>> >> >> >> >> >> up
>> >> >> >> >> >> > into ISR and reduced to the expected LEO.
>> >> >> >> >> >> > 4. Move to “MIRRORING” state and create the mirror
>> fetcher
>> >> >> >> >> >> >
>> >> >> >> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in
>> Fetch
>> >> >> API.
>> >> >> >> >> >> > KIP is updated.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR51. Make sense. To truncate based on LME, we need to
>> >> issue
>> >> >> >> >> another
>> >> >> >> >> >> RPC
>> >> >> >> >> >> > to translate the LME to an offset. It would be useful to
>> >> >> include
>> >> >> >> >> that in
>> >> >> >> >> >> > the workflow description.
>> >> >> >> >> >> >
>> >> >> >> >> >> > About the additional RPC to translate the LME to the
>> offset
>> >> >> (i.e.
>> >> >> >> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on
>> this
>> >> >> to do
>> >> >> >> >> the
>> >> >> >> >> >> > offset translation. We use the local leader epoch cache
>> in
>> >> the
>> >> >> >> >> >> destination
>> >> >> >> >> >> > cluster to translate the epoch to save the RPC call. This
>> >> works
>> >> >> >> >> because
>> >> >> >> >> >> of
>> >> >> >> >> >> > an invariant the LME truncation protocol maintains: for
>> any
>> >> >> leader
>> >> >> >> >> >> epoch ≤
>> >> >> >> >> >> > LME, that epoch's data belongs to exactly one cluster,
>> >> either
>> >> >> it
>> >> >> >> >> >> originated
>> >> >> >> >> >> > there, or it was mirrored. This holds even across
>> repeated
>> >> >> mirror
>> >> >> >> ->
>> >> >> >> >> >> > reverse-mirror -> mirror cycles, since each direction's
>> >> >> >> >> LOG_TRUNCATION
>> >> >> >> >> >> > phase re-establishes the invariant before that cluster
>> >> starts
>> >> >> >> >> mirroring.
>> >> >> >> >> >> > With this invariant, the replication protocol can find
>> out
>> >> the
>> >> >> >> >> diverge
>> >> >> >> >> >> > point and truncate the log correctly when mirroring. For
>> >> >> example:
>> >> >> >> >> >> >
>> >> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> >> >> > Offset 2: A (LE: 0)
>> >> >> >> >> >> > Offset 3: A (LE: 1)
>> >> >> >> >> >> >
>> >> >> >> >> >> > Cluster A crashes while Cluster B mirrors to offset 1.
>> >> Failover
>> >> >> >> to B
>> >> >> >> >> and
>> >> >> >> >> >> > store the LME=0 in cluster B. When cluster A recovers and
>> >> >> starts
>> >> >> >> the
>> >> >> >> >> >> > reverse mirroring, it’ll query the LME from B and get
>> back
>> >> >> LME: 0.
>> >> >> >> >> Then
>> >> >> >> >> >> > cluster A gets the end offset for epoch 0 from
>> >> >> leaderEpochCache,
>> >> >> >> and
>> >> >> >> >> >> gets
>> >> >> >> >> >> > offset 2. So offset 3 is truncated. The log is like this
>> >> now:
>> >> >> >> >> >> >
>> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE:
>> 10)
>> >> >> >> >> >> > Offset 3: B
>> (LE:
>> >> >> 10)
>> >> >> >> >> >> >
>> >> >> >> >> >> > When mirror starts, cluster A sends the fetch request
>> with {
>> >> >> >> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the
>> >> >> divergingEpoch
>> >> >> >> >> check
>> >> >> >> >> >> in
>> >> >> >> >> >> > the cluster B queries the endOffset for leader epoch 0,
>> and
>> >> >> found
>> >> >> >> the
>> >> >> >> >> >> end
>> >> >> >> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so
>> >> returns
>> >> >> >> fetch
>> >> >> >> >> >> > response with divergingEpoch info to cluster A for log
>> >> >> >> truncation. In
>> >> >> >> >> >> the
>> >> >> >> >> >> > end, offset 2 in cluster A will be truncated and then log
>> >> >> >> converges
>> >> >> >> >> with
>> >> >> >> >> >> > cluster B.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a
>> bit
>> >> >> >> strange
>> >> >> >> >> >> > because it mixes querying the mirror partition state with
>> >> >> querying
>> >> >> >> >> the
>> >> >> >> >> >> > mirror partition leader. Another option is to use
>> >> >> >> >> >> ReadMirrorStatesRequest
>> >> >> >> >> >> > to query the mirror partition state and a new request to
>> >> query
>> >> >> the
>> >> >> >> >> >> mirror
>> >> >> >> >> >> > partition leader. If we want the adminClient to query the
>> >> right
>> >> >> >> >> >> ClusterMirrorCoordinator
>> >> >> >> >> >> > directly, we can extend the FindCoordinatorRequest to
>> >> support
>> >> >> >> that.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > Good suggestion. Yes, we need to fan out to all brokers
>> >> >> because we
>> >> >> >> >> have
>> >> >> >> >> >> to
>> >> >> >> >> >> > collect info from mirror partition leaders and mirror
>> >> >> >> coordinators.
>> >> >> >> >> If
>> >> >> >> >> >> we
>> >> >> >> >> >> > create a new API to contact mirror partition leaders, it
>> >> will
>> >> >> make
>> >> >> >> >> the
>> >> >> >> >> >> API
>> >> >> >> >> >> > cleaner and clearer. So we will do this:
>> >> >> >> >> >> >
>> >> >> >> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields
>> >> from
>> >> >> >> >> >> > DescribeClusterMirror request/response
>> >> >> >> >> >> >
>> >> >> >> >> >> > b. Add a new API: ReadClusterMirrorLags API to request
>> for
>> >> >> >> >> SourceOffset,
>> >> >> >> >> >> > DestinationOffset, and Lag info.
>> >> >> >> >> >> >
>> >> >> >> >> >> > The workflow for admin client describeClusterMirror will
>> be
>> >> >> like
>> >> >> >> >> this:
>> >> >> >> >> >> > 1. adminClient sends describeClusterMirror to the
>> >> >> leastLoadedNode
>> >> >> >> >> broker
>> >> >> >> >> >> > 2. The received broker collects the current mirrored
>> >> >> partitions,
>> >> >> >> and
>> >> >> >> >> >> > groups them by coordinator node and by partition leader
>> >> node.
>> >> >> >> >> >> > 3. The broker sends ReadMirrorState request to the
>> >> coordinator
>> >> >> >> nodes
>> >> >> >> >> to
>> >> >> >> >> >> > get the mirror state related info
>> >> >> >> >> >> > 4. The broker sends ReadClusterMirrorLags request to the
>> >> mirror
>> >> >> >> >> >> partition
>> >> >> >> >> >> > leader nodes to get the mirror lag, source offset,
>> >> destination
>> >> >> >> offset
>> >> >> >> >> >> > (which updates on every fetch response).
>> >> >> >> >> >> > 5. The broker merge all the response into
>> >> describeClusterMirror
>> >> >> >> >> response
>> >> >> >> >> >> > and send to admin client
>> >> >> >> >> >> >
>> >> >> >> >> >> > KIP is updated. Thanks for the suggestion.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR55. For __mirror_state topic, could we describe
>> whether
>> >> it
>> >> >> is
>> >> >> >> >> >> > compacted?
>> >> >> >> >> >> >
>> >> >> >> >> >> > Yes, it is a compacted topic. I made it much clearer in
>> the
>> >> >> >> >> >> > “ClusterMirrorCoordinator” section.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > Thank you,
>> >> >> >> >> >> > Luke
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
>> >> >> >> dev@kafka.apache.org
>> >> >> >> >> >
>> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >
>> >> >> >> >> >> >> Hi, Luke,
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> Thanks for the reply.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> >> >> >> >> specifically
>> >> >> >> >> >> >> designed for the unclean leader election support in
>> Cluster
>> >> >> >> >> Mirroring
>> >> >> >> >> >> >> because the unclean leader election will break the
>> >> assumption
>> >> >> >> that
>> >> >> >> >> the
>> >> >> >> >> >> >> committed data won’t be truncated."
>> >> >> >> >> >> >> If the source cluster has an unclean leader election,
>> the
>> >> >> >> >> destination
>> >> >> >> >> >> >> cluster will detect the inconsistency and truncate the
>> log
>> >> at
>> >> >> the
>> >> >> >> >> >> >> destination leader. We can just
>> >> >> >> >> >> >> extend mirror.support.unclean.leader.election to cover
>> this
>> >> >> case,
>> >> >> >> >> >> i.e., we
>> >> >> >> >> >> >> wait until all replicas' log end offsets are reduced to
>> the
>> >> >> >> target
>> >> >> >> >> >> offset
>> >> >> >> >> >> >> before resuming mirroring. This avoids the
>> inconsistency.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> If the destination cluster has an unclean leader
>> election,
>> >> it
>> >> >> >> will
>> >> >> >> >> >> >> re-fetch
>> >> >> >> >> >> >> the same data from the source. There won't be any
>> >> >> inconsistency
>> >> >> >> >> either.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR51. Make sense. To truncate based on LME, we need to
>> >> issue
>> >> >> >> another
>> >> >> >> >> >> RPC
>> >> >> >> >> >> >> to
>> >> >> >> >> >> >> translate the LME to an offset. It would be useful to
>> >> include
>> >> >> >> that
>> >> >> >> >> in
>> >> >> >> >> >> the
>> >> >> >> >> >> >> workflow description.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a
>> bit
>> >> >> strange
>> >> >> >> >> >> because
>> >> >> >> >> >> >> it mixes querying the mirror partition state with
>> querying
>> >> the
>> >> >> >> >> mirror
>> >> >> >> >> >> >> partition leader. Another option is to use
>> >> >> >> ReadMirrorStatesRequest
>> >> >> >> >> to
>> >> >> >> >> >> >> query
>> >> >> >> >> >> >> the mirror partition state and a new request to query
>> the
>> >> >> mirror
>> >> >> >> >> >> partition
>> >> >> >> >> >> >> leader. If we want the adminClient to query the right
>> >> >> >> >> >> >> ClusterMirrorCoordinator
>> >> >> >> >> >> >> directly, we can extend the FindCoordinatorRequest to
>> >> support
>> >> >> >> that.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR55. For __mirror_state topic, could we describe
>> whether
>> >> it
>> >> >> is
>> >> >> >> >> >> compacted?
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> Jun
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <
>> >> showuon@gmail.com>
>> >> >> >> wrote:
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> > Hi Jun,
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Thanks for the comments.
>> >> >> >> >> >> >> > Answering them below:
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR31.3 It makes sense for the new partition leader
>> to
>> >> >> query
>> >> >> >> the
>> >> >> >> >> >> >> current
>> >> >> >> >> >> >> > mirror state from the local or remote
>> >> MirrorMetadataManager
>> >> >> >> that
>> >> >> >> >> owns
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > partition's state. However, the wording in the KIP is
>> >> >> >> confusing.
>> >> >> >> >> >> >> > MirrorMetadataManager
>> >> >> >> >> >> >> > mirror partition state cache should be populated only
>> by
>> >> one
>> >> >> >> way,
>> >> >> >> >> >> which
>> >> >> >> >> >> >> is
>> >> >> >> >> >> >> > replaying __mirror_state partitions this broker
>> leads. By
>> >> >> >> saying
>> >> >> >> >> >> "and by
>> >> >> >> >> >> >> > fetching state from remote coordinators via
>> >> ReadMirrorStates
>> >> >> >> >> RPCs.",
>> >> >> >> >> >> >> it's
>> >> >> >> >> >> >> > as if the mirror partition state cache on each broker
>> >> stores
>> >> >> >> the
>> >> >> >> >> >> state
>> >> >> >> >> >> >> for
>> >> >> >> >> >> >> > all mirror partitions.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > You’re right. KIP is updated to make it clear.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR36. I understand the intention now, but I still
>> don't
>> >> >> >> think we
>> >> >> >> >> >> need
>> >> >> >> >> >> >> > MirrorLeaderEpoch. Let's
>> >> >> >> >> >> >> > assume the common case: no unclean leader election in
>> the
>> >> >> >> source
>> >> >> >> >> >> cluster
>> >> >> >> >> >> >> > during mirroring. Mirroring only fetches the committed
>> >> data
>> >> >> >> from
>> >> >> >> >> the
>> >> >> >> >> >> >> > source, which never changes if obtained a second
>> time. In
>> >> >> your
>> >> >> >> >> >> example
>> >> >> >> >> >> >> > image, follower broker 1 in the destination cluster
>> can
>> >> >> never
>> >> >> >> >> obtain
>> >> >> >> >> >> the
>> >> >> >> >> >> >> > record B(0) because it's never committed in the source
>> >> and
>> >> >> >> thus is
>> >> >> >> >> >> never
>> >> >> >> >> >> >> > exposed to the destination leader. It's possible for
>> >> broker
>> >> >> 1
>> >> >> >> to
>> >> >> >> >> >> first
>> >> >> >> >> >> >> get
>> >> >> >> >> >> >> > D(1) which is truncated later when broker 0 follows a
>> new
>> >> >> >> leader.
>> >> >> >> >> But
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > new leader will refetch the same D(1) from the source
>> >> >> cluster.
>> >> >> >> So,
>> >> >> >> >> >> it's
>> >> >> >> >> >> >> ok
>> >> >> >> >> >> >> > for broker 1 to keep D(1).
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Yes, you are correct. Because mirroring only fetches
>> the
>> >> >> >> committed
>> >> >> >> >> >> data
>> >> >> >> >> >> >> > from the source, the MirrorLeaderEpoch is unnecessary
>> for
>> >> >> >> normal
>> >> >> >> >> >> >> mirroring.
>> >> >> >> >> >> >> > Thanks for the clarification. So I think we can say,
>> the
>> >> >> >> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the
>> >> unclean
>> >> >> >> >> leader
>> >> >> >> >> >> >> > election support in Cluster Mirroring because the
>> unclean
>> >> >> >> leader
>> >> >> >> >> >> >> election
>> >> >> >> >> >> >> > will break the assumption that the committed data
>> won’t
>> >> be
>> >> >> >> >> truncated.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Given that, we could gate the MirrorLeaderEpoch field
>> by
>> >> >> only
>> >> >> >> >> >> setting it
>> >> >> >> >> >> >> > in the Fetch request when
>> >> >> >> >> >> "mirror.support.unclean.leader.election=true"
>> >> >> >> >> >> >> > (false by default), and skip it entirely for normal
>> >> >> mirroring.
>> >> >> >> >> What
>> >> >> >> >> >> do
>> >> >> >> >> >> >> you
>> >> >> >> >> >> >> > think?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR51. Sounds good. Why do we choose to store the
>> last
>> >> >> mirror
>> >> >> >> >> epoch
>> >> >> >> >> >> >> > instead
>> >> >> >> >> >> >> > of a last mirror offset?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Yes, originally we stored the “last mirror offset”
>> >> instead
>> >> >> of
>> >> >> >> >> “last
>> >> >> >> >> >> >> mirror
>> >> >> >> >> >> >> > epoch”. But later, we found it doesn’t work for
>> unclean
>> >> >> leader
>> >> >> >> >> >> election
>> >> >> >> >> >> >> > cases. (yes, `unclean leader election` again :) ) For
>> >> >> example:
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Cluster B is mirroring from A. And A crashes and
>> >> failover to
>> >> >> >> B. B
>> >> >> >> >> >> stores
>> >> >> >> >> >> >> > the “last mirror offset” as 1.
>> >> >> >> >> >> >> > When cluster A recovers, somehow the unclean leader
>> >> election
>> >> >> >> >> happens,
>> >> >> >> >> >> >> log
>> >> >> >> >> >> >> > is empty in the new leader, and then some more
>> leadership
>> >> >> >> changes
>> >> >> >> >> to
>> >> >> >> >> >> >> make
>> >> >> >> >> >> >> > the leader epoch become 10. Then it starts the reverse
>> >> >> mirror
>> >> >> >> >> from B.
>> >> >> >> >> >> >> The
>> >> >> >> >> >> >> > log looks like this:
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
>> >> >> >> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
>> >> >> >> >> >> >> > Offset 2:
>> >> >> pid_res
>> >> >> >> (LE:
>> >> >> >> >> >> 10)
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > When A starts mirroring from B, log is truncated to
>> last
>> >> >> mirror
>> >> >> >> >> >> offset:
>> >> >> >> >> >> >> 1
>> >> >> >> >> >> >> > (no-op), and then fetch from cluster B. The 1st fetch
>> >> >> request
>> >> >> >> is
>> >> >> >> >> like
>> >> >> >> >> >> >> this:
>> >> >> >> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10,
>> >> fetchOffset:
>> >> >> >> 2, …
>> >> >> >> >> }.
>> >> >> >> >> >> The
>> >> >> >> >> >> >> > divergingEpoch check in the cluster B queries the
>> >> endOffset
>> >> >> for
>> >> >> >> >> >> leader
>> >> >> >> >> >> >> > epoch 10 (in Partition#readRecords), and found the end
>> >> >> offset
>> >> >> >> for
>> >> >> >> >> LE
>> >> >> >> >> >> 10
>> >> >> >> >> >> >> is
>> >> >> >> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch
>> >> >> validation
>> >> >> >> >> >> passes
>> >> >> >> >> >> >> and
>> >> >> >> >> >> >> > returns offset 2 to cluster A. In short, relying on
>> “last
>> >> >> >> mirror
>> >> >> >> >> >> offset”
>> >> >> >> >> >> >> > cannot resolve the log diverge issue in unclean leader
>> >> >> election
>> >> >> >> >> case.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > If instead, we truncate until “last mirror epoch”: 0,
>> the
>> >> >> >> offset 0
>> >> >> >> >> >> and 1
>> >> >> >> >> >> >> > in cluster A will be truncated as expected and then
>> log
>> >> >> >> converge
>> >> >> >> >> with
>> >> >> >> >> >> >> > cluster B correctly in the end.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Also, about JR53.2:
>> >> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to
>> all
>> >> >> >> brokers
>> >> >> >> >> and
>> >> >> >> >> >> >> > merges
>> >> >> >> >> >> >> > the responses client-side."
>> >> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
>> >> >> partition.
>> >> >> >> So
>> >> >> >> >> >> sending
>> >> >> >> >> >> >> > the request to every broker is wasteful. It's probably
>> >> >> better
>> >> >> >> to
>> >> >> >> >> send
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > request to a single broker, which can then contact the
>> >> right
>> >> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > While working on this, I found we need to add one more
>> >> >> tagged
>> >> >> >> >> field
>> >> >> >> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest
>> schema
>> >> >> because
>> >> >> >> >> >> >> otherwise
>> >> >> >> >> >> >> > the receiving broker doesn’t know if this is the
>> request
>> >> >> from
>> >> >> >> >> client
>> >> >> >> >> >> >> and it
>> >> >> >> >> >> >> > needs to contact other brokers, or this is just a
>> >> forwarded
>> >> >> >> >> request.
>> >> >> >> >> >> The
>> >> >> >> >> >> >> > workflow is like this:
>> >> >> >> >> >> >> > 1. the adminClient sends the
>> >> DescribeClusterMirrorsRequest
>> >> >> to a
>> >> >> >> >> >> >> > leastLoaded broker, which doesn’t contain the
>> "Forwarded"
>> >> >> >> field.
>> >> >> >> >> >> >> > 2. The broker receives the request and sends
>> >> >> >> >> >> >> DescribeClusterMirrorsRequest
>> >> >> >> >> >> >> > with "Forwarded=true" to the responsible brokers and
>> >> waits
>> >> >> for
>> >> >> >> the
>> >> >> >> >> >> >> response
>> >> >> >> >> >> >> > async.
>> >> >> >> >> >> >> > 3. When all responses are received, merge the results
>> and
>> >> >> >> respond
>> >> >> >> >> to
>> >> >> >> >> >> the
>> >> >> >> >> >> >> > client.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here
>> because
>> >> it
>> >> >> >> >> doesn’t
>> >> >> >> >> >> >> > contain the lag info in the coordinator.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > The alternative solution is the adminClient sends
>> >> >> >> >> >> >> > DescribeClusterMirrorsRequest to the “responsible
>> brokers
>> >> >> only”
>> >> >> >> >> (via
>> >> >> >> >> >> >> > describeTopics), instead of ALL brokers, and merge the
>> >> >> results
>> >> >> >> in
>> >> >> >> >> the
>> >> >> >> >> >> >> > client side. This solution doesn’t need to add a new
>> >> field
>> >> >> in
>> >> >> >> the
>> >> >> >> >> >> >> > DescribeClusterMirrorsRequest and still can avoid
>> >> >> unnecessary
>> >> >> >> >> >> requests
>> >> >> >> >> >> >> > sent.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > My preference is to keep the merge logic on the admin
>> >> client
>> >> >> >> side,
>> >> >> >> >> >> which
>> >> >> >> >> >> >> > saves the overhead in the broker side.
>> >> >> >> >> >> >> > What do you think?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Thank you,
>> >> >> >> >> >> >> > Luke
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
>> >> >> >> >> dev@kafka.apache.org
>> >> >> >> >> >> >
>> >> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> Hi, Luke,
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> Thanks for the reply.
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR31.3 It makes sense for the new partition leader to
>> >> query
>> >> >> >> the
>> >> >> >> >> >> current
>> >> >> >> >> >> >> >> mirror state from the local or remote
>> >> MirrorMetadataManager
>> >> >> >> that
>> >> >> >> >> >> owns
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> partition's state. However, the wording in the KIP is
>> >> >> >> confusing.
>> >> >> >> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> >> >> mirror partition state cache should be populated
>> only by
>> >> >> one
>> >> >> >> way,
>> >> >> >> >> >> >> which is
>> >> >> >> >> >> >> >> replaying __mirror_state partitions this broker
>> leads.
>> >> By
>> >> >> >> saying
>> >> >> >> >> >> "and
>> >> >> >> >> >> >> by
>> >> >> >> >> >> >> >> fetching state from remote coordinators via
>> >> >> ReadMirrorStates
>> >> >> >> >> RPCs.",
>> >> >> >> >> >> >> it's
>> >> >> >> >> >> >> >> as if the mirror partition state cache on each broker
>> >> >> stores
>> >> >> >> the
>> >> >> >> >> >> state
>> >> >> >> >> >> >> for
>> >> >> >> >> >> >> >> all mirror partitions.
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR36. I understand the intention now, but I still
>> don't
>> >> >> think
>> >> >> >> we
>> >> >> >> >> >> need
>> >> >> >> >> >> >> >> MirrorLeaderEpoch. Let's
>> >> >> >> >> >> >> >> assume the common case: no unclean leader election in
>> >> the
>> >> >> >> source
>> >> >> >> >> >> >> cluster
>> >> >> >> >> >> >> >> during mirroring. Mirroring only fetches the
>> committed
>> >> data
>> >> >> >> from
>> >> >> >> >> the
>> >> >> >> >> >> >> >> source, which never changes if obtained a second
>> time.
>> >> In
>> >> >> your
>> >> >> >> >> >> example
>> >> >> >> >> >> >> >> image, follower broker 1 in the destination cluster
>> can
>> >> >> never
>> >> >> >> >> obtain
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> record B(0) because it's never committed in the
>> source
>> >> and
>> >> >> >> thus
>> >> >> >> >> is
>> >> >> >> >> >> >> never
>> >> >> >> >> >> >> >> exposed to the destination leader. It's possible for
>> >> >> broker 1
>> >> >> >> to
>> >> >> >> >> >> first
>> >> >> >> >> >> >> get
>> >> >> >> >> >> >> >> D(1) which is truncated later when broker 0 follows a
>> >> new
>> >> >> >> leader.
>> >> >> >> >> >> But
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> new leader will refetch the same D(1) from the source
>> >> >> cluster.
>> >> >> >> >> So,
>> >> >> >> >> >> >> it's ok
>> >> >> >> >> >> >> >> for broker 1 to keep D(1).
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR51. Sounds good. Why do we choose to store the last
>> >> >> mirror
>> >> >> >> >> epoch
>> >> >> >> >> >> >> instead
>> >> >> >> >> >> >> >> of a last mirror offset?
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> Jun
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <
>> >> >> showuon@gmail.com>
>> >> >> >> >> wrote:
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> > Hi Jun,
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Thanks for the great comments again.
>> >> >> >> >> >> >> >> > Answering your question below.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror
>> >> partition
>> >> >> >> cache
>> >> >> >> >> >> lives
>> >> >> >> >> >> >> in
>> >> >> >> >> >> >> >> > MirrorMetadataManager. It is populated in two
>> ways: by
>> >> >> >> >> replaying
>> >> >> >> >> >> >> >> > __mirror_state partitions this broker leads, and by
>> >> >> fetching
>> >> >> >> >> state
>> >> >> >> >> >> >> >> > from remote coordinators via ReadMirrorStates
>> RPCs."
>> >> >> >> >> >> >> >> > Hmm, a given broker only caches the mirror
>> partition
>> >> >> state
>> >> >> >> for
>> >> >> >> >> >> >> >> partitions
>> >> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted on
>> >> this
>> >> >> >> broker,
>> >> >> >> >> >> >> right?
>> >> >> >> >> >> >> >> Why
>> >> >> >> >> >> >> >> > does it need to fetch the state from other remote
>> >> >> >> coordinators?
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > a given broker only caches the mirror partition
>> >> state
>> >> >> for
>> >> >> >> >> >> >> partitions
>> >> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted on
>> >> this
>> >> >> >> broker,
>> >> >> >> >> >> >> right?
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Right.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > Why does it need to fetch the state from other
>> >> remote
>> >> >> >> >> >> coordinators?
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Because the data partition leader needs to know the
>> >> >> current
>> >> >> >> >> mirror
>> >> >> >> >> >> >> state
>> >> >> >> >> >> >> >> > of the partitions it leads. On a leadership change,
>> >> the
>> >> >> >> >> metadata
>> >> >> >> >> >> >> delta
>> >> >> >> >> >> >> >> is
>> >> >> >> >> >> >> >> > published to the MirrorMetadataManager, which
>> queries
>> >> the
>> >> >> >> >> current
>> >> >> >> >> >> >> mirror
>> >> >> >> >> >> >> >> > state for its led partitions from the coordinator
>> >> (local
>> >> >> or
>> >> >> >> >> >> remote),
>> >> >> >> >> >> >> and
>> >> >> >> >> >> >> >> > processes tasks based on that state. This is the
>> >> reason
>> >> >> why
>> >> >> >> it
>> >> >> >> >> >> >> fetches
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > state from other remote coordinator. This part of
>> >> cache
>> >> >> >> will be
>> >> >> >> >> >> >> cleared
>> >> >> >> >> >> >> >> > after the leadership change.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR36. I am still not quite sure why we need
>> >> >> >> >> MirrorLeaderEpoch in
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > Fetch
>> >> >> >> >> >> >> >> > request. In the current Fetch request, in addition
>> to
>> >> >> >> >> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> >> >> >> > it has another field LastFetchedEpoch, which
>> captures
>> >> the
>> >> >> >> last
>> >> >> >> >> >> leader
>> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> > in the data and is used to resolve the
>> inconsistency
>> >> >> between
>> >> >> >> >> the
>> >> >> >> >> >> >> >> follower
>> >> >> >> >> >> >> >> > and the leader. Does it achieve the same purpose as
>> >> >> >> >> >> >> MirrorLeaderEpoch?
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used
>> to
>> >> do
>> >> >> the
>> >> >> >> >> >> >> validation
>> >> >> >> >> >> >> >> to
>> >> >> >> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any
>> >> fetched
>> >> >> >> >> batches
>> >> >> >> >> >> with
>> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> > higher than the LastFetchedEpoch, it means the
>> >> follower
>> >> >> can
>> >> >> >> >> never
>> >> >> >> >> >> >> accept
>> >> >> >> >> >> >> >> > new appended batches with higher epoch in the
>> leader
>> >> >> node.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2
>> >> >> purposes in
>> >> >> >> >> the
>> >> >> >> >> >> >> fetch
>> >> >> >> >> >> >> >> > API:
>> >> >> >> >> >> >> >> > 1. The receiver (leader node) uses it to validate
>> it
>> >> >> matches
>> >> >> >> >> the
>> >> >> >> >> >> >> leader
>> >> >> >> >> >> >> >> > epoch in the leader node
>> >> >> >> >> >> >> >> > 2. The sender (follower node) uses it to validate
>> the
>> >> >> epoch
>> >> >> >> in
>> >> >> >> >> the
>> >> >> >> >> >> >> >> fetched
>> >> >> >> >> >> >> >> > batches are all <= this value.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still
>> >> serving
>> >> >> for
>> >> >> >> >> >> >> followers
>> >> >> >> >> >> >> >> in
>> >> >> >> >> >> >> >> > the destination cluster. For purpose (2), my
>> >> >> understanding
>> >> >> >> of
>> >> >> >> >> the
>> >> >> >> >> >> >> >> > validation for KAFKA-18723 is like: the follower
>> knows
>> >> >> the
>> >> >> >> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the
>> leader
>> >> node
>> >> >> >> >> should
>> >> >> >> >> >> >> >> contain in
>> >> >> >> >> >> >> >> > the log batches when fetch request is sent. Any log
>> >> batch
>> >> >> >> >> beyond
>> >> >> >> >> >> this
>> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> > in the fetch response means the follower’s
>> metadata is
>> >> >> >> stale.
>> >> >> >> >> The
>> >> >> >> >> >> >> >> batches
>> >> >> >> >> >> >> >> > need to be rejected and follower needs to update
>> >> metadata
>> >> >> >> for
>> >> >> >> >> next
>> >> >> >> >> >> >> >> fetch.
>> >> >> >> >> >> >> >> > In the Cluster Mirroring’s world, the
>> >> MirrorLeaderEpoch
>> >> >> >> serves
>> >> >> >> >> >> this
>> >> >> >> >> >> >> >> purpose
>> >> >> >> >> >> >> >> > to let the followers know the highest epoch in the
>> >> leader
>> >> >> >> node
>> >> >> >> >> >> log in
>> >> >> >> >> >> >> >> each
>> >> >> >> >> >> >> >> > fetch response. Once the info is stale in the
>> follower
>> >> >> >> side, it
>> >> >> >> >> >> >> updates
>> >> >> >> >> >> >> >> it
>> >> >> >> >> >> >> >> > from fetch response, and send next fetch request
>> with
>> >> >> >> correct
>> >> >> >> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent
>> data is
>> >> >> >> possible
>> >> >> >> >> >> like
>> >> >> >> >> >> >> >> > KAFKA-18723.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > I tried not to add new field in Fetch API, but
>> this is
>> >> >> the
>> >> >> >> best
>> >> >> >> >> >> >> >> solution I
>> >> >> >> >> >> >> >> > can think of.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR48. "The NumPartitions field is needed so the
>> >> >> controller
>> >> >> >> >> can
>> >> >> >> >> >> >> create
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > destination topic if it does not already exist.
>> When a
>> >> >> >> topic is
>> >> >> >> >> >> being
>> >> >> >> >> >> >> >> > mirrored for the first time, the destination
>> cluster
>> >> has
>> >> >> no
>> >> >> >> >> >> knowledge
>> >> >> >> >> >> >> >> > of the source topic's partition count, so the
>> request
>> >> >> must
>> >> >> >> >> carry
>> >> >> >> >> >> it."
>> >> >> >> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is
>> >> started,
>> >> >> the
>> >> >> >> >> source
>> >> >> >> >> >> >> topic
>> >> >> >> >> >> >> >> > may not exist yet. When the source topic is created
>> >> >> later,
>> >> >> >> we
>> >> >> >> >> can
>> >> >> >> >> >> >> start
>> >> >> >> >> >> >> >> > mirroring without requiring the user to provide the
>> >> >> number
>> >> >> >> of
>> >> >> >> >> >> >> >> partitions.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > When the source topic is created later, we can
>> start
>> >> >> >> >> mirroring
>> >> >> >> >> >> >> without
>> >> >> >> >> >> >> >> > requiring the user to provide the number of
>> >> partitions.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > This is because this happens in periodical sync up
>> in
>> >> the
>> >> >> >> >> >> destination
>> >> >> >> >> >> >> >> > cluster, and the numPartitions and other topic
>> info in
>> >> >> the
>> >> >> >> >> source
>> >> >> >> >> >> >> >> cluster
>> >> >> >> >> >> >> >> > is already retrieved during the sync up process.
>> >> Also, if
>> >> >> >> users
>> >> >> >> >> >> are
>> >> >> >> >> >> >> >> > creating mirror topics via admin API, this info can
>> >> also
>> >> >> be
>> >> >> >> >> >> retrieved
>> >> >> >> >> >> >> >> > inside the admin client if not provided.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly
>> need
>> >> the
>> >> >> >> last
>> >> >> >> >> >> mirror
>> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> > for truncation when starting a mirror. When a
>> mirror
>> >> is
>> >> >> >> >> stopped at
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > source, we already bump up the leader epoch and
>> write
>> >> >> some
>> >> >> >> >> >> additional
>> >> >> >> >> >> >> >> > records for cleanup. It seems that we can just use
>> the
>> >> >> >> >> >> DivergingEpoch
>> >> >> >> >> >> >> >> > returned in the initial mirror Fetch request to
>> detect
>> >> >> the
>> >> >> >> >> >> diverging
>> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> > and use that for truncation at the mirror
>> destination.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > That’s a good question. I thought about this before
>> >> but
>> >> >> >> >> >> >> unfortunately it
>> >> >> >> >> >> >> >> > doesn’t work. The reason we need the last mirror
>> epoch
>> >> >> >> record
>> >> >> >> >> is
>> >> >> >> >> >> >> because
>> >> >> >> >> >> >> >> > the destination cluster might not completely sync
>> with
>> >> >> the
>> >> >> >> >> source
>> >> >> >> >> >> >> >> cluster
>> >> >> >> >> >> >> >> > before STOPPED. And this un-mirrored data might
>> >> confuse
>> >> >> the
>> >> >> >> >> >> >> replication
>> >> >> >> >> >> >> >> > protocol when reverse mirror. For example,
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
>> >> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A
>> (LE:
>> >> 0)
>> >> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A
>> (LE:
>> >> 0)
>> >> >> >> >> >> >> >> > Offset 2: from A (LE: 10)
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > While cluster B mirrors until offset 1, the
>> cluster A
>> >> >> >> crashes
>> >> >> >> >> and
>> >> >> >> >> >> >> >> failover
>> >> >> >> >> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
>> >> >> >> >> >> MIRROR_PID_RESET
>> >> >> >> >> >> >> >> with
>> >> >> >> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror
>> from
>> >> >> >> cluster B,
>> >> >> >> >> >> the
>> >> >> >> >> >> >> log
>> >> >> >> >> >> >> >> > will be like this:
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> >> >> >> > foo-0 leader log: foo-0 leader
>> log:
>> >> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A
>> (LE:
>> >> 0)
>> >> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A
>> (LE:
>> >> 0)
>> >> >> >> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET
>> (LE:
>> >> >> 10)
>> >> >> >> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE:
>> >> 10)
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > As you can see, cluster A (new destination cluster)
>> >> >> doesn’t
>> >> >> >> >> know
>> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > offset 2 is not belonging to the cluster B, which
>> >> causes
>> >> >> >> >> >> inconsistent
>> >> >> >> >> >> >> >> logs.
>> >> >> >> >> >> >> >> > So like what we described in the KIP, the last
>> mirror
>> >> >> epoch
>> >> >> >> is
>> >> >> >> >> the
>> >> >> >> >> >> >> >> > synchronization point between source and
>> destination.
>> >> Any
>> >> >> >> data
>> >> >> >> >> >> beyond
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > point is treated as unknown to the source cluster
>> and
>> >> >> >> should be
>> >> >> >> >> >> >> >> truncated.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR52. Stop Mirror:
>> >> >> >> >> >> >> >> > > JR52.1 "3. If patterns are provided, the
>> controller
>> >> >> >> removes
>> >> >> >> >> >> >> matching
>> >> >> >> >> >> >> >> > entries from mirror.topics.include or adds them to
>> >> >> >> >> >> >> >> mirror.topics.exclude on
>> >> >> >> >> >> >> >> > the CLUSTER_MIRROR resource."
>> >> >> >> >> >> >> >> > Why is it 'or' and not 'and'?
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> >> >> > discovers new source topics matching the persisted
>> >> >> >> >> include/exclude
>> >> >> >> >> >> >> >> patterns
>> >> >> >> >> >> >> >> > and repeats steps 3 through 7 for each."
>> >> >> >> >> >> >> >> > The broker will send a create topic request to the
>> >> >> >> controller.
>> >> >> >> >> How
>> >> >> >> >> >> >> does
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > controller associate it with the mirror name to
>> >> create a
>> >> >> >> >> >> >> >> > `MirrorTopicStateChangeRecord`?
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > No, we don’t send a create topic request to the
>> >> >> controller
>> >> >> >> >> here.
>> >> >> >> >> >> >> >> Actually
>> >> >> >> >> >> >> >> > we send a StartMirrorTopics request to the
>> controller,
>> >> >> which
>> >> >> >> >> >> contains
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > mirror name and topic metadata like NumPartitions,
>> >> >> topicId…
>> >> >> >> >> etc.
>> >> >> >> >> >> So
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > controller can create the non-existed topic as
>> >> explained
>> >> >> in
>> >> >> >> >> JR48,
>> >> >> >> >> >> and
>> >> >> >> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR53. Describe Mirrors:
>> >> >> >> >> >> >> >> > > JR53.1 "1. The user sends a
>> DescribeClusterMirrors
>> >> >> request
>> >> >> >> >> with
>> >> >> >> >> >> >> >> optional
>> >> >> >> >> >> >> >> > mirror names (empty means all mirrors)."
>> >> >> >> >> >> >> >> > In metadata request, we use a null arrary to
>> represent
>> >> >> all
>> >> >> >> >> >> topics. It
>> >> >> >> >> >> >> >> would
>> >> >> >> >> >> >> >> > be useful to be consistent here.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > OK, KIP updated.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request
>> to
>> >> all
>> >> >> >> >> brokers
>> >> >> >> >> >> and
>> >> >> >> >> >> >> >> > merges
>> >> >> >> >> >> >> >> > the responses client-side."
>> >> >> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
>> >> >> >> partition.
>> >> >> >> >> So
>> >> >> >> >> >> >> sending
>> >> >> >> >> >> >> >> > the request to every broker is wasteful. It's
>> probably
>> >> >> >> better
>> >> >> >> >> to
>> >> >> >> >> >> send
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > request to a single broker, which can then contact
>> the
>> >> >> right
>> >> >> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Agree, KIP updated.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState:
>> Are
>> >> >> >> >> MIRRORING,
>> >> >> >> >> >> >> >> PAUSED,
>> >> >> >> >> >> >> >> > STOPPED the only validate states? The state table
>> has
>> >> >> other
>> >> >> >> >> states
>> >> >> >> >> >> >> too.
>> >> >> >> >> >> >> >> > Ditto for MirrorPartitionState.
>> >> >> >> >> >> >> >> > The DesiredState field in
>> MirrorTopicStateChangeRecord
>> >> >> is to
>> >> >> >> >> >> define
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > target state the user wants. And the desired state
>> >> users
>> >> >> can
>> >> >> >> >> >> assign
>> >> >> >> >> >> >> is
>> >> >> >> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a
>> >> desired
>> >> >> >> state
>> >> >> >> >> as
>> >> >> >> >> >> >> >> STOPPING
>> >> >> >> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
>> >> >> >> intermediate
>> >> >> >> >> >> >> states.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > About MirrorPartitionState record, it’ll record any
>> >> >> state in
>> >> >> >> >> the
>> >> >> >> >> >> >> state
>> >> >> >> >> >> >> >> > table that the current partition is. This is the
>> real
>> >> >> state
>> >> >> >> for
>> >> >> >> >> >> each
>> >> >> >> >> >> >> >> > partition. It’s like in the kubernetes world, when
>> we
>> >> >> change
>> >> >> >> >> the
>> >> >> >> >> >> >> replica
>> >> >> >> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired
>> >> >> state), it
>> >> >> >> >> won’t
>> >> >> >> >> >> >> >> increase
>> >> >> >> >> >> >> >> > to 5 immediately. The current replica
>> >> >> (MirrorPartitionState
>> >> >> >> >> >> record)
>> >> >> >> >> >> >> is
>> >> >> >> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > Thank you,
>> >> >> >> >> >> >> >> > Luke
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
>> >> >> >> >> >> dev@kafka.apache.org
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >> Hi, Fede,
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> Thanks for the reply. A few more comments.
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror
>> partition
>> >> >> cache
>> >> >> >> >> >> lives in
>> >> >> >> >> >> >> >> >> MirrorMetadataManager. It is populated in two
>> ways:
>> >> by
>> >> >> >> >> replaying
>> >> >> >> >> >> >> >> >> __mirror_state partitions this broker leads, and
>> by
>> >> >> >> fetching
>> >> >> >> >> >> state
>> >> >> >> >> >> >> >> >> from remote coordinators via ReadMirrorStates
>> RPCs."
>> >> >> >> >> >> >> >> >> Hmm, a given broker only caches the mirror
>> partition
>> >> >> state
>> >> >> >> for
>> >> >> >> >> >> >> >> partitions
>> >> >> >> >> >> >> >> >> hashed into the __mirror_state partition hosted on
>> >> this
>> >> >> >> >> broker,
>> >> >> >> >> >> >> right?
>> >> >> >> >> >> >> >> Why
>> >> >> >> >> >> >> >> >> does it need to fetch the state from other remote
>> >> >> >> >> coordinators?
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> JR36. I am still not quite sure why we need
>> >> >> >> MirrorLeaderEpoch
>> >> >> >> >> in
>> >> >> >> >> >> the
>> >> >> >> >> >> >> >> Fetch
>> >> >> >> >> >> >> >> >> request. In the current Fetch request, in
>> addition to
>> >> >> >> >> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> >> >> >> >> it has another field LastFetchedEpoch, which
>> captures
>> >> >> the
>> >> >> >> last
>> >> >> >> >> >> >> leader
>> >> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> >> in the data and is used to resolve the
>> inconsistency
>> >> >> >> between
>> >> >> >> >> the
>> >> >> >> >> >> >> >> follower
>> >> >> >> >> >> >> >> >> and the leader. Does it achieve the same purpose
>> as
>> >> >> >> >> >> >> MirrorLeaderEpoch?
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> JR48. "The NumPartitions field is needed so the
>> >> >> controller
>> >> >> >> can
>> >> >> >> >> >> >> create
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> >> destination topic if it does not already exist.
>> When
>> >> a
>> >> >> >> topic
>> >> >> >> >> is
>> >> >> >> >> >> >> being
>> >> >> >> >> >> >> >> >> mirrored for the first time, the destination
>> cluster
>> >> >> has no
>> >> >> >> >> >> >> knowledge
>> >> >> >> >> >> >> >> >> of the source topic's partition count, so the
>> request
>> >> >> must
>> >> >> >> >> carry
>> >> >> >> >> >> >> it."
>> >> >> >> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is
>> >> started,
>> >> >> the
>> >> >> >> >> >> source
>> >> >> >> >> >> >> >> topic
>> >> >> >> >> >> >> >> >> may not exist yet. When the source topic is
>> created
>> >> >> later,
>> >> >> >> we
>> >> >> >> >> can
>> >> >> >> >> >> >> start
>> >> >> >> >> >> >> >> >> mirroring without requiring the user to provide
>> the
>> >> >> number
>> >> >> >> of
>> >> >> >> >> >> >> >> partitions.
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly
>> need
>> >> the
>> >> >> >> last
>> >> >> >> >> >> mirror
>> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> >> for truncation when starting a mirror. When a
>> mirror
>> >> is
>> >> >> >> >> stopped
>> >> >> >> >> >> at
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> >> source, we already bump up the leader epoch and
>> write
>> >> >> some
>> >> >> >> >> >> >> additional
>> >> >> >> >> >> >> >> >> records for cleanup. It seems that we can just use
>> >> the
>> >> >> >> >> >> >> DivergingEpoch
>> >> >> >> >> >> >> >> >> returned in the initial mirror Fetch request to
>> >> detect
>> >> >> the
>> >> >> >> >> >> diverging
>> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> >> and use that for truncation at the mirror
>> >> destination.
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> JR52. Stop Mirror:
>> >> >> >> >> >> >> >> >> JR52.1 "3. If patterns are provided, the
>> controller
>> >> >> removes
>> >> >> >> >> >> matching
>> >> >> >> >> >> >> >> >> entries from mirror.topics.include or adds them to
>> >> >> >> >> >> >> >> mirror.topics.exclude
>> >> >> >> >> >> >> >> >> on
>> >> >> >> >> >> >> >> >> the CLUSTER_MIRROR resource."
>> >> >> >> >> >> >> >> >> Why is it 'or' and not 'and'?
>> >> >> >> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> >> >> >> discovers new source topics matching the persisted
>> >> >> >> >> >> include/exclude
>> >> >> >> >> >> >> >> >> patterns
>> >> >> >> >> >> >> >> >> and repeats steps 3 through 7 for each."
>> >> >> >> >> >> >> >> >> The broker will send a create topic request to the
>> >> >> >> controller.
>> >> >> >> >> >> How
>> >> >> >> >> >> >> does
>> >> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> >> controller associate it with the mirror name to
>> >> create a
>> >> >> >> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> JR53. Describe Mirrors:
>> >> >> >> >> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors
>> >> >> request
>> >> >> >> >> with
>> >> >> >> >> >> >> >> optional
>> >> >> >> >> >> >> >> >> mirror names (empty means all mirrors)."
>> >> >> >> >> >> >> >> >> In metadata request, we use a null arrary to
>> >> represent
>> >> >> all
>> >> >> >> >> >> topics.
>> >> >> >> >> >> >> It
>> >> >> >> >> >> >> >> >> would
>> >> >> >> >> >> >> >> >> be useful to be consistent here.
>> >> >> >> >> >> >> >> >> JR53.2 "2. The admin client fans out the request
>> to
>> >> all
>> >> >> >> >> brokers
>> >> >> >> >> >> and
>> >> >> >> >> >> >> >> merges
>> >> >> >> >> >> >> >> >> the responses client-side."
>> >> >> >> >> >> >> >> >> This is not ideal. Not every broker hosts a mirror
>> >> >> >> partition.
>> >> >> >> >> So
>> >> >> >> >> >> >> >> sending
>> >> >> >> >> >> >> >> >> the request to every broker is wasteful. It's
>> >> probably
>> >> >> >> better
>> >> >> >> >> to
>> >> >> >> >> >> >> send
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> >> request to a single broker, which can then contact
>> >> the
>> >> >> >> right
>> >> >> >> >> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState:
>> Are
>> >> >> >> >> MIRRORING,
>> >> >> >> >> >> >> PAUSED,
>> >> >> >> >> >> >> >> >> STOPPED the only validate states? The state table
>> has
>> >> >> other
>> >> >> >> >> >> states
>> >> >> >> >> >> >> too.
>> >> >> >> >> >> >> >> >> Ditto for MirrorPartitionState.
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> Jun
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
>> >> >> >> >> >> >> fedevaleri@gmail.com>
>> >> >> >> >> >> >> >> >> wrote:
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> > Hi Andrew, yes, validation is called at the
>> start
>> >> of
>> >> >> >> >> >> createTopic.
>> >> >> >> >> >> >> If
>> >> >> >> >> >> >> >> >> > the topic ID is invalid, the request fails with
>> >> >> >> >> >> INVALID_REQUEST.
>> >> >> >> >> >> >> If
>> >> >> >> >> >> >> >> >> > the topic ID is already used by a different
>> topic
>> >> >> name,
>> >> >> >> it
>> >> >> >> >> >> fails
>> >> >> >> >> >> >> with
>> >> >> >> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the
>> >> >> individual
>> >> >> >> >> topic
>> >> >> >> >> >> in
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> >> > batch fails; other topics in the same
>> CreateTopics
>> >> >> >> request
>> >> >> >> >> >> >> continue
>> >> >> >> >> >> >> >> >> > processing normally. Added this information to
>> the
>> >> >> KIP.
>> >> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew
>> Schofield <
>> >> >> >> >> >> >> >> >> schofielaj@apache.org>
>> >> >> >> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> >> >> > > Hi Fede,
>> >> >> >> >> >> >> >> >> > > One more small question.
>> >> >> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> >> >> > > AS37: If the controller fails validation for
>> the
>> >> >> >> >> >> >> MirrorInfo.TopicId
>> >> >> >> >> >> >> >> >> > added to the CreateTopic request, what does it
>> do?
>> >> I
>> >> >> >> suspect
>> >> >> >> >> >> that
>> >> >> >> >> >> >> it
>> >> >> >> >> >> >> >> >> fails
>> >> >> >> >> >> >> >> >> > the request with a particular error code.
>> >> >> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> >> >> > > Thanks,
>> >> >> >> >> >> >> >> >> > > Andrew
>> >> >> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> >> >> >> >> >> >> >> >> > > > Hi all,
>> >> >> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> >> >> > > > We updated the KIP to include Coordinator
>> >> Runtime
>> >> >> >> >> >> >> configurations
>> >> >> >> >> >> >> >> and
>> >> >> >> >> >> >> >> >> > > > the coordinator state transition validation
>> >> >> >> paragraph.
>> >> >> >> >> The
>> >> >> >> >> >> >> latter
>> >> >> >> >> >> >> >> >> > > > covers the per-partition epoch fencing
>> >> mechanism
>> >> >> >> (leader
>> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> and
>> >> >> >> >> >> >> >> >> > > > state epoch) that complements the existing
>> >> >> per-topic
>> >> >> >> >> >> >> StateOffset
>> >> >> >> >> >> >> >> >> > > > fencing for lifecycle operations.
>> >> >> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> >> >> > > > Thanks,
>> >> >> >> >> >> >> >> >> > > > Fede
>> >> >> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >>
>> >> >> >> >> >> >
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for the comments.
Answering them below.
> JR80. AdminClient.startMirrorTopics(): Is includePatterns() optional? It
seems that mirroring can't start without it.
Ah, right. I’ve moved the includePatterns from the StartMirrorTopicsOptions
into the required parameter in AdminClient.startMirrorTopics().
> JR91. Mirror configs: Some of the configs are prefixed with "mirror,"
while
others are not. Could we make them consistent? Since all those configs are
for Mirror, the mirror prefix seems redundant.
Indeed, we added the “mirror” prefix because we were afraid of confusing
users like “mirror.fetch.min.bytes” and “replica.fetch.min.bytes”. But like
you said, these configurations are for CLUSTER_MIRROR resource type, not
for brokers. Users should be able to distinguish between them easily.
Removed the mirror prefix from the mirror-level configs.
Thank you,
Luke
On Tue, Aug 18, 2026 at 2:04 AM Jun Rao via dev <de...@kafka.apache.org>
wrote:
> Hi, Luke,
>
> Thanks for the reply.
>
> JR80. AdminClient.startMirrorTopics(): Is includePatterns() optional? It
> seems that mirroring can't start without it.
>
> JR91. Mirror configs: Some of the configs are prefixed with "mirror," while
> others are not. Could we make them consistent? Since all those configs are
> for Mirror, the mirror prefix seems redundant.
>
> Jun
>
>
> On Mon, Aug 17, 2026 at 1:11 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the good comments.
> > Answering them below.
> >
> > > JR80. In that case, it seems that there is no need for
> > AdminClient.startMirrorTopics() to take topics. The adminClient only uses
> > it to register the included topics, which can be passed in as
> > includePatterns (specific names or regex). The broker will use
> > StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
> > through the AdminClient.
> >
> > Make sense. So when users invoke admin.startMirrorTopics, they only need
> > to set the topic patterns. And the receiving broker will resolve the
> > patterns into the actual topics using the listTopic response from the
> > source cluster, and get the topic id, numPartitions, … via the
> > describeTopics request from the source cluster if the topic is
> non-existent
> > in the destination cluster. KIP is updated.
> >
> >
> > > JR82. If “IncludeMirrorOffset” is disabled,
> > DescribeClusterMirrorsResponse.SourceOffset is still returned in the
> > response and what is its value in this case?
> >
> > It will be -1 because when the `IncludeMirrorOffset` is disabled, we
> won’t
> > query the mirror partition leader. I’ve updated the description to make
> it
> > clear. Thanks.
> >
> >
> > > JR88. Why is partition-load-time a Meter instead of a Gauge?
> >
> > You’re right. It is not the Meter type. It’s the SampledStat state like
> > “batch-size-avg” and “batch-size-max” in producer. I’ve updated the KIP.
> >
> >
> > > JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
> >
> > You’re right. It’s the rate type. KIP is updated
> >
> >
> > > JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
> > Meter.
> > Yes, it makes more sense to make it as Meter type. I’ve updated these
> > error-related metrics to Meter type:
> > 1. MetadataRefreshError
> > 2. TopicConfigMetadataSyncError
> > 3. ConsumerGroupOffsetSyncError
> > 4. ShareGroupOffsetSyncError
> > 5. AclSyncError
> >
> >
> > Thank you,
> > Luke
> >
> >
> > On Sat, Aug 15, 2026 at 8:10 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Luke,
> >>
> >> Thanks for the reply.
> >>
> >> JR80. In that case, it seems that there is no need for
> >> AdminClient.startMirrorTopics() to take topics. The adminClient only
> uses
> >> it to register the included topics, which can be passed in as
> >> includePatterns (specific names or regex). The broker will use
> >> StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
> >> through the AdminClient.
> >>
> >> JR82. If “IncludeMirrorOffset” is disabled,
> >> DescribeClusterMirrorsResponse.SourceOffset is still returned in the
> >> response and what is its value in this case?
> >>
> >> JR88. Why is partition-load-time a Meter instead of a Gauge?
> >>
> >> JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
> >>
> >> JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
> >> Meter.
> >>
> >> Jun
> >>
> >> On Fri, Aug 14, 2026 at 4:32 AM Luke Chen <sh...@gmail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thanks for the good comments,
> >> > Answering them below.
> >> >
> >> > > JR80. PauseMirrorTopicsOptions:
> "PauseMirrorTopics/ResumeMirrorTopics
> >> > supports regex in the ClusterMirrorComamnd, but we don’t need to add
> the
> >> > “patterns”."
> >> > So, in this case, we use topics to support both specific topics and
> >> regex.
> >> > However, startMirrorTopicsOptions uses topics for specific topics and
> >> > includePatterns for regex. Could we make them consistent?
> >> >
> >> >
> >> > The purpose of the “includePatterns” field is for the
> >> > “mirror.topics.include” update, so that if there is any newly created
> >> topic
> >> > in the source cluster that matches the regex, we can create it. And
> the
> >> > “Topics” field in the StartMirrorTopicsRequest is to specify the
> topicId
> >> > and numPartitions in the source cluster. So I think “includePatterns”
> >> and
> >> > “Topics” both are necessary in the StartMirrorTopicsRequest. On the
> >> other
> >> > hand, in PauseMirrorTopics, because it won’t change the
> >> > “mirror.topics.include”/”mirror.topics.exclude” configs, we don’t need
> >> to
> >> > provide the regex in the request.
> >> >
> >> >
> >> >
> >> > > JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also
> >> -1
> >> > if
> >> > it's not included in the request?
> >> >
> >> > If “IncludeMirrorOffset” is disabled, we won’t query the mirror
> >> partition
> >> > leader, so we can’t get the sourceOffset.
> >> >
> >> > I assume you want to ask `if it’s still -1 if not actively mirroring,
> >> > while “IncludeMirrorOffset” is enabled`. That’s a good question. I
> think
> >> > our purpose of this command is to show the lag to users. So if the
> >> state is
> >> > not MIRRORING (ex: STOPPED/PAUSED), we think it’s not helpful to users
> >> to
> >> > show only the sourceOffset. Of course users can use
> >> `kafka-get-offsets.sh`
> >> > to get the offset if they really want to know.
> >> >
> >> >
> >> > > JR83. Start Mirror Topics:
> >> > > JR83.1 Should the controller verify the expected state for the
> topic?
> >> >
> >> > As replied in JR84, the desired state validation should happen in the
> >> > broker. And the controller validates the StateOffset. I’ve updated the
> >> > workflow to make it clear.
> >> >
> >> > > JR83.2 "d. truncates the log accordingly,"
> >> > Could you add a description of the truncation offset?
> >> >
> >> > OK, added one more sentence:
> >> > d. based on the last mirror epoch response in (b), the broker
> translates
> >> > the epoch into the offset by querying the leaderEpochCache
> >> > (endOffsetForEpoch), and truncates the log accordingly, and waits for
> >> all
> >> > ISR members…
> >> >
> >> >
> >> > > JR84. Stop Mirror Topics: "The controller verifies that the desired
> >> state
> >> > of all topics are in MIRRORING or PAUSED state."
> >> > Should the broker verfy the topic level state? I thought the
> controller
> >> > will use StateOffset in the request to make sure the validation on the
> >> > broker is stil valid. If the controller performs the verification, is
> >> > StateOffset still useful?
> >> >
> >> > Oh, you’re right. We’re doing duplicated work here. Yes, the broker
> >> > already verifies the “desired state” of all topics are in MIRRORING or
> >> > PAUSED state, as well as the “actual state” is in MIRRORING or PAUSED
> >> > state. If all validation passes, we’ll forward it to the controller.
> So
> >> the
> >> > controller only needs to validate the StateOffset to make sure the
> >> > validation on the broker is still valid. So, yes, we don’t need the
> >> desired
> >> > state validation on the controller side. I’ve updated the KIP to make
> it
> >> > clear.
> >> >
> >> > One more thing to add is that, I’ve thought about not adding
> >> StateOffset,
> >> > and relying on the desired state validation to make sure we’re safe to
> >> do
> >> > the deletion or other operations. The problem is that, even if the
> >> broker
> >> > sees the desired state for topic A is STOPPED, and then the controller
> >> also
> >> > sees the desired state is STOPPED, it still cannot make sure the
> >> > controller’s STOPPED state equals the broker’s STOPPED state. For
> >> example:
> >> > 1. Broker sees STOPPED, then forward to controller
> >> > 2. User changes the desired state to MIRRORING
> >> > 3. User changes the desired state to STOPPED
> >> > 4. Controller sees the desired state is STOPPED
> >> > So I think StateOffset is required.
> >> >
> >> > Thanks for catching it BTW.
> >> >
> >> >
> >> > > JR85. There are still a few references to includeMirrorLags. They
> >> should
> >> > be
> >> > IncludeMirrorOffset.
> >> >
> >> > Sorry, just updated.
> >> >
> >> >
> >> > > JR86. Metrics:
> >> > > JR86.1 Could you describe the type (gauge, meter, etc) of each
> metric?
> >> >
> >> > KIP is updated. I also group the metrics into new added metrics and
> >> other
> >> > inherited metrics.
> >> >
> >> > > JR86.2 Could you describe a bit more what MinFetchRate represents?
> >> >
> >> > It means the lowest fetch request rate among all of this mirror
> fetcher
> >> > manager's fetcher threads. It inherits from the
> AbstractFetcherManager’s
> >> > metric.
> >> >
> >> >
> >> > > JR87. Version Compatibility: "For migration or DR use cases where
> >> > failback
> >> > is not required, this proposal supports source brokers up to version
> 2.1
> >> > included"
> >> > This KIP needs topicId and KIP-516 was introduced later than 2.1.
> >> > Yes, when mirroring from a cluster without topic id support, we will
> >> > assign a random topic ID in the destination cluster and then fetch
> >> (without
> >> > topic id in the fetch request using the old version of Fetch API) as
> >> usual.
> >> > Of course this is for the one-way mirroring use case. It can’t start
> the
> >> > reverse mirror(failback) under this setup. You can check the
> >> “Compatibility
> >> > Matrix” section for more detail.
> >> >
> >> >
> >> > Thank you,
> >> > Luke
> >> >
> >> > On Fri, Aug 14, 2026 at 7:09 AM Jun Rao via dev <dev@kafka.apache.org
> >
> >> > wrote:
> >> >
> >> >> Hi, Luke,
> >> >>
> >> >> Thanks for the reply.
> >> >>
> >> >> JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
> >> >> supports regex in the ClusterMirrorComamnd, but we don’t need to add
> >> the
> >> >> “patterns”."
> >> >> So, in this case, we use topics to support both specific topics and
> >> regex.
> >> >> However, startMirrorTopicsOptions uses topics for specific topics and
> >> >> includePatterns for regex. Could we make them consistent?
> >> >>
> >> >> JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also
> >> -1 if
> >> >> it's not included in the request?
> >> >>
> >> >> JR83. Start Mirror Topics:
> >> >> JR83.1 Should the controller verify the expected state for the topic?
> >> >> JR83.2 "d. truncates the log accordingly,"
> >> >> Could you add a description of the truncation offset?
> >> >>
> >> >> JR84. Stop Mirror Topics: "The controller verifies that the desired
> >> state
> >> >> of all topics are in MIRRORING or PAUSED state."
> >> >> Should the broker verfy the topic level state? I thought the
> controller
> >> >> will use StateOffset in the request to make sure the validation on
> the
> >> >> broker is stil valid. If the controller performs the verification, is
> >> >> StateOffset still useful?
> >> >>
> >> >> JR85. There are still a few references to includeMirrorLags. They
> >> should
> >> >> be
> >> >> IncludeMirrorOffset.
> >> >>
> >> >> JR86. Metrics:
> >> >> JR86.1 Could you describe the type (gauge, meter, etc) of each
> metric?
> >> >> JR86.2 Could you describe a bit more what MinFetchRate represents?
> >> >>
> >> >> JR87. Version Compatibility: "For migration or DR use cases where
> >> failback
> >> >> is not required, this proposal supports source brokers up to version
> >> 2.1
> >> >> included"
> >> >> This KIP needs topicId and KIP-516 was introduced later than 2.1.
> >> >>
> >> >> Jun
> >> >>
> >> >>
> >> >> On Thu, Aug 13, 2026 at 5:14 AM Luke Chen <sh...@gmail.com> wrote:
> >> >>
> >> >> > Hi Jun,
> >> >> >
> >> >> > Thanks for the good comments.
> >> >> > Answering them below.
> >> >> >
> >> >> > > JR59.1 Could you add the RPC to the workflow? The broker first
> >> needs
> >> >> to
> >> >> > issue ListClusterMirrorsRequest with SourceClusterIdFilter to get
> the
> >> >> > mirror name and then issue DescribeClusterMirrors to get the LME.
> >> >> >
> >> >> > Yes, you’re right. The workflow is updated to make them clear.
> >> >> >
> >> >> > > JR71. mirror.support.unclean.leader.election : The name is
> >> confusing.
> >> >> > It's
> >> >> > also used in LOG_TRUNCATION, which is not related to unclean leader
> >> >> > election. Could we use a better name?
> >> >> >
> >> >> > I personally think the name should be fine because even though we
> >> use in
> >> >> > “LOG_TRUNCATION” state, the config is to allow us to know if users
> >> want
> >> >> to
> >> >> > support unclean leader election, if so, we wait for all replicas to
> >> >> catch
> >> >> > up. If we want to explicitly show the intention of the config, how
> >> about
> >> >> > “mirror.full.replica.convergence.required”? I’m open to other
> >> >> suggestions.
> >> >> >
> >> >> >
> >> >> > > JR72. DescribeClusterMirrors
> >> >> > > JR72.1 A user may specify just the topic name when describing a
> >> >> mirror.
> >> >> > Should we include the topic name in the request?
> >> >> >
> >> >> > Hmm.. we didn’t think about the use case. But yes, if users only
> care
> >> >> > about the state of a topic, they don’t need the info for other
> >> topics.
> >> >> I’ve
> >> >> > combined with JR72.2, added “TopicPartitions” filed into
> >> >> > DescribeClusterMirrorsRequest, and by default is null, which means
> >> all
> >> >> > topic partitions under the requested mirrors will be returned. If
> >> set,
> >> >> > we’ll query the mirrors and return the topic partitions that users
> >> >> > expected. KIP is updated.
> >> >> >
> >> >> >
> >> >> > > JR72.2 LastMirrorEpochLookups is a bit weird since the response
> >> >> returns
> >> >> > either mirror state or mirror lag and LastMirrorEpochLookups
> doesn't
> >> >> match
> >> >> > either.
> >> >> >
> >> >> > Yes, after the recent refactor, I think the LastMirrorEpochLookups
> >> can
> >> >> be
> >> >> > merged into one level up, combining with the topic name filter in
> >> above
> >> >> > comment, which will allow users to query down to the partition
> level.
> >> >> The
> >> >> > LastMirrorEpochLookups will query along with normal mirror state by
> >> the
> >> >> > readMirrorState from the coordinator. The difference is that after
> >> >> > receiving the response, it will filter out the mirrors that don’t
> >> match
> >> >> the
> >> >> > requested sourceClusterId, and then add the lastMirrorEpoch into
> >> >> > DescribeClusterMirrorResponse. KIP is updated.
> >> >> >
> >> >> >
> >> >> > > JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> >
> >> >> > Right. Added.
> >> >> >
> >> >> >
> >> >> > > JR72.4 "includeMirrorLags: source offset, destination offset,
> lag"
> >> >> > DescribeClusterMirrorsRequest doesn't include those fields. Also,
> lag
> >> >> > should be omitted.
> >> >> >
> >> >> > Sorry, I removed the “SourceOffset” and “DestinationOffset” fields
> >> from
> >> >> > DescribeClusterMirrorsResponse because I added them into
> >> >> ReadMirrorOffsets
> >> >> > (ex-ReadMirrorLags) API. But I forgot we still need them in the
> >> >> > DescribeClusterMirrorsResponse so that the adminClient can receive
> >> the
> >> >> > results. KIP is updated.
> >> >> >
> >> >> >
> >> >> > > JR73. MirrorPartitionState record, LastMirrorEpochs record and
> >> >> > MirrorPartitionStateRequest: The values for type and name are
> >> incorrect.
> >> >> >
> >> >> > Ah, that’s my bad. Fixed.
> >> >> >
> >> >> > > JR74. ListClusterMirrorsRequest
> >> >> > > JR74.1 When is DesiredMirrorStateFilter used? There is no option
> to
> >> >> > specify
> >> >> > the desired state in the mirror-topic command.
> >> >> >
> >> >> > Yes, referred to kafka-consumer-groups.sh, I added the `--state`
> >> option
> >> >> to
> >> >> > allow users to set the state when listMirror.
> >> >> >
> >> >> > > JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
> >> >> >
> >> >> > Yes, KIP updated.
> >> >> >
> >> >> > > JR75. ReadMirrorLags
> >> >> > > JR75.1 Should it be named ReadMirrorOffsets since the response
> >> doesn't
> >> >> > include lag?
> >> >> >
> >> >> > Indeed. Updated.
> >> >> >
> >> >> > > JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> >
> >> >> > Yes, added
> >> >> >
> >> >> > > JR75.3 Some places refer to this as ReadClusterMirrorLags. We
> need
> >> to
> >> >> be
> >> >> > consistent.
> >> >> >
> >> >> > Thanks for pointing it out. KIP is updated.
> >> >> >
> >> >> > > JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR
> >> too?
> >> >> >
> >> >> > Yes, we should validate the mirror name before writing the data.
> KIP
> >> is
> >> >> > updated.
> >> >> >
> >> >> > > JR77. Delete Mirror: "3. The controller verifies that no topics
> >> are in
> >> >> > MIRRORING or PAUSED state."
> >> >> > Is this necessary since step 2 has verified all partitions are in
> >> >> STOPPED
> >> >> > state, which can only happen if the topics are in STOPPED state.
> >> >> >
> >> >> > It’s true, but there could be some delay between “actual state” and
> >> >> > “desired state”. In step (2), we verify all partitions’ “actual
> >> state”
> >> >> are
> >> >> > in STOPPED state. But the current desired state could be MIRRORING,
> >> it’s
> >> >> > just that the mirror partition leaders haven’t got the metadata
> >> update
> >> >> yet.
> >> >> > So, the controller still needs to verify the desired state is not
> in
> >> >> > MIRRORING or PAUSED. I found we didn’t specify that the controller
> is
> >> >> > actually verifying the “desired state”, not “actual state”. And
> this
> >> >> > controller’s desired state validation also applies in
> >> >> >
> >> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
> >> >> > I’ve updated the KIP to make it clear.
> >> >> >
> >> >> > > JR78. STOPPING: "The mirror fetcher is removed, the leader epoch
> is
> >> >> > bumped,
> >> >> > ABORT markers are appended for all ongoing transactions, last
> mirror
> >> >> epochs
> >> >> > are persisted, and a MIRROR_PID_RESET record appended."
> >> >> > This describes the state for STOPPED, not STOPPING.
> >> >> >
> >> >> > No, these are the tasks during STOPPING state, like what we
> >> described in
> >> >> > “Stop Mirror Topics” workflow. If we do these in STOPPED state, we
> >> can’t
> >> >> > safely move the readonly topic into writable because not all
> required
> >> >> tasks
> >> >> > are done. I think I know why it confuses you, it’s because all the
> >> tasks
> >> >> > are in the completed state. I’ve rephrased it to make it clear.
> >> >> >
> >> >> >
> >> >> > > JR79. RPCs like DescribeClusterMirrors can take time. Should we
> >> add a
> >> >> > timeout field?
> >> >> >
> >> >> > Make sense. Added into `DescribeClusterMirrors` request, as well as
> >> >> >
> >> >>
> >>
> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
> >> >> > because we will query the partition states from coordinator for
> state
> >> >> > validation in these APIs.
> >> >> >
> >> >> >
> >> >> > > JR80. PauseMirrorTopicsOptions: Should it support patterns? The
> >> >> command
> >> >> > line seems to suppor it.
> >> >> >
> >> >> > Sorry, it’s a little confusing here. The “pattern” in
> >> >> > startMirrorTopicsOptions or stopMirrorTopicsOptions is to update
> the
> >> >> > “mirror.topics.include” and “mirror.topics.exclude” configuration
> for
> >> >> topic
> >> >> > pattern discovery periodically. For the PauseMirrorTopics and
> >> >> > ResumeMirrorTopics in the clusterMirrorCommand, users can provide
> >> regex
> >> >> in
> >> >> > the “--topics” option, and the clusterMirrorCommand will listTopic
> in
> >> >> the
> >> >> > destination cluster and match with the patterns users provided, and
> >> then
> >> >> > feeds the matched topics into the
> >> PauseMirrorTopics/ResumeMirrorTopics
> >> >> > requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics
> >> >> supports
> >> >> > regex in the ClusterMirrorComamnd, but we don’t need to add the
> >> >> “patterns”.
> >> >> >
> >> >> >
> >> >> > > JR81. AdminClient
> >> >> > > JR81.1 ListClusterMirrors: We need to add the option to specify
> >> >> different
> >> >> > filters supported in the request.
> >> >> >
> >> >> > We already have the `--mirror` option. Added `--state` and
> >> >> > `--sourceClusterId` options to support the other filters.
> >> >> >
> >> >> > > JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations:
> >> The
> >> >> > request doesn't support this option. Could we add other options
> >> >> supported
> >> >> > in the request?
> >> >> >
> >> >> > Currently, we have “includeAuthorizedOperations” in
> DescribeCluster,
> >> >> > DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
> >> >> > DescribeStreamsGroups requests. But none of them include the option
> >> in
> >> >> the
> >> >> > tool to allow users to enable it. I think the reason is that we
> don’t
> >> >> want
> >> >> > to show this to users on the tool output. But having this field, we
> >> >> allow
> >> >> > the advanced users to invoke admin API with
> >> includeAuthorizedOperations
> >> >> > enabled to get the results they want.
> >> >> >
> >> >> >
> >> >> > > JR81.3 LeaderStateDescription: Should it be named
> >> >> > MirrorPartitionStateDescription? Also, could we make the fields
> >> >> consistent
> >> >> > with those in the response?
> >> >> >
> >> >> > OK. KIP is updated.
> >> >> >
> >> >> >
> >> >> > Thank you,
> >> >> > Luke
> >> >> >
> >> >> > On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <
> >> dev@kafka.apache.org>
> >> >> > wrote:
> >> >> >
> >> >> >> Hi, Luke,
> >> >> >>
> >> >> >> Thanks for the reply.
> >> >> >>
> >> >> >> JR59.1 Could you add the RPC to the workflow? The broker first
> >> needs to
> >> >> >> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get
> >> the
> >> >> >> mirror name and then issue DescribeClusterMirrors to get the LME.
> >> >> >>
> >> >> >> JR71. mirror.support.unclean.leader.election : The name is
> >> confusing.
> >> >> It's
> >> >> >> also used in LOG_TRUNCATION, which is not related to unclean
> leader
> >> >> >> election. Could we use a better name?
> >> >> >>
> >> >> >> JR72. DescribeClusterMirrors
> >> >> >> JR72.1 A user may specify just the topic name when describing a
> >> mirror.
> >> >> >> Should we include the topic name in the request?
> >> >> >> JR72.2 LastMirrorEpochLookups is a bit weird since the response
> >> returns
> >> >> >> either mirror state or mirror lag and LastMirrorEpochLookups
> doesn't
> >> >> match
> >> >> >> either.
> >> >> >> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> >> JR72.4 "includeMirrorLags: source offset, destination offset, lag"
> >> >> >> DescribeClusterMirrorsRequest doesn't include those fields. Also,
> >> lag
> >> >> >> should be omitted.
> >> >> >>
> >> >> >> JR73. MirrorPartitionState record, LastMirrorEpochs record and
> >> >> >> MirrorPartitionStateRequest: The values for type and name are
> >> >> incorrect.
> >> >> >>
> >> >> >> JR74. ListClusterMirrorsRequest
> >> >> >> JR74.1 When is DesiredMirrorStateFilter used? There is no option
> to
> >> >> >> specify
> >> >> >> the desired state in the mirror-topic command.
> >> >> >> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
> >> >> >>
> >> >> >> JR75. ReadMirrorLags
> >> >> >> JR75.1 Should it be named ReadMirrorOffsets since the response
> >> doesn't
> >> >> >> include lag?
> >> >> >> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> >> JR75.3 Some places refer to this as ReadClusterMirrorLags. We need
> >> to
> >> >> be
> >> >> >> consistent.
> >> >> >>
> >> >> >> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR
> >> too?
> >> >> >>
> >> >> >> JR77. Delete Mirror: "3. The controller verifies that no topics
> are
> >> in
> >> >> >> MIRRORING or PAUSED state."
> >> >> >> Is this necessary since step 2 has verified all partitions are in
> >> >> STOPPED
> >> >> >> state, which can only happen if the topics are in STOPPED state.
> >> >> >>
> >> >> >> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch
> is
> >> >> >> bumped,
> >> >> >> ABORT markers are appended for all ongoing transactions, last
> mirror
> >> >> >> epochs
> >> >> >> are persisted, and a MIRROR_PID_RESET record appended."
> >> >> >> This describes the state for STOPPED, not STOPPING.
> >> >> >>
> >> >> >> JR79. RPCs like DescribeClusterMirrors can take time. Should we
> add
> >> a
> >> >> >> timeout field?
> >> >> >>
> >> >> >> JR80. PauseMirrorTopicsOptions: Should it support patterns? The
> >> command
> >> >> >> line seems to suppor it.
> >> >> >>
> >> >> >> JR81. AdminClient
> >> >> >> JR81.1 ListClusterMirrors: We need to add the option to specify
> >> >> different
> >> >> >> filters supported in the request.
> >> >> >> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations:
> >> The
> >> >> >> request doesn't support this option. Could we add other options
> >> >> supported
> >> >> >> in the request?
> >> >> >> JR81.3 LeaderStateDescription: Should it be named
> >> >> >> MirrorPartitionStateDescription? Also, could we make the fields
> >> >> consistent
> >> >> >> with those in the response?
> >> >> >>
> >> >> >> Jun
> >> >> >>
> >> >> >> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com>
> >> wrote:
> >> >> >>
> >> >> >> > Hi Jun,
> >> >> >> >
> >> >> >> > Thanks for the good comments again.
> >> >> >> > Answering them below.
> >> >> >> >
> >> >> >> > > JR51. "ULE_LOG_TRUNCATION When
> >> >> mirror.support.unclean.leader.election
> >> >> >> is
> >> >> >> > enabled and the destination leader has log truncation, the state
> >> will
> >> >> >> move
> >> >> >> > from MIRRORING to this state."
> >> >> >> > This still suggests that divergeEpoch is only handled if
> >> >> >> > mirror.support.unclean.leader.election
> >> >> >> > is enabled.
> >> >> >> >
> >> >> >> >
> >> >> >> > OK, KIP is updated to make it clear.
> >> >> >> >
> >> >> >> > > JR55. Should the mirror state and the mirror lags always be
> >> exposed
> >> >> >> > together? If so, we can remove IncludeMirrorState and
> >> >> IncludeMirrorLags
> >> >> >> > from DescribeClusterMirrorsRequest. If not, it's better to use
> two
> >> >> >> separate
> >> >> >> > RPCs.
> >> >> >> >
> >> >> >> >
> >> >> >> > No, we sometimes only need MirrorState, and sometimes need both.
> >> So
> >> >> I’ll
> >> >> >> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
> >> >> >> > DescribeClusterMirrorsRequest. And we will use ReadMirrorStates
> >> and
> >> >> >> > ReadMirrorLags to get each info.
> >> >> >> >
> >> >> >> >
> >> >> >> > > DescribeClusterMirrorsRequest should be used in the Describe
> >> Mirror
> >> >> >> flow,
> >> >> >> > right? Currently it's missing. Also, I assume the admin client
> >> >> refers to
> >> >> >> > the one in the kafka-cluster-mirrors tool. But it seems that
> >> >> sometimes
> >> >> >> it
> >> >> >> > also refers to the admin client on the broker? Perhaps we can
> use
> >> >> 'the
> >> >> >> > tool' instead to make it clear.
> >> >> >> >
> >> >> >> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
> >> >> >> > And sorry for the confusion. I’d like to keep the admin client
> as
> >> the
> >> >> >> one
> >> >> >> > invoking from the tool (or user’s direct admin client API
> >> >> invocation).
> >> >> >> On
> >> >> >> > the broker side, we actually use InterBrokerSendThread to invoke
> >> >> these
> >> >> >> RPCs
> >> >> >> > or get directly from MetadataImage. I’ve updated the KIP to make
> >> it
> >> >> >> clear.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR58. BumpLeaderEpochs: Why does it need to be issued to the
> >> >> broker,
> >> >> >> > instead of just to the controller?
> >> >> >> >
> >> >> >> > Yes, we only need to issue to the controller. Updated to the
> >> >> >> > BumpLeaderEpochs API.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR59. Start Mirror Topics:
> >> >> >> > > JR59.1 "During this state, the broker queries the source
> cluster
> >> >> for
> >> >> >> last
> >> >> >> > mirror epochs".
> >> >> >> > Could you describe which RPC is issued to find the LME?
> >> >> >> >
> >> >> >> > We’re using describeClusterMirror API via admin client here.
> >> >> >> >
> >> >> >> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
> >> request
> >> >> >> > doesn't
> >> >> >> > contain a replication factor.
> >> >> >> >
> >> >> >> > We use “default.replication.factor” when topic creation, so the
> >> >> >> > INVALID_REPLICATION_FACTOR will be thrown like normal topic
> >> creation.
> >> >> >> Also,
> >> >> >> > because we require the replicas for the partition assignment to
> be
> >> >> all
> >> >> >> > active and not in fenced or controlled shutdown(JR35), we’ll
> throw
> >> >> >> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used
> >> by a
> >> >> >> > different topic name"
> >> >> >> > Should we also return this error when the topic name is used as
> a
> >> >> >> regular
> >> >> >> > topic?
> >> >> >> >
> >> >> >> > There are 2 errors we will throw when topic id <-> topic name
> >> >> mismatch.
> >> >> >> > Suppose provided topic id: ID_A, topic name: NAME_A
> >> >> >> > 1. Destination cluster exists topic id: ID_A, but topic name is:
> >> >> NAME_B
> >> >> >> > TOPIC_ALREADY_EXISTS will be thrown
> >> >> >> > 2. Destination cluster exists topic id: ID_B, but topic name is:
> >> >> NAME_A
> >> >> >> > INCONSISTENT_TOPIC_ID will be thrown
> >> >> >> >
> >> >> >> > Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS
> >> error
> >> >> >> for
> >> >> >> > these 2 cases makes more sense. KIP updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR60. CreateClusterMirrorRequest: When will it receive an
> >> >> >> > INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> >> >> > CLUSTER_MIRROR_ALREADY_EXISTS?
> >> >> >> >
> >> >> >> >
> >> >> >> > Sorry, createClusterMirrorRequest doesn’t throw
> >> >> >> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate
> >> the
> >> >> >> mirror
> >> >> >> > states when creating a cluster mirror. We only care if the
> “mirror
> >> >> name”
> >> >> >> > exists or not. KIP updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we
> >> just
> >> >> >> remove
> >> >> >> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> >> >> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to
> describe
> >> >> the
> >> >> >> > specific state causing the error.
> >> >> >> >
> >> >> >> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following
> >> >> errors:
> >> >> >> > a. MIRROR_TOPIC_ALREADY_PAUSED
> >> >> >> > b. MIRROR_TOPIC_NOT_PAUSED,
> >> >> >> > c. MIRROR_TOPIC_ALREADY_STOPPED,
> >> >> >> > d. CLUSTER_MIRROR_NOT_EMPTY
> >> >> >> > e. MIRROR_TOPIC_NOT_STOPPED
> >> >> >> >
> >> >> >> > KIP is updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in
> some
> >> >> >> requests
> >> >> >> > (e.g. StopMirrorTopics), but not others (e.g.
> StartMirrorTopics).
> >> >> Could
> >> >> >> we
> >> >> >> > make them consistent?
> >> >> >> >
> >> >> >> >
> >> >> >> > Good catch. The updated errors for the new added RPC are this:
> >> >> >> >
> >> >> >> > CreateClusterMirror: NOT_CONTROLLER
> >> >> >> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR63. StopMirrorTopics:
> >> >> >> > > JR63.1 "The broker validates that all target topic partitions
> >> are
> >> >> in
> >> >> >> > either
> >> >> >> > LOG_TRUNCATION or MIRRORING or FAILED state."
> >> >> >> > Is this accurate? LOG_TRUNCATION can't transition to STOP.
> PAUSED
> >> >> can.
> >> >> >> >
> >> >> >> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is
> >> >> updated.
> >> >> >> >
> >> >> >> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error
> >> since
> >> >> >> PAUSED
> >> >> >> > -> STOPPED is valid?
> >> >> >> >
> >> >> >> > Yes, it is not correct. KIP updated to remove this error.
> >> >> >> >
> >> >> >> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
> >> >> stopped
> >> >> >> > state"
> >> >> >> > This is weird. The reason to issue StopMirrorTopics is that it's
> >> not
> >> >> in
> >> >> >> > stopped state.
> >> >> >> >
> >> >> >> > Indeed. KIP is updated to only include
> >> INVALID_CLUSTER_MIRROR_STATES
> >> >> >> > error, and will be thrown when states are not in MIRRORING or
> >> PAUSED
> >> >> >> state.
> >> >> >> >
> >> >> >> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since
> it's
> >> >> >> derived
> >> >> >> > from SourceOffset and DestinationOffset.
> >> >> >> >
> >> >> >> > True. Removed.
> >> >> >> >
> >> >> >> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems
> >> >> that we
> >> >> >> > could just set the partition level state to STOPPED.
> >> >> >> >
> >> >> >> > Nice catch. We don’t use it at all. That is some old leftover
> >> field
> >> >> we
> >> >> >> > forgot to remove. KIP updated.
> >> >> >> >
> >> >> >> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to
> be
> >> >> sent
> >> >> >> to
> >> >> >> > the controller?
> >> >> >> >
> >> >> >> > No, these 2 only send to the broker. KIP updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and
> >> >> StateEpoch?
> >> >> >> It
> >> >> >> > seems both are used for fencing and fencing based on the latter
> is
> >> >> >> > stronger.
> >> >> >> >
> >> >> >> > Yes, basically the StateEpoch can fence for most cases. The
> >> >> LeaderEpoch
> >> >> >> > fencing is used for cases like:
> >> >> >> >
> >> >> >> > 1. b1 is the current leader for foo-0, preparing write state
> from
> >> >> >> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
> >> >> >> > 2. leadership change to b2 for foo-0, LE changes to 2.
> >> >> >> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1
> }.
> >> >> >> > 4. The coordinator receives the read from b2 in step (3), the
> >> highest
> >> >> >> > lastSeen LE for foo-0 updates to 2 in coordinator.
> >> >> >> > 5. The writes state from b1 in step (1) arrives the coordinator,
> >> but
> >> >> it
> >> >> >> > rejects b1 writes because the highest lastSeen LE is 2, which is
> >> > 1
> >> >> in
> >> >> >> the
> >> >> >> > request.
> >> >> >> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2,
> >> stateEpoch: 1
> >> >> and
> >> >> >> > succeeds.
> >> >> >> >
> >> >> >> > The leaderEpoch fence is to avoid any stale broker writes after
> >> the
> >> >> new
> >> >> >> > leader epoch read. This cannot be protected by the stateEpoch.
> The
> >> >> >> > stateEpoch can prevent the write with older stateEpoch. So in
> the
> >> >> >> situation
> >> >> >> > above, the stateEpoch cannot determine if we should accept b1
> >> writes
> >> >> or
> >> >> >> > not, but leaderEpoch can. By the way, we borrowed this concept
> >> from
> >> >> >> > KIP-932:
> >> >> >> >
> >> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
> >> <
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!t3M02ge9e-mRqsshxasG_K0qLi9sX21suoBd69nLUl_gWUvn9ZLLekJiQspa_q3gTVSzWyUQBQfQLHk$
> >
> >> >> <
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rX3-NwDU3s0U7POQYH1R-DvASnCPTTtFRWxZTVaS-PQHg-C1TAcu9xMe3MQF4fIXGO67WKOqrlz-lc8$
> >> >
> >> >> >> <
> >> >>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!u8EkPGXm8_VKdltXy8lrIW14mR2kgYIoaBPD7LO2uAN6rjhrfCX37Pfa6r85QPpSwSPh4V6eW_bKlns$
> >> >> >
> >> >> >> > <
> >> >> >>
> >> >>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR68. Could we make the usage of topicId and topicName in the
> >> RPCs
> >> >> >> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> >> >> > BumpLeaderEpochsReponse uses TopicName.
> >> >> >> >
> >> >> >> > Yes, we have made:
> >> >> >> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use
> TopicName.
> >> >> >> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse
> use
> >> >> >> > TopicName.
> >> >> >> > 3. MirrorTopicStateChangeRecord uses TopicName because we will
> do
> >> >> >> > validation before writing the metadata log.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR69. MIRROR_PID_RESET:
> >> >> >> > > JR69.1 Could you specify the value of the control record's
> type?
> >> >> >> >
> >> >> >> > It’s MIRROR_PID_RESET(7). KIP updated.
> >> >> >> >
> >> >> >> > > JR69.2 How is the SourceClusterId field used?
> >> >> >> >
> >> >> >> > I’m also surprised to see this field in the
> MirrorPidResetRecord.
> >> I
> >> >> >> traced
> >> >> >> > back to the internal PR we added for this field, and confirmed
> it
> >> is
> >> >> not
> >> >> >> > used since it was added. Sorry for the confusion and nice catch!
> >> KIP
> >> >> >> > updated.
> >> >> >> >
> >> >> >> > > JR70. MirrorPartitionState: type should be "data" not
> >> >> >> "coordinator-key",
> >> >> >> > right? Ditto for LastMirrorEpochs.
> >> >> >> > We store both MirrorPartitionState and LastMirrorEpochs records
> >> into
> >> >> >> > internal __mirror_state topic, and rely on Coordinator Runtime
> to
> >> >> >> > write/read the records, like how `OffsetCommit` and
> >> `GroupMetadata`
> >> >> >> work.
> >> >> >> > So I think we should keep them as
> >> >> "coordinator-key"/"coordinator-value"
> >> >> >> > type.
> >> >> >> >
> >> >> >> >
> >> >> >> > Thank you,
> >> >> >> > Luke
> >> >> >> >
> >> >> >> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <
> >> >> dev@kafka.apache.org>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Hi, Luke,
> >> >> >> >>
> >> >> >> >> Thanks for the reply.
> >> >> >> >>
> >> >> >> >> JR51. "ULE_LOG_TRUNCATION When
> >> >> mirror.support.unclean.leader.election
> >> >> >> is
> >> >> >> >> enabled and the destination leader has log truncation, the
> state
> >> >> will
> >> >> >> move
> >> >> >> >> from MIRRORING to this state."
> >> >> >> >> This still suggests that divergeEpoch is only handled if
> >> >> >> >> mirror.support.unclean.leader.election
> >> >> >> >> is enabled.
> >> >> >> >>
> >> >> >> >> JR55. Should the mirror state and the mirror lags always be
> >> exposed
> >> >> >> >> together? If so, we can remove IncludeMirrorState and
> >> >> IncludeMirrorLags
> >> >> >> >> from DescribeClusterMirrorsRequest. If not, it's better to use
> >> two
> >> >> >> >> separate
> >> >> >> >> RPCs.
> >> >> >> >>
> >> >> >> >> DescribeClusterMirrorsRequest should be used in the Describe
> >> Mirror
> >> >> >> flow,
> >> >> >> >> right? Currently it's missing. Also, I assume the admin client
> >> >> refers
> >> >> >> to
> >> >> >> >> the one in the kafka-cluster-mirrors tool. But it seems that
> >> >> sometimes
> >> >> >> it
> >> >> >> >> also refers to the admin client on the broker? Perhaps we can
> use
> >> >> 'the
> >> >> >> >> tool' instead to make it clear.
> >> >> >> >>
> >> >> >> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the
> >> broker,
> >> >> >> >> instead of just to the controller?
> >> >> >> >>
> >> >> >> >> JR59. Start Mirror Topics:
> >> >> >> >> JR59.1 "During this state, the broker queries the source
> cluster
> >> for
> >> >> >> last
> >> >> >> >> mirror epochs".
> >> >> >> >> Could you describe which RPC is issued to find the LME?
> >> >> >> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
> >> request
> >> >> >> >> doesn't
> >> >> >> >> contain a replication factor.
> >> >> >> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used
> >> by a
> >> >> >> >> different topic name"
> >> >> >> >> Should we also return this error when the topic name is used
> as a
> >> >> >> regular
> >> >> >> >> topic?
> >> >> >> >>
> >> >> >> >> JR60. CreateClusterMirrorRequest: When will it receive an
> >> >> >> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> >> >> >> CLUSTER_MIRROR_ALREADY_EXISTS?
> >> >> >> >>
> >> >> >> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we
> just
> >> >> >> remove
> >> >> >> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> >> >> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to
> >> describe
> >> >> the
> >> >> >> >> specific state causing the error.
> >> >> >> >>
> >> >> >> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in
> some
> >> >> >> requests
> >> >> >> >> (e.g. StopMirrorTopics), but not others (e.g.
> StartMirrorTopics).
> >> >> >> Could we
> >> >> >> >> make them consistent?
> >> >> >> >>
> >> >> >> >> JR63. StopMirrorTopics:
> >> >> >> >> JR63.1 "The broker validates that all target topic partitions
> >> are in
> >> >> >> >> either
> >> >> >> >> LOG_TRUNCATION or MIRRORING or FAILED state."
> >> >> >> >> Is this accurate? LOG_TRUNCATION can't transition to STOP.
> PAUSED
> >> >> can.
> >> >> >> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error
> >> since
> >> >> >> PAUSED
> >> >> >> >> -> STOPPED is valid?
> >> >> >> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
> >> stopped
> >> >> >> >> state"
> >> >> >> >> This is weird. The reason to issue StopMirrorTopics is that
> it's
> >> >> not in
> >> >> >> >> stopped state.
> >> >> >> >>
> >> >> >> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
> >> >> derived
> >> >> >> >> from SourceOffset and DestinationOffset.
> >> >> >> >>
> >> >> >> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems
> >> that
> >> >> we
> >> >> >> >> could just set the partition level state to STOPPED.
> >> >> >> >>
> >> >> >> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to
> be
> >> >> sent
> >> >> >> to
> >> >> >> >> the controller?
> >> >> >> >>
> >> >> >> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and
> >> >> StateEpoch? It
> >> >> >> >> seems both are used for fencing and fencing based on the latter
> >> is
> >> >> >> >> stronger.
> >> >> >> >>
> >> >> >> >> JR68. Could we make the usage of topicId and topicName in the
> >> RPCs
> >> >> >> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> >> >> >> BumpLeaderEpochsReponse uses TopicName.
> >> >> >> >>
> >> >> >> >> JR69. MIRROR_PID_RESET:
> >> >> >> >> JR69.1 Could you specify the value of the control record's
> type?
> >> >> >> >> JR69.2 How is the SourceClusterId field used?
> >> >> >> >>
> >> >> >> >> JR70. MirrorPartitionState: type should be "data" not
> >> >> >> "coordinator-key",
> >> >> >> >> right? Ditto for LastMirrorEpochs.
> >> >> >> >>
> >> >> >> >> Jun
> >> >> >> >>
> >> >> >> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com>
> >> >> wrote:
> >> >> >> >>
> >> >> >> >> > Hi Jun,
> >> >> >> >> >
> >> >> >> >> > Thanks for the comments.
> >> >> >> >> > Answering them below.
> >> >> >> >> >
> >> >> >> >> > > JR36. Sounds good. Should we remove
> >> >> >> >> MirrorLeaderEpochExceededException?
> >> >> >> >> >
> >> >> >> >> > Sorry, it’s a confusing name. The
> >> >> MirrorLeaderEpochExceededException
> >> >> >> is
> >> >> >> >> > used and thrown when the fetched mirror batch leader epoch >
> >> local
> >> >> >> >> leader
> >> >> >> >> > epoch in destination cluster. And then we’ll stop mirroring
> and
> >> >> move
> >> >> >> the
> >> >> >> >> > partition to EPOCH_FENSING state to wait for local leader
> epoch
> >> >> >> bump. I
> >> >> >> >> > think since we don’t have the concept of `MirrorLeaderEpoch`
> >> >> anymore,
> >> >> >> >> the
> >> >> >> >> > `MirrorLeaderEpochExceededException` should not cause
> confusion
> >> >> now.
> >> >> >> >> I’ll
> >> >> >> >> > keep the name for the exception.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR51. Sounds good. Do we plan to always set
> lastFetchedEpoch
> >> in
> >> >> the
> >> >> >> >> > mirror
> >> >> >> >> > fetch request issued to the source cluster, independent of
> the
> >> >> value
> >> >> >> of
> >> >> >> >> > mirror.support.unclean.leader.election? It's needed to detect
> >> >> >> divergence
> >> >> >> >> > during failover. However, if it's also set when
> >> >> >> >> > mirror.support.unclean.leader.election=false,
> >> >> >> >> > an unclean leader election in the source cluster can halt the
> >> >> >> mirroring
> >> >> >> >> > since we don’t do anything with the returned divergeEpoch in
> >> that
> >> >> >> mode
> >> >> >> >> > and continue
> >> >> >> >> > the next fetch.
> >> >> >> >> >
> >> >> >> >> > > Do we plan to always set lastFetchedEpoch in the mirror
> >> >> >> >> > fetch request issued to the source cluster, independent of
> the
> >> >> value
> >> >> >> of
> >> >> >> >> > Mirror.support.unclean.leader.election?
> >> >> >> >> >
> >> >> >> >> > Yes, we always set lastFetchedEpoch in the mirror fetch
> >> request.
> >> >> >> >> >
> >> >> >> >> > > However, if it's also set when
> >> >> >> >> > mirror.support.unclean.leader.election=false,
> >> >> >> >> > an unclean leader election in the source cluster can halt the
> >> >> >> mirroring
> >> >> >> >> > since we don’t do anything with the returned divergeEpoch in
> >> that
> >> >> >> mode
> >> >> >> >> > and continue the next fetch.
> >> >> >> >> >
> >> >> >> >> > Sorry I didn’t make it clear. When
> >> >> >> >> > “mirror.support.unclean.leader.election=false” and a
> >> >> “divergeEpoch”
> >> >> >> is
> >> >> >> >> set
> >> >> >> >> > in fetch response, the leader node will handle the
> divergeEpoch
> >> >> and
> >> >> >> then
> >> >> >> >> > continue the next fetch as usual. The difference with
> >> >> >> >> > “mirror.support.unclean.leader.election=true” is that “should
> >> we
> >> >> wait
> >> >> >> >> for
> >> >> >> >> > ALL replicas caught up (complete truncation)”. Without the
> >> >> waiting,
> >> >> >> it’s
> >> >> >> >> > possible the inconsistent log issue in KAFKA-18723 could
> happen
> >> >> >> between
> >> >> >> >> > leader and follower nodes in the destination cluster because
> we
> >> >> don’t
> >> >> >> >> have
> >> >> >> >> > “MirrorLeaderEpoch” concept now to reject higher leader epoch
> >> >> >> batches.
> >> >> >> >> That
> >> >> >> >> > is, we won't halt the mirroring if "divergeEpoch" is set but
> >> >> >> >> > “mirror.support.unclean.leader.election=false”.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR55. The DescribeClusterMirrorsRequest still returns both
> >> the
> >> >> >> mirror
> >> >> >> >> > partition state and the mirror partition leader state. The
> >> >> problem is
> >> >> >> >> that
> >> >> >> >> > if a caller only needs one of the state, the other state
> >> becomes
> >> >> >> >> redundant.
> >> >> >> >> > For example, when starting the mirror, to obtain LME, we only
> >> need
> >> >> >> the
> >> >> >> >> > mirror partition state. Here is an alternative design: We
> >> expose
> >> >> the
> >> >> >> >> mirror
> >> >> >> >> > partition state through ReadMirrorState. The client will use
> >> the
> >> >> >> >> > FindCoordinatorRequest
> >> >> >> >> > to find the right mirror partition coordinator to issue that
> >> >> >> request. We
> >> >> >> >> > expose the mirror partition leader state through the
> >> >> >> >> ReadClusterMirrorLags
> >> >> >> >> > request. The client can use the metadata request to find the
> >> >> >> partition
> >> >> >> >> > leader to issue that request.
> >> >> >> >> >
> >> >> >> >> > > In the describing mirror flow, the admin client first
> issues
> >> the
> >> >> >> >> > ListMirrorRequest to find all the mirrors and the topics
> being
> >> >> >> >> mirrored. It
> >> >> >> >> > then issues the metadata request to find the partitions for
> >> those
> >> >> >> topics
> >> >> >> >> > and issues a FindCoordinatorRequest to find the mirror
> >> partition
> >> >> >> state
> >> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the
> mirror
> >> >> >> partition
> >> >> >> >> > state coordinator and ReadClusterMirrorLags to the partition
> >> >> leader.
> >> >> >> >> >
> >> >> >> >> > > In the starting mirror flow, the partition leader in the
> >> >> >> destination
> >> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name
> >> based
> >> >> on
> >> >> >> its
> >> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to find
> the
> >> >> >> mirror
> >> >> >> >> > partition state coordinators. Finally, it issues
> >> ReadMirrorState
> >> >> to
> >> >> >> the
> >> >> >> >> > mirror partition state coordinator to get the LME.
> >> >> >> >> >
> >> >> >> >> > > We can also improve the ListMirror request to accept a
> filter
> >> >> >> based on
> >> >> >> >> > mirror name, source cluster ID and topic name.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Answers:
> >> >> >> >> >
> >> >> >> >> > > The DescribeClusterMirrorsRequest still returns both the
> >> mirror
> >> >> >> >> > partition state and the mirror partition leader state. The
> >> >> problem is
> >> >> >> >> that
> >> >> >> >> > if a caller only needs one of the state, the other state
> >> becomes
> >> >> >> >> redundant.
> >> >> >> >> > For example, when starting the mirror, to obtain LME, we only
> >> need
> >> >> >> the
> >> >> >> >> > mirror partition state.
> >> >> >> >> >
> >> >> >> >> > Good point!
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > Here is an alternative design: We expose the mirror
> >> >> >> >> > partition state through ReadMirrorState. The client will use
> >> the
> >> >> >> >> > FindCoordinatorRequest
> >> >> >> >> > to find the right mirror partition coordinator to issue that
> >> >> >> request. We
> >> >> >> >> > expose the mirror partition leader state through the
> >> >> >> >> ReadClusterMirrorLags
> >> >> >> >> > request. The client can use the metadata request to find the
> >> >> >> partition
> >> >> >> >> > leader to issue that request.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Sounds good. But we try not to expose
> >> >> >> >> > ReadMirrorState/ReadClusterMirrorLags directly to the admin
> >> client
> >> >> >> >> because
> >> >> >> >> > users might get confused about these 2 requests, and they are
> >> also
> >> >> >> both
> >> >> >> >> > related to describeClusterMirror. I think we can add 2
> >> additional
> >> >> >> >> fields to
> >> >> >> >> > describeClusterMirrorRequest: “includeMirrorStates” and
> >> >> >> >> > “includeMirrorLags”. So the admin client can include either
> >> one or
> >> >> >> both
> >> >> >> >> of
> >> >> >> >> > them in the DescrbieClusterMirrorRequest based on the
> provided
> >> >> >> options.
> >> >> >> >> KIP
> >> >> >> >> > is updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > In the describing mirror flow, the admin client first
> issues
> >> the
> >> >> >> >> > ListMirrorRequest to find all the mirrors and the topics
> being
> >> >> >> >> mirrored. It
> >> >> >> >> > then issues the metadata request to find the partitions for
> >> those
> >> >> >> topics
> >> >> >> >> > and issues a FindCoordinatorRequest to find the mirror
> >> partition
> >> >> >> state
> >> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the
> mirror
> >> >> >> partition
> >> >> >> >> > state coordinator and ReadClusterMirrorLags to the partition
> >> >> leader.
> >> >> >> >> >
> >> >> >> >> > Makes sense. KIP is updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > In the starting mirror flow, the partition leader in the
> >> >> >> destination
> >> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name
> >> based
> >> >> on
> >> >> >> its
> >> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to find
> the
> >> >> >> mirror
> >> >> >> >> > partition state coordinators. Finally, it issues
> >> ReadMirrorState
> >> >> to
> >> >> >> the
> >> >> >> >> > mirror partition state coordinator to get the LME.
> >> >> >> >> >
> >> >> >> >> > Make sense. So we can send DescribeClusterMirrorRequest with
> >> >> >> >> > “includeMirrorStates” only.
> >> >> >> >> >
> >> >> >> >> > > We can also improve the ListMirror request to accept a
> filter
> >> >> >> based on
> >> >> >> >> > mirror name, source cluster ID and topic name.
> >> >> >> >> >
> >> >> >> >> > Good suggestion, just like what ListGroupsRequest does.
> >> >> >> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
> >> >> >> >> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest.
> >> KIP
> >> >> >> >> updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR56. Could you list the ACL needed for each new RPC?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Yes, it’s already listed in “Security Control” section.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR57. In the command workflow, it sometimes says "The user
> >> >> sends"
> >> >> >> and
> >> >> >> >> > some
> >> >> >> >> > other times says "The admin client sends". It's better to use
> >> "The
> >> >> >> admin
> >> >> >> >> > client" consistently in all those places.
> >> >> >> >> >
> >> >> >> >> > Agree. KIP updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Thank you,
> >> >> >> >> > Luke
> >> >> >> >> >
> >> >> >> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <
> >> >> dev@kafka.apache.org
> >> >> >> >
> >> >> >> >> > wrote:
> >> >> >> >> >
> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >>
> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >>
> >> >> >> >> >> JR36. Sounds good. Should we remove
> >> >> >> MirrorLeaderEpochExceededException?
> >> >> >> >> >>
> >> >> >> >> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch
> >> in
> >> >> the
> >> >> >> >> mirror
> >> >> >> >> >> fetch request issued to the source cluster, independent of
> the
> >> >> >> value of
> >> >> >> >> >> mirror.support.unclean.leader.election? It's needed to
> detect
> >> >> >> >> divergence
> >> >> >> >> >> during failover. However, if it's also set when
> >> >> >> >> >> mirror.support.unclean.leader.election=false,
> >> >> >> >> >> an unclean leader election in the source cluster can halt
> the
> >> >> >> mirroring
> >> >> >> >> >> since we don’t do anything with the returned divergeEpoch in
> >> that
> >> >> >> mode
> >> >> >> >> >> and continue
> >> >> >> >> >> the next fetch.
> >> >> >> >> >>
> >> >> >> >> >> JR55. The DescribeClusterMirrorsRequest still returns both
> the
> >> >> >> mirror
> >> >> >> >> >> partition state and the mirror partition leader state. The
> >> >> problem
> >> >> >> is
> >> >> >> >> that
> >> >> >> >> >> if a caller only needs one of the state, the other state
> >> becomes
> >> >> >> >> >> redundant.
> >> >> >> >> >> For example, when starting the mirror, to obtain LME, we
> only
> >> >> need
> >> >> >> the
> >> >> >> >> >> mirror partition state. Here is an alternative design: We
> >> expose
> >> >> the
> >> >> >> >> >> mirror
> >> >> >> >> >> partition state through ReadMirrorState. The client will use
> >> the
> >> >> >> >> >> FindCoordinatorRequest
> >> >> >> >> >> to find the right mirror partition coordinator to issue that
> >> >> >> request.
> >> >> >> >> We
> >> >> >> >> >> expose the mirror partition leader state through the
> >> >> >> >> ReadClusterMirrorLags
> >> >> >> >> >> request. The client can use the metadata request to find the
> >> >> >> partition
> >> >> >> >> >> leader to issue that request.
> >> >> >> >> >>
> >> >> >> >> >> In the describing mirror flow, the admin client first issues
> >> the
> >> >> >> >> >> ListMirrorRequest to find all the mirrors and the topics
> being
> >> >> >> >> mirrored.
> >> >> >> >> >> It
> >> >> >> >> >> then issues the metadata request to find the partitions for
> >> those
> >> >> >> >> topics
> >> >> >> >> >> and issues a FindCoordinatorRequest to find the mirror
> >> partition
> >> >> >> state
> >> >> >> >> >> coordinators. Finally, it issues ReadMirrorState to the
> mirror
> >> >> >> >> partition
> >> >> >> >> >> state coordinator and ReadClusterMirrorLags to the partition
> >> >> leader.
> >> >> >> >> >>
> >> >> >> >> >> In the starting mirror flow, the partition leader in the
> >> >> destination
> >> >> >> >> >> cluster issues a ListMirrorRequest to find the mirror name
> >> based
> >> >> on
> >> >> >> its
> >> >> >> >> >> cluster id. It then issues a FindCoordinatorRequest to find
> >> the
> >> >> >> mirror
> >> >> >> >> >> partition state coordinators. Finally, it issues
> >> ReadMirrorState
> >> >> to
> >> >> >> the
> >> >> >> >> >> mirror partition state coordinator to get the LME.
> >> >> >> >> >>
> >> >> >> >> >> We can also improve the ListMirror request to accept a
> filter
> >> >> based
> >> >> >> on
> >> >> >> >> >> mirror name, source cluster ID and topic name.
> >> >> >> >> >>
> >> >> >> >> >> JR56. Could you list the ACL needed for each new RPC?
> >> >> >> >> >>
> >> >> >> >> >> JR57. In the command workflow, it sometimes says "The user
> >> sends"
> >> >> >> and
> >> >> >> >> some
> >> >> >> >> >> other times says "The admin client sends". It's better to
> use
> >> >> "The
> >> >> >> >> admin
> >> >> >> >> >> client" consistently in all those places.
> >> >> >> >> >>
> >> >> >> >> >> Jun
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <showuon@gmail.com
> >
> >> >> wrote:
> >> >> >> >> >>
> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >
> >> >> >> >> >> > Thanks for your comments.
> >> >> >> >> >> > Answering them below.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> >> >> >> >> specifically
> >> >> >> >> >> > designed for the unclean leader election support in
> Cluster
> >> >> >> Mirroring
> >> >> >> >> >> > because the unclean leader election will break the
> >> assumption
> >> >> that
> >> >> >> >> the
> >> >> >> >> >> > committed data won’t be truncated."
> >> >> >> >> >> > If the source cluster has an unclean leader election, the
> >> >> >> destination
> >> >> >> >> >> > cluster will detect the inconsistency and truncate the log
> >> at
> >> >> the
> >> >> >> >> >> > destination leader. We can just
> >> >> >> >> >> > extend mirror.support.unclean.leader.election to cover
> this
> >> >> case,
> >> >> >> >> i.e.,
> >> >> >> >> >> we
> >> >> >> >> >> > wait until all replicas' log end offsets are reduced to
> the
> >> >> target
> >> >> >> >> >> offset
> >> >> >> >> >> > before resuming mirroring. This avoids the inconsistency.
> >> >> >> >> >> >
> >> >> >> >> >> > If the destination cluster has an unclean leader election,
> >> it
> >> >> will
> >> >> >> >> >> > re-fetch the same data from the source. There won't be any
> >> >> >> >> inconsistency
> >> >> >> >> >> > either.
> >> >> >> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Yes, that works, too. I agree this is better because the
> >> >> unclean
> >> >> >> >> leader
> >> >> >> >> >> > election is rare, we don't want to occupy a field just for
> >> it.
> >> >> >> >> >> >
> >> >> >> >> >> > So to achieve this, we have to:
> >> >> >> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
> >> >> >> >> >> > 2. When the destination cluster leader completes a log
> >> >> truncation
> >> >> >> >> after
> >> >> >> >> >> a
> >> >> >> >> >> > fetch response due to divergeEpoch, we know it must be
> >> caused
> >> >> by
> >> >> >> >> unclean
> >> >> >> >> >> > leader election in the source cluster because we only
> fetch
> >> >> >> committed
> >> >> >> >> >> > records. If
> “mirror.support.unclean.leader.election=false”,
> >> we
> >> >> >> don’t
> >> >> >> >> do
> >> >> >> >> >> > anything and continue the next fetch. If
> >> >> >> >> >> > “mirror.support.unclean.leader.election=true”, we remove
> the
> >> >> >> mirror
> >> >> >> >> >> fetcher
> >> >> >> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
> >> >> >> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas
> >> to
> >> >> be
> >> >> >> >> caught
> >> >> >> >> >> up
> >> >> >> >> >> > into ISR and reduced to the expected LEO.
> >> >> >> >> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
> >> >> >> >> >> >
> >> >> >> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in
> Fetch
> >> >> API.
> >> >> >> >> >> > KIP is updated.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR51. Make sense. To truncate based on LME, we need to
> >> issue
> >> >> >> >> another
> >> >> >> >> >> RPC
> >> >> >> >> >> > to translate the LME to an offset. It would be useful to
> >> >> include
> >> >> >> >> that in
> >> >> >> >> >> > the workflow description.
> >> >> >> >> >> >
> >> >> >> >> >> > About the additional RPC to translate the LME to the
> offset
> >> >> (i.e.
> >> >> >> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on
> this
> >> >> to do
> >> >> >> >> the
> >> >> >> >> >> > offset translation. We use the local leader epoch cache in
> >> the
> >> >> >> >> >> destination
> >> >> >> >> >> > cluster to translate the epoch to save the RPC call. This
> >> works
> >> >> >> >> because
> >> >> >> >> >> of
> >> >> >> >> >> > an invariant the LME truncation protocol maintains: for
> any
> >> >> leader
> >> >> >> >> >> epoch ≤
> >> >> >> >> >> > LME, that epoch's data belongs to exactly one cluster,
> >> either
> >> >> it
> >> >> >> >> >> originated
> >> >> >> >> >> > there, or it was mirrored. This holds even across repeated
> >> >> mirror
> >> >> >> ->
> >> >> >> >> >> > reverse-mirror -> mirror cycles, since each direction's
> >> >> >> >> LOG_TRUNCATION
> >> >> >> >> >> > phase re-establishes the invariant before that cluster
> >> starts
> >> >> >> >> mirroring.
> >> >> >> >> >> > With this invariant, the replication protocol can find out
> >> the
> >> >> >> >> diverge
> >> >> >> >> >> > point and truncate the log correctly when mirroring. For
> >> >> example:
> >> >> >> >> >> >
> >> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> >> > Offset 2: A (LE: 0)
> >> >> >> >> >> > Offset 3: A (LE: 1)
> >> >> >> >> >> >
> >> >> >> >> >> > Cluster A crashes while Cluster B mirrors to offset 1.
> >> Failover
> >> >> >> to B
> >> >> >> >> and
> >> >> >> >> >> > store the LME=0 in cluster B. When cluster A recovers and
> >> >> starts
> >> >> >> the
> >> >> >> >> >> > reverse mirroring, it’ll query the LME from B and get back
> >> >> LME: 0.
> >> >> >> >> Then
> >> >> >> >> >> > cluster A gets the end offset for epoch 0 from
> >> >> leaderEpochCache,
> >> >> >> and
> >> >> >> >> >> gets
> >> >> >> >> >> > offset 2. So offset 3 is truncated. The log is like this
> >> now:
> >> >> >> >> >> >
> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE:
> 10)
> >> >> >> >> >> > Offset 3: B
> (LE:
> >> >> 10)
> >> >> >> >> >> >
> >> >> >> >> >> > When mirror starts, cluster A sends the fetch request
> with {
> >> >> >> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the
> >> >> divergingEpoch
> >> >> >> >> check
> >> >> >> >> >> in
> >> >> >> >> >> > the cluster B queries the endOffset for leader epoch 0,
> and
> >> >> found
> >> >> >> the
> >> >> >> >> >> end
> >> >> >> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so
> >> returns
> >> >> >> fetch
> >> >> >> >> >> > response with divergingEpoch info to cluster A for log
> >> >> >> truncation. In
> >> >> >> >> >> the
> >> >> >> >> >> > end, offset 2 in cluster A will be truncated and then log
> >> >> >> converges
> >> >> >> >> with
> >> >> >> >> >> > cluster B.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a
> bit
> >> >> >> strange
> >> >> >> >> >> > because it mixes querying the mirror partition state with
> >> >> querying
> >> >> >> >> the
> >> >> >> >> >> > mirror partition leader. Another option is to use
> >> >> >> >> >> ReadMirrorStatesRequest
> >> >> >> >> >> > to query the mirror partition state and a new request to
> >> query
> >> >> the
> >> >> >> >> >> mirror
> >> >> >> >> >> > partition leader. If we want the adminClient to query the
> >> right
> >> >> >> >> >> ClusterMirrorCoordinator
> >> >> >> >> >> > directly, we can extend the FindCoordinatorRequest to
> >> support
> >> >> >> that.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Good suggestion. Yes, we need to fan out to all brokers
> >> >> because we
> >> >> >> >> have
> >> >> >> >> >> to
> >> >> >> >> >> > collect info from mirror partition leaders and mirror
> >> >> >> coordinators.
> >> >> >> >> If
> >> >> >> >> >> we
> >> >> >> >> >> > create a new API to contact mirror partition leaders, it
> >> will
> >> >> make
> >> >> >> >> the
> >> >> >> >> >> API
> >> >> >> >> >> > cleaner and clearer. So we will do this:
> >> >> >> >> >> >
> >> >> >> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields
> >> from
> >> >> >> >> >> > DescribeClusterMirror request/response
> >> >> >> >> >> >
> >> >> >> >> >> > b. Add a new API: ReadClusterMirrorLags API to request for
> >> >> >> >> SourceOffset,
> >> >> >> >> >> > DestinationOffset, and Lag info.
> >> >> >> >> >> >
> >> >> >> >> >> > The workflow for admin client describeClusterMirror will
> be
> >> >> like
> >> >> >> >> this:
> >> >> >> >> >> > 1. adminClient sends describeClusterMirror to the
> >> >> leastLoadedNode
> >> >> >> >> broker
> >> >> >> >> >> > 2. The received broker collects the current mirrored
> >> >> partitions,
> >> >> >> and
> >> >> >> >> >> > groups them by coordinator node and by partition leader
> >> node.
> >> >> >> >> >> > 3. The broker sends ReadMirrorState request to the
> >> coordinator
> >> >> >> nodes
> >> >> >> >> to
> >> >> >> >> >> > get the mirror state related info
> >> >> >> >> >> > 4. The broker sends ReadClusterMirrorLags request to the
> >> mirror
> >> >> >> >> >> partition
> >> >> >> >> >> > leader nodes to get the mirror lag, source offset,
> >> destination
> >> >> >> offset
> >> >> >> >> >> > (which updates on every fetch response).
> >> >> >> >> >> > 5. The broker merge all the response into
> >> describeClusterMirror
> >> >> >> >> response
> >> >> >> >> >> > and send to admin client
> >> >> >> >> >> >
> >> >> >> >> >> > KIP is updated. Thanks for the suggestion.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR55. For __mirror_state topic, could we describe
> whether
> >> it
> >> >> is
> >> >> >> >> >> > compacted?
> >> >> >> >> >> >
> >> >> >> >> >> > Yes, it is a compacted topic. I made it much clearer in
> the
> >> >> >> >> >> > “ClusterMirrorCoordinator” section.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Thank you,
> >> >> >> >> >> > Luke
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
> >> >> >> dev@kafka.apache.org
> >> >> >> >> >
> >> >> >> >> >> > wrote:
> >> >> >> >> >> >
> >> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >> >>
> >> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> >> >> >> >> specifically
> >> >> >> >> >> >> designed for the unclean leader election support in
> Cluster
> >> >> >> >> Mirroring
> >> >> >> >> >> >> because the unclean leader election will break the
> >> assumption
> >> >> >> that
> >> >> >> >> the
> >> >> >> >> >> >> committed data won’t be truncated."
> >> >> >> >> >> >> If the source cluster has an unclean leader election, the
> >> >> >> >> destination
> >> >> >> >> >> >> cluster will detect the inconsistency and truncate the
> log
> >> at
> >> >> the
> >> >> >> >> >> >> destination leader. We can just
> >> >> >> >> >> >> extend mirror.support.unclean.leader.election to cover
> this
> >> >> case,
> >> >> >> >> >> i.e., we
> >> >> >> >> >> >> wait until all replicas' log end offsets are reduced to
> the
> >> >> >> target
> >> >> >> >> >> offset
> >> >> >> >> >> >> before resuming mirroring. This avoids the inconsistency.
> >> >> >> >> >> >>
> >> >> >> >> >> >> If the destination cluster has an unclean leader
> election,
> >> it
> >> >> >> will
> >> >> >> >> >> >> re-fetch
> >> >> >> >> >> >> the same data from the source. There won't be any
> >> >> inconsistency
> >> >> >> >> either.
> >> >> >> >> >> >>
> >> >> >> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR51. Make sense. To truncate based on LME, we need to
> >> issue
> >> >> >> another
> >> >> >> >> >> RPC
> >> >> >> >> >> >> to
> >> >> >> >> >> >> translate the LME to an offset. It would be useful to
> >> include
> >> >> >> that
> >> >> >> >> in
> >> >> >> >> >> the
> >> >> >> >> >> >> workflow description.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit
> >> >> strange
> >> >> >> >> >> because
> >> >> >> >> >> >> it mixes querying the mirror partition state with
> querying
> >> the
> >> >> >> >> mirror
> >> >> >> >> >> >> partition leader. Another option is to use
> >> >> >> ReadMirrorStatesRequest
> >> >> >> >> to
> >> >> >> >> >> >> query
> >> >> >> >> >> >> the mirror partition state and a new request to query the
> >> >> mirror
> >> >> >> >> >> partition
> >> >> >> >> >> >> leader. If we want the adminClient to query the right
> >> >> >> >> >> >> ClusterMirrorCoordinator
> >> >> >> >> >> >> directly, we can extend the FindCoordinatorRequest to
> >> support
> >> >> >> that.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR55. For __mirror_state topic, could we describe whether
> >> it
> >> >> is
> >> >> >> >> >> compacted?
> >> >> >> >> >> >>
> >> >> >> >> >> >> Jun
> >> >> >> >> >> >>
> >> >> >> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <
> >> showuon@gmail.com>
> >> >> >> wrote:
> >> >> >> >> >> >>
> >> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Thanks for the comments.
> >> >> >> >> >> >> > Answering them below:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR31.3 It makes sense for the new partition leader to
> >> >> query
> >> >> >> the
> >> >> >> >> >> >> current
> >> >> >> >> >> >> > mirror state from the local or remote
> >> MirrorMetadataManager
> >> >> >> that
> >> >> >> >> owns
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > partition's state. However, the wording in the KIP is
> >> >> >> confusing.
> >> >> >> >> >> >> > MirrorMetadataManager
> >> >> >> >> >> >> > mirror partition state cache should be populated only
> by
> >> one
> >> >> >> way,
> >> >> >> >> >> which
> >> >> >> >> >> >> is
> >> >> >> >> >> >> > replaying __mirror_state partitions this broker leads.
> By
> >> >> >> saying
> >> >> >> >> >> "and by
> >> >> >> >> >> >> > fetching state from remote coordinators via
> >> ReadMirrorStates
> >> >> >> >> RPCs.",
> >> >> >> >> >> >> it's
> >> >> >> >> >> >> > as if the mirror partition state cache on each broker
> >> stores
> >> >> >> the
> >> >> >> >> >> state
> >> >> >> >> >> >> for
> >> >> >> >> >> >> > all mirror partitions.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > You’re right. KIP is updated to make it clear.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR36. I understand the intention now, but I still
> don't
> >> >> >> think we
> >> >> >> >> >> need
> >> >> >> >> >> >> > MirrorLeaderEpoch. Let's
> >> >> >> >> >> >> > assume the common case: no unclean leader election in
> the
> >> >> >> source
> >> >> >> >> >> cluster
> >> >> >> >> >> >> > during mirroring. Mirroring only fetches the committed
> >> data
> >> >> >> from
> >> >> >> >> the
> >> >> >> >> >> >> > source, which never changes if obtained a second time.
> In
> >> >> your
> >> >> >> >> >> example
> >> >> >> >> >> >> > image, follower broker 1 in the destination cluster can
> >> >> never
> >> >> >> >> obtain
> >> >> >> >> >> the
> >> >> >> >> >> >> > record B(0) because it's never committed in the source
> >> and
> >> >> >> thus is
> >> >> >> >> >> never
> >> >> >> >> >> >> > exposed to the destination leader. It's possible for
> >> broker
> >> >> 1
> >> >> >> to
> >> >> >> >> >> first
> >> >> >> >> >> >> get
> >> >> >> >> >> >> > D(1) which is truncated later when broker 0 follows a
> new
> >> >> >> leader.
> >> >> >> >> But
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > new leader will refetch the same D(1) from the source
> >> >> cluster.
> >> >> >> So,
> >> >> >> >> >> it's
> >> >> >> >> >> >> ok
> >> >> >> >> >> >> > for broker 1 to keep D(1).
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Yes, you are correct. Because mirroring only fetches
> the
> >> >> >> committed
> >> >> >> >> >> data
> >> >> >> >> >> >> > from the source, the MirrorLeaderEpoch is unnecessary
> for
> >> >> >> normal
> >> >> >> >> >> >> mirroring.
> >> >> >> >> >> >> > Thanks for the clarification. So I think we can say,
> the
> >> >> >> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the
> >> unclean
> >> >> >> >> leader
> >> >> >> >> >> >> > election support in Cluster Mirroring because the
> unclean
> >> >> >> leader
> >> >> >> >> >> >> election
> >> >> >> >> >> >> > will break the assumption that the committed data won’t
> >> be
> >> >> >> >> truncated.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Given that, we could gate the MirrorLeaderEpoch field
> by
> >> >> only
> >> >> >> >> >> setting it
> >> >> >> >> >> >> > in the Fetch request when
> >> >> >> >> >> "mirror.support.unclean.leader.election=true"
> >> >> >> >> >> >> > (false by default), and skip it entirely for normal
> >> >> mirroring.
> >> >> >> >> What
> >> >> >> >> >> do
> >> >> >> >> >> >> you
> >> >> >> >> >> >> > think?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR51. Sounds good. Why do we choose to store the last
> >> >> mirror
> >> >> >> >> epoch
> >> >> >> >> >> >> > instead
> >> >> >> >> >> >> > of a last mirror offset?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Yes, originally we stored the “last mirror offset”
> >> instead
> >> >> of
> >> >> >> >> “last
> >> >> >> >> >> >> mirror
> >> >> >> >> >> >> > epoch”. But later, we found it doesn’t work for unclean
> >> >> leader
> >> >> >> >> >> election
> >> >> >> >> >> >> > cases. (yes, `unclean leader election` again :) ) For
> >> >> example:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Cluster B is mirroring from A. And A crashes and
> >> failover to
> >> >> >> B. B
> >> >> >> >> >> stores
> >> >> >> >> >> >> > the “last mirror offset” as 1.
> >> >> >> >> >> >> > When cluster A recovers, somehow the unclean leader
> >> election
> >> >> >> >> happens,
> >> >> >> >> >> >> log
> >> >> >> >> >> >> > is empty in the new leader, and then some more
> leadership
> >> >> >> changes
> >> >> >> >> to
> >> >> >> >> >> >> make
> >> >> >> >> >> >> > the leader epoch become 10. Then it starts the reverse
> >> >> mirror
> >> >> >> >> from B.
> >> >> >> >> >> >> The
> >> >> >> >> >> >> > log looks like this:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> >> >> >> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> >> >> >> >> >> >> > Offset 2:
> >> >> pid_res
> >> >> >> (LE:
> >> >> >> >> >> 10)
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > When A starts mirroring from B, log is truncated to
> last
> >> >> mirror
> >> >> >> >> >> offset:
> >> >> >> >> >> >> 1
> >> >> >> >> >> >> > (no-op), and then fetch from cluster B. The 1st fetch
> >> >> request
> >> >> >> is
> >> >> >> >> like
> >> >> >> >> >> >> this:
> >> >> >> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10,
> >> fetchOffset:
> >> >> >> 2, …
> >> >> >> >> }.
> >> >> >> >> >> The
> >> >> >> >> >> >> > divergingEpoch check in the cluster B queries the
> >> endOffset
> >> >> for
> >> >> >> >> >> leader
> >> >> >> >> >> >> > epoch 10 (in Partition#readRecords), and found the end
> >> >> offset
> >> >> >> for
> >> >> >> >> LE
> >> >> >> >> >> 10
> >> >> >> >> >> >> is
> >> >> >> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch
> >> >> validation
> >> >> >> >> >> passes
> >> >> >> >> >> >> and
> >> >> >> >> >> >> > returns offset 2 to cluster A. In short, relying on
> “last
> >> >> >> mirror
> >> >> >> >> >> offset”
> >> >> >> >> >> >> > cannot resolve the log diverge issue in unclean leader
> >> >> election
> >> >> >> >> case.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > If instead, we truncate until “last mirror epoch”: 0,
> the
> >> >> >> offset 0
> >> >> >> >> >> and 1
> >> >> >> >> >> >> > in cluster A will be truncated as expected and then log
> >> >> >> converge
> >> >> >> >> with
> >> >> >> >> >> >> > cluster B correctly in the end.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Also, about JR53.2:
> >> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to
> all
> >> >> >> brokers
> >> >> >> >> and
> >> >> >> >> >> >> > merges
> >> >> >> >> >> >> > the responses client-side."
> >> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
> >> >> partition.
> >> >> >> So
> >> >> >> >> >> sending
> >> >> >> >> >> >> > the request to every broker is wasteful. It's probably
> >> >> better
> >> >> >> to
> >> >> >> >> send
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > request to a single broker, which can then contact the
> >> right
> >> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > While working on this, I found we need to add one more
> >> >> tagged
> >> >> >> >> field
> >> >> >> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema
> >> >> because
> >> >> >> >> >> >> otherwise
> >> >> >> >> >> >> > the receiving broker doesn’t know if this is the
> request
> >> >> from
> >> >> >> >> client
> >> >> >> >> >> >> and it
> >> >> >> >> >> >> > needs to contact other brokers, or this is just a
> >> forwarded
> >> >> >> >> request.
> >> >> >> >> >> The
> >> >> >> >> >> >> > workflow is like this:
> >> >> >> >> >> >> > 1. the adminClient sends the
> >> DescribeClusterMirrorsRequest
> >> >> to a
> >> >> >> >> >> >> > leastLoaded broker, which doesn’t contain the
> "Forwarded"
> >> >> >> field.
> >> >> >> >> >> >> > 2. The broker receives the request and sends
> >> >> >> >> >> >> DescribeClusterMirrorsRequest
> >> >> >> >> >> >> > with "Forwarded=true" to the responsible brokers and
> >> waits
> >> >> for
> >> >> >> the
> >> >> >> >> >> >> response
> >> >> >> >> >> >> > async.
> >> >> >> >> >> >> > 3. When all responses are received, merge the results
> and
> >> >> >> respond
> >> >> >> >> to
> >> >> >> >> >> the
> >> >> >> >> >> >> > client.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here
> because
> >> it
> >> >> >> >> doesn’t
> >> >> >> >> >> >> > contain the lag info in the coordinator.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > The alternative solution is the adminClient sends
> >> >> >> >> >> >> > DescribeClusterMirrorsRequest to the “responsible
> brokers
> >> >> only”
> >> >> >> >> (via
> >> >> >> >> >> >> > describeTopics), instead of ALL brokers, and merge the
> >> >> results
> >> >> >> in
> >> >> >> >> the
> >> >> >> >> >> >> > client side. This solution doesn’t need to add a new
> >> field
> >> >> in
> >> >> >> the
> >> >> >> >> >> >> > DescribeClusterMirrorsRequest and still can avoid
> >> >> unnecessary
> >> >> >> >> >> requests
> >> >> >> >> >> >> > sent.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > My preference is to keep the merge logic on the admin
> >> client
> >> >> >> side,
> >> >> >> >> >> which
> >> >> >> >> >> >> > saves the overhead in the broker side.
> >> >> >> >> >> >> > What do you think?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Thank you,
> >> >> >> >> >> >> > Luke
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
> >> >> >> >> dev@kafka.apache.org
> >> >> >> >> >> >
> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR31.3 It makes sense for the new partition leader to
> >> query
> >> >> >> the
> >> >> >> >> >> current
> >> >> >> >> >> >> >> mirror state from the local or remote
> >> MirrorMetadataManager
> >> >> >> that
> >> >> >> >> >> owns
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> partition's state. However, the wording in the KIP is
> >> >> >> confusing.
> >> >> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> >> mirror partition state cache should be populated only
> by
> >> >> one
> >> >> >> way,
> >> >> >> >> >> >> which is
> >> >> >> >> >> >> >> replaying __mirror_state partitions this broker leads.
> >> By
> >> >> >> saying
> >> >> >> >> >> "and
> >> >> >> >> >> >> by
> >> >> >> >> >> >> >> fetching state from remote coordinators via
> >> >> ReadMirrorStates
> >> >> >> >> RPCs.",
> >> >> >> >> >> >> it's
> >> >> >> >> >> >> >> as if the mirror partition state cache on each broker
> >> >> stores
> >> >> >> the
> >> >> >> >> >> state
> >> >> >> >> >> >> for
> >> >> >> >> >> >> >> all mirror partitions.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR36. I understand the intention now, but I still
> don't
> >> >> think
> >> >> >> we
> >> >> >> >> >> need
> >> >> >> >> >> >> >> MirrorLeaderEpoch. Let's
> >> >> >> >> >> >> >> assume the common case: no unclean leader election in
> >> the
> >> >> >> source
> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> >> during mirroring. Mirroring only fetches the committed
> >> data
> >> >> >> from
> >> >> >> >> the
> >> >> >> >> >> >> >> source, which never changes if obtained a second time.
> >> In
> >> >> your
> >> >> >> >> >> example
> >> >> >> >> >> >> >> image, follower broker 1 in the destination cluster
> can
> >> >> never
> >> >> >> >> obtain
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> record B(0) because it's never committed in the source
> >> and
> >> >> >> thus
> >> >> >> >> is
> >> >> >> >> >> >> never
> >> >> >> >> >> >> >> exposed to the destination leader. It's possible for
> >> >> broker 1
> >> >> >> to
> >> >> >> >> >> first
> >> >> >> >> >> >> get
> >> >> >> >> >> >> >> D(1) which is truncated later when broker 0 follows a
> >> new
> >> >> >> leader.
> >> >> >> >> >> But
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> new leader will refetch the same D(1) from the source
> >> >> cluster.
> >> >> >> >> So,
> >> >> >> >> >> >> it's ok
> >> >> >> >> >> >> >> for broker 1 to keep D(1).
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR51. Sounds good. Why do we choose to store the last
> >> >> mirror
> >> >> >> >> epoch
> >> >> >> >> >> >> instead
> >> >> >> >> >> >> >> of a last mirror offset?
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> Jun
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <
> >> >> showuon@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Thanks for the great comments again.
> >> >> >> >> >> >> >> > Answering your question below.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror
> >> partition
> >> >> >> cache
> >> >> >> >> >> lives
> >> >> >> >> >> >> in
> >> >> >> >> >> >> >> > MirrorMetadataManager. It is populated in two ways:
> by
> >> >> >> >> replaying
> >> >> >> >> >> >> >> > __mirror_state partitions this broker leads, and by
> >> >> fetching
> >> >> >> >> state
> >> >> >> >> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> >> >> >> >> > Hmm, a given broker only caches the mirror partition
> >> >> state
> >> >> >> for
> >> >> >> >> >> >> >> partitions
> >> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted on
> >> this
> >> >> >> broker,
> >> >> >> >> >> >> right?
> >> >> >> >> >> >> >> Why
> >> >> >> >> >> >> >> > does it need to fetch the state from other remote
> >> >> >> coordinators?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > a given broker only caches the mirror partition
> >> state
> >> >> for
> >> >> >> >> >> >> partitions
> >> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted on
> >> this
> >> >> >> broker,
> >> >> >> >> >> >> right?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Right.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > Why does it need to fetch the state from other
> >> remote
> >> >> >> >> >> coordinators?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Because the data partition leader needs to know the
> >> >> current
> >> >> >> >> mirror
> >> >> >> >> >> >> state
> >> >> >> >> >> >> >> > of the partitions it leads. On a leadership change,
> >> the
> >> >> >> >> metadata
> >> >> >> >> >> >> delta
> >> >> >> >> >> >> >> is
> >> >> >> >> >> >> >> > published to the MirrorMetadataManager, which
> queries
> >> the
> >> >> >> >> current
> >> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> > state for its led partitions from the coordinator
> >> (local
> >> >> or
> >> >> >> >> >> remote),
> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> > processes tasks based on that state. This is the
> >> reason
> >> >> why
> >> >> >> it
> >> >> >> >> >> >> fetches
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > state from other remote coordinator. This part of
> >> cache
> >> >> >> will be
> >> >> >> >> >> >> cleared
> >> >> >> >> >> >> >> > after the leadership change.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR36. I am still not quite sure why we need
> >> >> >> >> MirrorLeaderEpoch in
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > Fetch
> >> >> >> >> >> >> >> > request. In the current Fetch request, in addition
> to
> >> >> >> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> >> >> > it has another field LastFetchedEpoch, which
> captures
> >> the
> >> >> >> last
> >> >> >> >> >> leader
> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> > in the data and is used to resolve the inconsistency
> >> >> between
> >> >> >> >> the
> >> >> >> >> >> >> >> follower
> >> >> >> >> >> >> >> > and the leader. Does it achieve the same purpose as
> >> >> >> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used
> to
> >> do
> >> >> the
> >> >> >> >> >> >> validation
> >> >> >> >> >> >> >> to
> >> >> >> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any
> >> fetched
> >> >> >> >> batches
> >> >> >> >> >> with
> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> > higher than the LastFetchedEpoch, it means the
> >> follower
> >> >> can
> >> >> >> >> never
> >> >> >> >> >> >> accept
> >> >> >> >> >> >> >> > new appended batches with higher epoch in the leader
> >> >> node.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2
> >> >> purposes in
> >> >> >> >> the
> >> >> >> >> >> >> fetch
> >> >> >> >> >> >> >> > API:
> >> >> >> >> >> >> >> > 1. The receiver (leader node) uses it to validate it
> >> >> matches
> >> >> >> >> the
> >> >> >> >> >> >> leader
> >> >> >> >> >> >> >> > epoch in the leader node
> >> >> >> >> >> >> >> > 2. The sender (follower node) uses it to validate
> the
> >> >> epoch
> >> >> >> in
> >> >> >> >> the
> >> >> >> >> >> >> >> fetched
> >> >> >> >> >> >> >> > batches are all <= this value.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still
> >> serving
> >> >> for
> >> >> >> >> >> >> followers
> >> >> >> >> >> >> >> in
> >> >> >> >> >> >> >> > the destination cluster. For purpose (2), my
> >> >> understanding
> >> >> >> of
> >> >> >> >> the
> >> >> >> >> >> >> >> > validation for KAFKA-18723 is like: the follower
> knows
> >> >> the
> >> >> >> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader
> >> node
> >> >> >> >> should
> >> >> >> >> >> >> >> contain in
> >> >> >> >> >> >> >> > the log batches when fetch request is sent. Any log
> >> batch
> >> >> >> >> beyond
> >> >> >> >> >> this
> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> > in the fetch response means the follower’s metadata
> is
> >> >> >> stale.
> >> >> >> >> The
> >> >> >> >> >> >> >> batches
> >> >> >> >> >> >> >> > need to be rejected and follower needs to update
> >> metadata
> >> >> >> for
> >> >> >> >> next
> >> >> >> >> >> >> >> fetch.
> >> >> >> >> >> >> >> > In the Cluster Mirroring’s world, the
> >> MirrorLeaderEpoch
> >> >> >> serves
> >> >> >> >> >> this
> >> >> >> >> >> >> >> purpose
> >> >> >> >> >> >> >> > to let the followers know the highest epoch in the
> >> leader
> >> >> >> node
> >> >> >> >> >> log in
> >> >> >> >> >> >> >> each
> >> >> >> >> >> >> >> > fetch response. Once the info is stale in the
> follower
> >> >> >> side, it
> >> >> >> >> >> >> updates
> >> >> >> >> >> >> >> it
> >> >> >> >> >> >> >> > from fetch response, and send next fetch request
> with
> >> >> >> correct
> >> >> >> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data
> is
> >> >> >> possible
> >> >> >> >> >> like
> >> >> >> >> >> >> >> > KAFKA-18723.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > I tried not to add new field in Fetch API, but this
> is
> >> >> the
> >> >> >> best
> >> >> >> >> >> >> >> solution I
> >> >> >> >> >> >> >> > can think of.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR48. "The NumPartitions field is needed so the
> >> >> controller
> >> >> >> >> can
> >> >> >> >> >> >> create
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > destination topic if it does not already exist.
> When a
> >> >> >> topic is
> >> >> >> >> >> being
> >> >> >> >> >> >> >> > mirrored for the first time, the destination cluster
> >> has
> >> >> no
> >> >> >> >> >> knowledge
> >> >> >> >> >> >> >> > of the source topic's partition count, so the
> request
> >> >> must
> >> >> >> >> carry
> >> >> >> >> >> it."
> >> >> >> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is
> >> started,
> >> >> the
> >> >> >> >> source
> >> >> >> >> >> >> topic
> >> >> >> >> >> >> >> > may not exist yet. When the source topic is created
> >> >> later,
> >> >> >> we
> >> >> >> >> can
> >> >> >> >> >> >> start
> >> >> >> >> >> >> >> > mirroring without requiring the user to provide the
> >> >> number
> >> >> >> of
> >> >> >> >> >> >> >> partitions.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > When the source topic is created later, we can
> start
> >> >> >> >> mirroring
> >> >> >> >> >> >> without
> >> >> >> >> >> >> >> > requiring the user to provide the number of
> >> partitions.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > This is because this happens in periodical sync up
> in
> >> the
> >> >> >> >> >> destination
> >> >> >> >> >> >> >> > cluster, and the numPartitions and other topic info
> in
> >> >> the
> >> >> >> >> source
> >> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> >> > is already retrieved during the sync up process.
> >> Also, if
> >> >> >> users
> >> >> >> >> >> are
> >> >> >> >> >> >> >> > creating mirror topics via admin API, this info can
> >> also
> >> >> be
> >> >> >> >> >> retrieved
> >> >> >> >> >> >> >> > inside the admin client if not provided.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly
> need
> >> the
> >> >> >> last
> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> > for truncation when starting a mirror. When a mirror
> >> is
> >> >> >> >> stopped at
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > source, we already bump up the leader epoch and
> write
> >> >> some
> >> >> >> >> >> additional
> >> >> >> >> >> >> >> > records for cleanup. It seems that we can just use
> the
> >> >> >> >> >> DivergingEpoch
> >> >> >> >> >> >> >> > returned in the initial mirror Fetch request to
> detect
> >> >> the
> >> >> >> >> >> diverging
> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> > and use that for truncation at the mirror
> destination.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > That’s a good question. I thought about this before
> >> but
> >> >> >> >> >> >> unfortunately it
> >> >> >> >> >> >> >> > doesn’t work. The reason we need the last mirror
> epoch
> >> >> >> record
> >> >> >> >> is
> >> >> >> >> >> >> because
> >> >> >> >> >> >> >> > the destination cluster might not completely sync
> with
> >> >> the
> >> >> >> >> source
> >> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> >> > before STOPPED. And this un-mirrored data might
> >> confuse
> >> >> the
> >> >> >> >> >> >> replication
> >> >> >> >> >> >> >> > protocol when reverse mirror. For example,
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
> >> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE:
> >> 0)
> >> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE:
> >> 0)
> >> >> >> >> >> >> >> > Offset 2: from A (LE: 10)
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > While cluster B mirrors until offset 1, the cluster
> A
> >> >> >> crashes
> >> >> >> >> and
> >> >> >> >> >> >> >> failover
> >> >> >> >> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
> >> >> >> >> >> MIRROR_PID_RESET
> >> >> >> >> >> >> >> with
> >> >> >> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from
> >> >> >> cluster B,
> >> >> >> >> >> the
> >> >> >> >> >> >> log
> >> >> >> >> >> >> >> > will be like this:
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> >> >> >> > foo-0 leader log: foo-0 leader log:
> >> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE:
> >> 0)
> >> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE:
> >> 0)
> >> >> >> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET
> (LE:
> >> >> 10)
> >> >> >> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE:
> >> 10)
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > As you can see, cluster A (new destination cluster)
> >> >> doesn’t
> >> >> >> >> know
> >> >> >> >> >> the
> >> >> >> >> >> >> >> > offset 2 is not belonging to the cluster B, which
> >> causes
> >> >> >> >> >> inconsistent
> >> >> >> >> >> >> >> logs.
> >> >> >> >> >> >> >> > So like what we described in the KIP, the last
> mirror
> >> >> epoch
> >> >> >> is
> >> >> >> >> the
> >> >> >> >> >> >> >> > synchronization point between source and
> destination.
> >> Any
> >> >> >> data
> >> >> >> >> >> beyond
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > point is treated as unknown to the source cluster
> and
> >> >> >> should be
> >> >> >> >> >> >> >> truncated.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR52. Stop Mirror:
> >> >> >> >> >> >> >> > > JR52.1 "3. If patterns are provided, the
> controller
> >> >> >> removes
> >> >> >> >> >> >> matching
> >> >> >> >> >> >> >> > entries from mirror.topics.include or adds them to
> >> >> >> >> >> >> >> mirror.topics.exclude on
> >> >> >> >> >> >> >> > the CLUSTER_MIRROR resource."
> >> >> >> >> >> >> >> > Why is it 'or' and not 'and'?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> >> > discovers new source topics matching the persisted
> >> >> >> >> include/exclude
> >> >> >> >> >> >> >> patterns
> >> >> >> >> >> >> >> > and repeats steps 3 through 7 for each."
> >> >> >> >> >> >> >> > The broker will send a create topic request to the
> >> >> >> controller.
> >> >> >> >> How
> >> >> >> >> >> >> does
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > controller associate it with the mirror name to
> >> create a
> >> >> >> >> >> >> >> > `MirrorTopicStateChangeRecord`?
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > No, we don’t send a create topic request to the
> >> >> controller
> >> >> >> >> here.
> >> >> >> >> >> >> >> Actually
> >> >> >> >> >> >> >> > we send a StartMirrorTopics request to the
> controller,
> >> >> which
> >> >> >> >> >> contains
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > mirror name and topic metadata like NumPartitions,
> >> >> topicId…
> >> >> >> >> etc.
> >> >> >> >> >> So
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > controller can create the non-existed topic as
> >> explained
> >> >> in
> >> >> >> >> JR48,
> >> >> >> >> >> and
> >> >> >> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR53. Describe Mirrors:
> >> >> >> >> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors
> >> >> request
> >> >> >> >> with
> >> >> >> >> >> >> >> optional
> >> >> >> >> >> >> >> > mirror names (empty means all mirrors)."
> >> >> >> >> >> >> >> > In metadata request, we use a null arrary to
> represent
> >> >> all
> >> >> >> >> >> topics. It
> >> >> >> >> >> >> >> would
> >> >> >> >> >> >> >> > be useful to be consistent here.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > OK, KIP updated.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request
> to
> >> all
> >> >> >> >> brokers
> >> >> >> >> >> and
> >> >> >> >> >> >> >> > merges
> >> >> >> >> >> >> >> > the responses client-side."
> >> >> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
> >> >> >> partition.
> >> >> >> >> So
> >> >> >> >> >> >> sending
> >> >> >> >> >> >> >> > the request to every broker is wasteful. It's
> probably
> >> >> >> better
> >> >> >> >> to
> >> >> >> >> >> send
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > request to a single broker, which can then contact
> the
> >> >> right
> >> >> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Agree, KIP updated.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState:
> Are
> >> >> >> >> MIRRORING,
> >> >> >> >> >> >> >> PAUSED,
> >> >> >> >> >> >> >> > STOPPED the only validate states? The state table
> has
> >> >> other
> >> >> >> >> states
> >> >> >> >> >> >> too.
> >> >> >> >> >> >> >> > Ditto for MirrorPartitionState.
> >> >> >> >> >> >> >> > The DesiredState field in
> MirrorTopicStateChangeRecord
> >> >> is to
> >> >> >> >> >> define
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> > target state the user wants. And the desired state
> >> users
> >> >> can
> >> >> >> >> >> assign
> >> >> >> >> >> >> is
> >> >> >> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a
> >> desired
> >> >> >> state
> >> >> >> >> as
> >> >> >> >> >> >> >> STOPPING
> >> >> >> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
> >> >> >> intermediate
> >> >> >> >> >> >> states.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > About MirrorPartitionState record, it’ll record any
> >> >> state in
> >> >> >> >> the
> >> >> >> >> >> >> state
> >> >> >> >> >> >> >> > table that the current partition is. This is the
> real
> >> >> state
> >> >> >> for
> >> >> >> >> >> each
> >> >> >> >> >> >> >> > partition. It’s like in the kubernetes world, when
> we
> >> >> change
> >> >> >> >> the
> >> >> >> >> >> >> replica
> >> >> >> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired
> >> >> state), it
> >> >> >> >> won’t
> >> >> >> >> >> >> >> increase
> >> >> >> >> >> >> >> > to 5 immediately. The current replica
> >> >> (MirrorPartitionState
> >> >> >> >> >> record)
> >> >> >> >> >> >> is
> >> >> >> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > Thank you,
> >> >> >> >> >> >> >> > Luke
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
> >> >> >> >> >> dev@kafka.apache.org
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> Hi, Fede,
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> Thanks for the reply. A few more comments.
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror
> partition
> >> >> cache
> >> >> >> >> >> lives in
> >> >> >> >> >> >> >> >> MirrorMetadataManager. It is populated in two ways:
> >> by
> >> >> >> >> replaying
> >> >> >> >> >> >> >> >> __mirror_state partitions this broker leads, and by
> >> >> >> fetching
> >> >> >> >> >> state
> >> >> >> >> >> >> >> >> from remote coordinators via ReadMirrorStates
> RPCs."
> >> >> >> >> >> >> >> >> Hmm, a given broker only caches the mirror
> partition
> >> >> state
> >> >> >> for
> >> >> >> >> >> >> >> partitions
> >> >> >> >> >> >> >> >> hashed into the __mirror_state partition hosted on
> >> this
> >> >> >> >> broker,
> >> >> >> >> >> >> right?
> >> >> >> >> >> >> >> Why
> >> >> >> >> >> >> >> >> does it need to fetch the state from other remote
> >> >> >> >> coordinators?
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR36. I am still not quite sure why we need
> >> >> >> MirrorLeaderEpoch
> >> >> >> >> in
> >> >> >> >> >> the
> >> >> >> >> >> >> >> Fetch
> >> >> >> >> >> >> >> >> request. In the current Fetch request, in addition
> to
> >> >> >> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> >> >> >> it has another field LastFetchedEpoch, which
> captures
> >> >> the
> >> >> >> last
> >> >> >> >> >> >> leader
> >> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> in the data and is used to resolve the
> inconsistency
> >> >> >> between
> >> >> >> >> the
> >> >> >> >> >> >> >> follower
> >> >> >> >> >> >> >> >> and the leader. Does it achieve the same purpose as
> >> >> >> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR48. "The NumPartitions field is needed so the
> >> >> controller
> >> >> >> can
> >> >> >> >> >> >> create
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> destination topic if it does not already exist.
> When
> >> a
> >> >> >> topic
> >> >> >> >> is
> >> >> >> >> >> >> being
> >> >> >> >> >> >> >> >> mirrored for the first time, the destination
> cluster
> >> >> has no
> >> >> >> >> >> >> knowledge
> >> >> >> >> >> >> >> >> of the source topic's partition count, so the
> request
> >> >> must
> >> >> >> >> carry
> >> >> >> >> >> >> it."
> >> >> >> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is
> >> started,
> >> >> the
> >> >> >> >> >> source
> >> >> >> >> >> >> >> topic
> >> >> >> >> >> >> >> >> may not exist yet. When the source topic is created
> >> >> later,
> >> >> >> we
> >> >> >> >> can
> >> >> >> >> >> >> start
> >> >> >> >> >> >> >> >> mirroring without requiring the user to provide the
> >> >> number
> >> >> >> of
> >> >> >> >> >> >> >> partitions.
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need
> >> the
> >> >> >> last
> >> >> >> >> >> mirror
> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> for truncation when starting a mirror. When a
> mirror
> >> is
> >> >> >> >> stopped
> >> >> >> >> >> at
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> source, we already bump up the leader epoch and
> write
> >> >> some
> >> >> >> >> >> >> additional
> >> >> >> >> >> >> >> >> records for cleanup. It seems that we can just use
> >> the
> >> >> >> >> >> >> DivergingEpoch
> >> >> >> >> >> >> >> >> returned in the initial mirror Fetch request to
> >> detect
> >> >> the
> >> >> >> >> >> diverging
> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> >> and use that for truncation at the mirror
> >> destination.
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR52. Stop Mirror:
> >> >> >> >> >> >> >> >> JR52.1 "3. If patterns are provided, the controller
> >> >> removes
> >> >> >> >> >> matching
> >> >> >> >> >> >> >> >> entries from mirror.topics.include or adds them to
> >> >> >> >> >> >> >> mirror.topics.exclude
> >> >> >> >> >> >> >> >> on
> >> >> >> >> >> >> >> >> the CLUSTER_MIRROR resource."
> >> >> >> >> >> >> >> >> Why is it 'or' and not 'and'?
> >> >> >> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> >> >> discovers new source topics matching the persisted
> >> >> >> >> >> include/exclude
> >> >> >> >> >> >> >> >> patterns
> >> >> >> >> >> >> >> >> and repeats steps 3 through 7 for each."
> >> >> >> >> >> >> >> >> The broker will send a create topic request to the
> >> >> >> controller.
> >> >> >> >> >> How
> >> >> >> >> >> >> does
> >> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> controller associate it with the mirror name to
> >> create a
> >> >> >> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR53. Describe Mirrors:
> >> >> >> >> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors
> >> >> request
> >> >> >> >> with
> >> >> >> >> >> >> >> optional
> >> >> >> >> >> >> >> >> mirror names (empty means all mirrors)."
> >> >> >> >> >> >> >> >> In metadata request, we use a null arrary to
> >> represent
> >> >> all
> >> >> >> >> >> topics.
> >> >> >> >> >> >> It
> >> >> >> >> >> >> >> >> would
> >> >> >> >> >> >> >> >> be useful to be consistent here.
> >> >> >> >> >> >> >> >> JR53.2 "2. The admin client fans out the request to
> >> all
> >> >> >> >> brokers
> >> >> >> >> >> and
> >> >> >> >> >> >> >> merges
> >> >> >> >> >> >> >> >> the responses client-side."
> >> >> >> >> >> >> >> >> This is not ideal. Not every broker hosts a mirror
> >> >> >> partition.
> >> >> >> >> So
> >> >> >> >> >> >> >> sending
> >> >> >> >> >> >> >> >> the request to every broker is wasteful. It's
> >> probably
> >> >> >> better
> >> >> >> >> to
> >> >> >> >> >> >> send
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> request to a single broker, which can then contact
> >> the
> >> >> >> right
> >> >> >> >> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState:
> Are
> >> >> >> >> MIRRORING,
> >> >> >> >> >> >> PAUSED,
> >> >> >> >> >> >> >> >> STOPPED the only validate states? The state table
> has
> >> >> other
> >> >> >> >> >> states
> >> >> >> >> >> >> too.
> >> >> >> >> >> >> >> >> Ditto for MirrorPartitionState.
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> Jun
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
> >> >> >> >> >> >> fedevaleri@gmail.com>
> >> >> >> >> >> >> >> >> wrote:
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >> > Hi Andrew, yes, validation is called at the start
> >> of
> >> >> >> >> >> createTopic.
> >> >> >> >> >> >> If
> >> >> >> >> >> >> >> >> > the topic ID is invalid, the request fails with
> >> >> >> >> >> INVALID_REQUEST.
> >> >> >> >> >> >> If
> >> >> >> >> >> >> >> >> > the topic ID is already used by a different topic
> >> >> name,
> >> >> >> it
> >> >> >> >> >> fails
> >> >> >> >> >> >> with
> >> >> >> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the
> >> >> individual
> >> >> >> >> topic
> >> >> >> >> >> in
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> >> > batch fails; other topics in the same
> CreateTopics
> >> >> >> request
> >> >> >> >> >> >> continue
> >> >> >> >> >> >> >> >> > processing normally. Added this information to
> the
> >> >> KIP.
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew
> Schofield <
> >> >> >> >> >> >> >> >> schofielaj@apache.org>
> >> >> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> >> > > Hi Fede,
> >> >> >> >> >> >> >> >> > > One more small question.
> >> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> >> > > AS37: If the controller fails validation for
> the
> >> >> >> >> >> >> MirrorInfo.TopicId
> >> >> >> >> >> >> >> >> > added to the CreateTopic request, what does it
> do?
> >> I
> >> >> >> suspect
> >> >> >> >> >> that
> >> >> >> >> >> >> it
> >> >> >> >> >> >> >> >> fails
> >> >> >> >> >> >> >> >> > the request with a particular error code.
> >> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> >> > > Thanks,
> >> >> >> >> >> >> >> >> > > Andrew
> >> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> >> >> >> >> >> >> >> >> > > > Hi all,
> >> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> >> > > > We updated the KIP to include Coordinator
> >> Runtime
> >> >> >> >> >> >> configurations
> >> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> > > > the coordinator state transition validation
> >> >> >> paragraph.
> >> >> >> >> The
> >> >> >> >> >> >> latter
> >> >> >> >> >> >> >> >> > > > covers the per-partition epoch fencing
> >> mechanism
> >> >> >> (leader
> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> >> > > > state epoch) that complements the existing
> >> >> per-topic
> >> >> >> >> >> >> StateOffset
> >> >> >> >> >> >> >> >> > > > fencing for lifecycle operations.
> >> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> >> > > > Thanks,
> >> >> >> >> >> >> >> >> > > > Fede
> >> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >
> >> >> >> >> >> >>
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply.
JR80. AdminClient.startMirrorTopics(): Is includePatterns() optional? It
seems that mirroring can't start without it.
JR91. Mirror configs: Some of the configs are prefixed with "mirror," while
others are not. Could we make them consistent? Since all those configs are
for Mirror, the mirror prefix seems redundant.
Jun
On Mon, Aug 17, 2026 at 1:11 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the good comments.
> Answering them below.
>
> > JR80. In that case, it seems that there is no need for
> AdminClient.startMirrorTopics() to take topics. The adminClient only uses
> it to register the included topics, which can be passed in as
> includePatterns (specific names or regex). The broker will use
> StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
> through the AdminClient.
>
> Make sense. So when users invoke admin.startMirrorTopics, they only need
> to set the topic patterns. And the receiving broker will resolve the
> patterns into the actual topics using the listTopic response from the
> source cluster, and get the topic id, numPartitions, … via the
> describeTopics request from the source cluster if the topic is non-existent
> in the destination cluster. KIP is updated.
>
>
> > JR82. If “IncludeMirrorOffset” is disabled,
> DescribeClusterMirrorsResponse.SourceOffset is still returned in the
> response and what is its value in this case?
>
> It will be -1 because when the `IncludeMirrorOffset` is disabled, we won’t
> query the mirror partition leader. I’ve updated the description to make it
> clear. Thanks.
>
>
> > JR88. Why is partition-load-time a Meter instead of a Gauge?
>
> You’re right. It is not the Meter type. It’s the SampledStat state like
> “batch-size-avg” and “batch-size-max” in producer. I’ve updated the KIP.
>
>
> > JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
>
> You’re right. It’s the rate type. KIP is updated
>
>
> > JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
> Meter.
> Yes, it makes more sense to make it as Meter type. I’ve updated these
> error-related metrics to Meter type:
> 1. MetadataRefreshError
> 2. TopicConfigMetadataSyncError
> 3. ConsumerGroupOffsetSyncError
> 4. ShareGroupOffsetSyncError
> 5. AclSyncError
>
>
> Thank you,
> Luke
>
>
> On Sat, Aug 15, 2026 at 8:10 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Luke,
>>
>> Thanks for the reply.
>>
>> JR80. In that case, it seems that there is no need for
>> AdminClient.startMirrorTopics() to take topics. The adminClient only uses
>> it to register the included topics, which can be passed in as
>> includePatterns (specific names or regex). The broker will use
>> StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
>> through the AdminClient.
>>
>> JR82. If “IncludeMirrorOffset” is disabled,
>> DescribeClusterMirrorsResponse.SourceOffset is still returned in the
>> response and what is its value in this case?
>>
>> JR88. Why is partition-load-time a Meter instead of a Gauge?
>>
>> JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
>>
>> JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
>> Meter.
>>
>> Jun
>>
>> On Fri, Aug 14, 2026 at 4:32 AM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for the good comments,
>> > Answering them below.
>> >
>> > > JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
>> > supports regex in the ClusterMirrorComamnd, but we don’t need to add the
>> > “patterns”."
>> > So, in this case, we use topics to support both specific topics and
>> regex.
>> > However, startMirrorTopicsOptions uses topics for specific topics and
>> > includePatterns for regex. Could we make them consistent?
>> >
>> >
>> > The purpose of the “includePatterns” field is for the
>> > “mirror.topics.include” update, so that if there is any newly created
>> topic
>> > in the source cluster that matches the regex, we can create it. And the
>> > “Topics” field in the StartMirrorTopicsRequest is to specify the topicId
>> > and numPartitions in the source cluster. So I think “includePatterns”
>> and
>> > “Topics” both are necessary in the StartMirrorTopicsRequest. On the
>> other
>> > hand, in PauseMirrorTopics, because it won’t change the
>> > “mirror.topics.include”/”mirror.topics.exclude” configs, we don’t need
>> to
>> > provide the regex in the request.
>> >
>> >
>> >
>> > > JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also
>> -1
>> > if
>> > it's not included in the request?
>> >
>> > If “IncludeMirrorOffset” is disabled, we won’t query the mirror
>> partition
>> > leader, so we can’t get the sourceOffset.
>> >
>> > I assume you want to ask `if it’s still -1 if not actively mirroring,
>> > while “IncludeMirrorOffset” is enabled`. That’s a good question. I think
>> > our purpose of this command is to show the lag to users. So if the
>> state is
>> > not MIRRORING (ex: STOPPED/PAUSED), we think it’s not helpful to users
>> to
>> > show only the sourceOffset. Of course users can use
>> `kafka-get-offsets.sh`
>> > to get the offset if they really want to know.
>> >
>> >
>> > > JR83. Start Mirror Topics:
>> > > JR83.1 Should the controller verify the expected state for the topic?
>> >
>> > As replied in JR84, the desired state validation should happen in the
>> > broker. And the controller validates the StateOffset. I’ve updated the
>> > workflow to make it clear.
>> >
>> > > JR83.2 "d. truncates the log accordingly,"
>> > Could you add a description of the truncation offset?
>> >
>> > OK, added one more sentence:
>> > d. based on the last mirror epoch response in (b), the broker translates
>> > the epoch into the offset by querying the leaderEpochCache
>> > (endOffsetForEpoch), and truncates the log accordingly, and waits for
>> all
>> > ISR members…
>> >
>> >
>> > > JR84. Stop Mirror Topics: "The controller verifies that the desired
>> state
>> > of all topics are in MIRRORING or PAUSED state."
>> > Should the broker verfy the topic level state? I thought the controller
>> > will use StateOffset in the request to make sure the validation on the
>> > broker is stil valid. If the controller performs the verification, is
>> > StateOffset still useful?
>> >
>> > Oh, you’re right. We’re doing duplicated work here. Yes, the broker
>> > already verifies the “desired state” of all topics are in MIRRORING or
>> > PAUSED state, as well as the “actual state” is in MIRRORING or PAUSED
>> > state. If all validation passes, we’ll forward it to the controller. So
>> the
>> > controller only needs to validate the StateOffset to make sure the
>> > validation on the broker is still valid. So, yes, we don’t need the
>> desired
>> > state validation on the controller side. I’ve updated the KIP to make it
>> > clear.
>> >
>> > One more thing to add is that, I’ve thought about not adding
>> StateOffset,
>> > and relying on the desired state validation to make sure we’re safe to
>> do
>> > the deletion or other operations. The problem is that, even if the
>> broker
>> > sees the desired state for topic A is STOPPED, and then the controller
>> also
>> > sees the desired state is STOPPED, it still cannot make sure the
>> > controller’s STOPPED state equals the broker’s STOPPED state. For
>> example:
>> > 1. Broker sees STOPPED, then forward to controller
>> > 2. User changes the desired state to MIRRORING
>> > 3. User changes the desired state to STOPPED
>> > 4. Controller sees the desired state is STOPPED
>> > So I think StateOffset is required.
>> >
>> > Thanks for catching it BTW.
>> >
>> >
>> > > JR85. There are still a few references to includeMirrorLags. They
>> should
>> > be
>> > IncludeMirrorOffset.
>> >
>> > Sorry, just updated.
>> >
>> >
>> > > JR86. Metrics:
>> > > JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
>> >
>> > KIP is updated. I also group the metrics into new added metrics and
>> other
>> > inherited metrics.
>> >
>> > > JR86.2 Could you describe a bit more what MinFetchRate represents?
>> >
>> > It means the lowest fetch request rate among all of this mirror fetcher
>> > manager's fetcher threads. It inherits from the AbstractFetcherManager’s
>> > metric.
>> >
>> >
>> > > JR87. Version Compatibility: "For migration or DR use cases where
>> > failback
>> > is not required, this proposal supports source brokers up to version 2.1
>> > included"
>> > This KIP needs topicId and KIP-516 was introduced later than 2.1.
>> > Yes, when mirroring from a cluster without topic id support, we will
>> > assign a random topic ID in the destination cluster and then fetch
>> (without
>> > topic id in the fetch request using the old version of Fetch API) as
>> usual.
>> > Of course this is for the one-way mirroring use case. It can’t start the
>> > reverse mirror(failback) under this setup. You can check the
>> “Compatibility
>> > Matrix” section for more detail.
>> >
>> >
>> > Thank you,
>> > Luke
>> >
>> > On Fri, Aug 14, 2026 at 7:09 AM Jun Rao via dev <de...@kafka.apache.org>
>> > wrote:
>> >
>> >> Hi, Luke,
>> >>
>> >> Thanks for the reply.
>> >>
>> >> JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
>> >> supports regex in the ClusterMirrorComamnd, but we don’t need to add
>> the
>> >> “patterns”."
>> >> So, in this case, we use topics to support both specific topics and
>> regex.
>> >> However, startMirrorTopicsOptions uses topics for specific topics and
>> >> includePatterns for regex. Could we make them consistent?
>> >>
>> >> JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also
>> -1 if
>> >> it's not included in the request?
>> >>
>> >> JR83. Start Mirror Topics:
>> >> JR83.1 Should the controller verify the expected state for the topic?
>> >> JR83.2 "d. truncates the log accordingly,"
>> >> Could you add a description of the truncation offset?
>> >>
>> >> JR84. Stop Mirror Topics: "The controller verifies that the desired
>> state
>> >> of all topics are in MIRRORING or PAUSED state."
>> >> Should the broker verfy the topic level state? I thought the controller
>> >> will use StateOffset in the request to make sure the validation on the
>> >> broker is stil valid. If the controller performs the verification, is
>> >> StateOffset still useful?
>> >>
>> >> JR85. There are still a few references to includeMirrorLags. They
>> should
>> >> be
>> >> IncludeMirrorOffset.
>> >>
>> >> JR86. Metrics:
>> >> JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
>> >> JR86.2 Could you describe a bit more what MinFetchRate represents?
>> >>
>> >> JR87. Version Compatibility: "For migration or DR use cases where
>> failback
>> >> is not required, this proposal supports source brokers up to version
>> 2.1
>> >> included"
>> >> This KIP needs topicId and KIP-516 was introduced later than 2.1.
>> >>
>> >> Jun
>> >>
>> >>
>> >> On Thu, Aug 13, 2026 at 5:14 AM Luke Chen <sh...@gmail.com> wrote:
>> >>
>> >> > Hi Jun,
>> >> >
>> >> > Thanks for the good comments.
>> >> > Answering them below.
>> >> >
>> >> > > JR59.1 Could you add the RPC to the workflow? The broker first
>> needs
>> >> to
>> >> > issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
>> >> > mirror name and then issue DescribeClusterMirrors to get the LME.
>> >> >
>> >> > Yes, you’re right. The workflow is updated to make them clear.
>> >> >
>> >> > > JR71. mirror.support.unclean.leader.election : The name is
>> confusing.
>> >> > It's
>> >> > also used in LOG_TRUNCATION, which is not related to unclean leader
>> >> > election. Could we use a better name?
>> >> >
>> >> > I personally think the name should be fine because even though we
>> use in
>> >> > “LOG_TRUNCATION” state, the config is to allow us to know if users
>> want
>> >> to
>> >> > support unclean leader election, if so, we wait for all replicas to
>> >> catch
>> >> > up. If we want to explicitly show the intention of the config, how
>> about
>> >> > “mirror.full.replica.convergence.required”? I’m open to other
>> >> suggestions.
>> >> >
>> >> >
>> >> > > JR72. DescribeClusterMirrors
>> >> > > JR72.1 A user may specify just the topic name when describing a
>> >> mirror.
>> >> > Should we include the topic name in the request?
>> >> >
>> >> > Hmm.. we didn’t think about the use case. But yes, if users only care
>> >> > about the state of a topic, they don’t need the info for other
>> topics.
>> >> I’ve
>> >> > combined with JR72.2, added “TopicPartitions” filed into
>> >> > DescribeClusterMirrorsRequest, and by default is null, which means
>> all
>> >> > topic partitions under the requested mirrors will be returned. If
>> set,
>> >> > we’ll query the mirrors and return the topic partitions that users
>> >> > expected. KIP is updated.
>> >> >
>> >> >
>> >> > > JR72.2 LastMirrorEpochLookups is a bit weird since the response
>> >> returns
>> >> > either mirror state or mirror lag and LastMirrorEpochLookups doesn't
>> >> match
>> >> > either.
>> >> >
>> >> > Yes, after the recent refactor, I think the LastMirrorEpochLookups
>> can
>> >> be
>> >> > merged into one level up, combining with the topic name filter in
>> above
>> >> > comment, which will allow users to query down to the partition level.
>> >> The
>> >> > LastMirrorEpochLookups will query along with normal mirror state by
>> the
>> >> > readMirrorState from the coordinator. The difference is that after
>> >> > receiving the response, it will filter out the mirrors that don’t
>> match
>> >> the
>> >> > requested sourceClusterId, and then add the lastMirrorEpoch into
>> >> > DescribeClusterMirrorResponse. KIP is updated.
>> >> >
>> >> >
>> >> > > JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> >
>> >> > Right. Added.
>> >> >
>> >> >
>> >> > > JR72.4 "includeMirrorLags: source offset, destination offset, lag"
>> >> > DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
>> >> > should be omitted.
>> >> >
>> >> > Sorry, I removed the “SourceOffset” and “DestinationOffset” fields
>> from
>> >> > DescribeClusterMirrorsResponse because I added them into
>> >> ReadMirrorOffsets
>> >> > (ex-ReadMirrorLags) API. But I forgot we still need them in the
>> >> > DescribeClusterMirrorsResponse so that the adminClient can receive
>> the
>> >> > results. KIP is updated.
>> >> >
>> >> >
>> >> > > JR73. MirrorPartitionState record, LastMirrorEpochs record and
>> >> > MirrorPartitionStateRequest: The values for type and name are
>> incorrect.
>> >> >
>> >> > Ah, that’s my bad. Fixed.
>> >> >
>> >> > > JR74. ListClusterMirrorsRequest
>> >> > > JR74.1 When is DesiredMirrorStateFilter used? There is no option to
>> >> > specify
>> >> > the desired state in the mirror-topic command.
>> >> >
>> >> > Yes, referred to kafka-consumer-groups.sh, I added the `--state`
>> option
>> >> to
>> >> > allow users to set the state when listMirror.
>> >> >
>> >> > > JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>> >> >
>> >> > Yes, KIP updated.
>> >> >
>> >> > > JR75. ReadMirrorLags
>> >> > > JR75.1 Should it be named ReadMirrorOffsets since the response
>> doesn't
>> >> > include lag?
>> >> >
>> >> > Indeed. Updated.
>> >> >
>> >> > > JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> >
>> >> > Yes, added
>> >> >
>> >> > > JR75.3 Some places refer to this as ReadClusterMirrorLags. We need
>> to
>> >> be
>> >> > consistent.
>> >> >
>> >> > Thanks for pointing it out. KIP is updated.
>> >> >
>> >> > > JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR
>> too?
>> >> >
>> >> > Yes, we should validate the mirror name before writing the data. KIP
>> is
>> >> > updated.
>> >> >
>> >> > > JR77. Delete Mirror: "3. The controller verifies that no topics
>> are in
>> >> > MIRRORING or PAUSED state."
>> >> > Is this necessary since step 2 has verified all partitions are in
>> >> STOPPED
>> >> > state, which can only happen if the topics are in STOPPED state.
>> >> >
>> >> > It’s true, but there could be some delay between “actual state” and
>> >> > “desired state”. In step (2), we verify all partitions’ “actual
>> state”
>> >> are
>> >> > in STOPPED state. But the current desired state could be MIRRORING,
>> it’s
>> >> > just that the mirror partition leaders haven’t got the metadata
>> update
>> >> yet.
>> >> > So, the controller still needs to verify the desired state is not in
>> >> > MIRRORING or PAUSED. I found we didn’t specify that the controller is
>> >> > actually verifying the “desired state”, not “actual state”. And this
>> >> > controller’s desired state validation also applies in
>> >> >
>> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
>> >> > I’ve updated the KIP to make it clear.
>> >> >
>> >> > > JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
>> >> > bumped,
>> >> > ABORT markers are appended for all ongoing transactions, last mirror
>> >> epochs
>> >> > are persisted, and a MIRROR_PID_RESET record appended."
>> >> > This describes the state for STOPPED, not STOPPING.
>> >> >
>> >> > No, these are the tasks during STOPPING state, like what we
>> described in
>> >> > “Stop Mirror Topics” workflow. If we do these in STOPPED state, we
>> can’t
>> >> > safely move the readonly topic into writable because not all required
>> >> tasks
>> >> > are done. I think I know why it confuses you, it’s because all the
>> tasks
>> >> > are in the completed state. I’ve rephrased it to make it clear.
>> >> >
>> >> >
>> >> > > JR79. RPCs like DescribeClusterMirrors can take time. Should we
>> add a
>> >> > timeout field?
>> >> >
>> >> > Make sense. Added into `DescribeClusterMirrors` request, as well as
>> >> >
>> >>
>> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
>> >> > because we will query the partition states from coordinator for state
>> >> > validation in these APIs.
>> >> >
>> >> >
>> >> > > JR80. PauseMirrorTopicsOptions: Should it support patterns? The
>> >> command
>> >> > line seems to suppor it.
>> >> >
>> >> > Sorry, it’s a little confusing here. The “pattern” in
>> >> > startMirrorTopicsOptions or stopMirrorTopicsOptions is to update the
>> >> > “mirror.topics.include” and “mirror.topics.exclude” configuration for
>> >> topic
>> >> > pattern discovery periodically. For the PauseMirrorTopics and
>> >> > ResumeMirrorTopics in the clusterMirrorCommand, users can provide
>> regex
>> >> in
>> >> > the “--topics” option, and the clusterMirrorCommand will listTopic in
>> >> the
>> >> > destination cluster and match with the patterns users provided, and
>> then
>> >> > feeds the matched topics into the
>> PauseMirrorTopics/ResumeMirrorTopics
>> >> > requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics
>> >> supports
>> >> > regex in the ClusterMirrorComamnd, but we don’t need to add the
>> >> “patterns”.
>> >> >
>> >> >
>> >> > > JR81. AdminClient
>> >> > > JR81.1 ListClusterMirrors: We need to add the option to specify
>> >> different
>> >> > filters supported in the request.
>> >> >
>> >> > We already have the `--mirror` option. Added `--state` and
>> >> > `--sourceClusterId` options to support the other filters.
>> >> >
>> >> > > JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations:
>> The
>> >> > request doesn't support this option. Could we add other options
>> >> supported
>> >> > in the request?
>> >> >
>> >> > Currently, we have “includeAuthorizedOperations” in DescribeCluster,
>> >> > DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
>> >> > DescribeStreamsGroups requests. But none of them include the option
>> in
>> >> the
>> >> > tool to allow users to enable it. I think the reason is that we don’t
>> >> want
>> >> > to show this to users on the tool output. But having this field, we
>> >> allow
>> >> > the advanced users to invoke admin API with
>> includeAuthorizedOperations
>> >> > enabled to get the results they want.
>> >> >
>> >> >
>> >> > > JR81.3 LeaderStateDescription: Should it be named
>> >> > MirrorPartitionStateDescription? Also, could we make the fields
>> >> consistent
>> >> > with those in the response?
>> >> >
>> >> > OK. KIP is updated.
>> >> >
>> >> >
>> >> > Thank you,
>> >> > Luke
>> >> >
>> >> > On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <
>> dev@kafka.apache.org>
>> >> > wrote:
>> >> >
>> >> >> Hi, Luke,
>> >> >>
>> >> >> Thanks for the reply.
>> >> >>
>> >> >> JR59.1 Could you add the RPC to the workflow? The broker first
>> needs to
>> >> >> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get
>> the
>> >> >> mirror name and then issue DescribeClusterMirrors to get the LME.
>> >> >>
>> >> >> JR71. mirror.support.unclean.leader.election : The name is
>> confusing.
>> >> It's
>> >> >> also used in LOG_TRUNCATION, which is not related to unclean leader
>> >> >> election. Could we use a better name?
>> >> >>
>> >> >> JR72. DescribeClusterMirrors
>> >> >> JR72.1 A user may specify just the topic name when describing a
>> mirror.
>> >> >> Should we include the topic name in the request?
>> >> >> JR72.2 LastMirrorEpochLookups is a bit weird since the response
>> returns
>> >> >> either mirror state or mirror lag and LastMirrorEpochLookups doesn't
>> >> match
>> >> >> either.
>> >> >> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> >> JR72.4 "includeMirrorLags: source offset, destination offset, lag"
>> >> >> DescribeClusterMirrorsRequest doesn't include those fields. Also,
>> lag
>> >> >> should be omitted.
>> >> >>
>> >> >> JR73. MirrorPartitionState record, LastMirrorEpochs record and
>> >> >> MirrorPartitionStateRequest: The values for type and name are
>> >> incorrect.
>> >> >>
>> >> >> JR74. ListClusterMirrorsRequest
>> >> >> JR74.1 When is DesiredMirrorStateFilter used? There is no option to
>> >> >> specify
>> >> >> the desired state in the mirror-topic command.
>> >> >> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>> >> >>
>> >> >> JR75. ReadMirrorLags
>> >> >> JR75.1 Should it be named ReadMirrorOffsets since the response
>> doesn't
>> >> >> include lag?
>> >> >> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> >> JR75.3 Some places refer to this as ReadClusterMirrorLags. We need
>> to
>> >> be
>> >> >> consistent.
>> >> >>
>> >> >> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR
>> too?
>> >> >>
>> >> >> JR77. Delete Mirror: "3. The controller verifies that no topics are
>> in
>> >> >> MIRRORING or PAUSED state."
>> >> >> Is this necessary since step 2 has verified all partitions are in
>> >> STOPPED
>> >> >> state, which can only happen if the topics are in STOPPED state.
>> >> >>
>> >> >> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
>> >> >> bumped,
>> >> >> ABORT markers are appended for all ongoing transactions, last mirror
>> >> >> epochs
>> >> >> are persisted, and a MIRROR_PID_RESET record appended."
>> >> >> This describes the state for STOPPED, not STOPPING.
>> >> >>
>> >> >> JR79. RPCs like DescribeClusterMirrors can take time. Should we add
>> a
>> >> >> timeout field?
>> >> >>
>> >> >> JR80. PauseMirrorTopicsOptions: Should it support patterns? The
>> command
>> >> >> line seems to suppor it.
>> >> >>
>> >> >> JR81. AdminClient
>> >> >> JR81.1 ListClusterMirrors: We need to add the option to specify
>> >> different
>> >> >> filters supported in the request.
>> >> >> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations:
>> The
>> >> >> request doesn't support this option. Could we add other options
>> >> supported
>> >> >> in the request?
>> >> >> JR81.3 LeaderStateDescription: Should it be named
>> >> >> MirrorPartitionStateDescription? Also, could we make the fields
>> >> consistent
>> >> >> with those in the response?
>> >> >>
>> >> >> Jun
>> >> >>
>> >> >> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com>
>> wrote:
>> >> >>
>> >> >> > Hi Jun,
>> >> >> >
>> >> >> > Thanks for the good comments again.
>> >> >> > Answering them below.
>> >> >> >
>> >> >> > > JR51. "ULE_LOG_TRUNCATION When
>> >> mirror.support.unclean.leader.election
>> >> >> is
>> >> >> > enabled and the destination leader has log truncation, the state
>> will
>> >> >> move
>> >> >> > from MIRRORING to this state."
>> >> >> > This still suggests that divergeEpoch is only handled if
>> >> >> > mirror.support.unclean.leader.election
>> >> >> > is enabled.
>> >> >> >
>> >> >> >
>> >> >> > OK, KIP is updated to make it clear.
>> >> >> >
>> >> >> > > JR55. Should the mirror state and the mirror lags always be
>> exposed
>> >> >> > together? If so, we can remove IncludeMirrorState and
>> >> IncludeMirrorLags
>> >> >> > from DescribeClusterMirrorsRequest. If not, it's better to use two
>> >> >> separate
>> >> >> > RPCs.
>> >> >> >
>> >> >> >
>> >> >> > No, we sometimes only need MirrorState, and sometimes need both.
>> So
>> >> I’ll
>> >> >> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
>> >> >> > DescribeClusterMirrorsRequest. And we will use ReadMirrorStates
>> and
>> >> >> > ReadMirrorLags to get each info.
>> >> >> >
>> >> >> >
>> >> >> > > DescribeClusterMirrorsRequest should be used in the Describe
>> Mirror
>> >> >> flow,
>> >> >> > right? Currently it's missing. Also, I assume the admin client
>> >> refers to
>> >> >> > the one in the kafka-cluster-mirrors tool. But it seems that
>> >> sometimes
>> >> >> it
>> >> >> > also refers to the admin client on the broker? Perhaps we can use
>> >> 'the
>> >> >> > tool' instead to make it clear.
>> >> >> >
>> >> >> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
>> >> >> > And sorry for the confusion. I’d like to keep the admin client as
>> the
>> >> >> one
>> >> >> > invoking from the tool (or user’s direct admin client API
>> >> invocation).
>> >> >> On
>> >> >> > the broker side, we actually use InterBrokerSendThread to invoke
>> >> these
>> >> >> RPCs
>> >> >> > or get directly from MetadataImage. I’ve updated the KIP to make
>> it
>> >> >> clear.
>> >> >> >
>> >> >> >
>> >> >> > > JR58. BumpLeaderEpochs: Why does it need to be issued to the
>> >> broker,
>> >> >> > instead of just to the controller?
>> >> >> >
>> >> >> > Yes, we only need to issue to the controller. Updated to the
>> >> >> > BumpLeaderEpochs API.
>> >> >> >
>> >> >> >
>> >> >> > > JR59. Start Mirror Topics:
>> >> >> > > JR59.1 "During this state, the broker queries the source cluster
>> >> for
>> >> >> last
>> >> >> > mirror epochs".
>> >> >> > Could you describe which RPC is issued to find the LME?
>> >> >> >
>> >> >> > We’re using describeClusterMirror API via admin client here.
>> >> >> >
>> >> >> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
>> request
>> >> >> > doesn't
>> >> >> > contain a replication factor.
>> >> >> >
>> >> >> > We use “default.replication.factor” when topic creation, so the
>> >> >> > INVALID_REPLICATION_FACTOR will be thrown like normal topic
>> creation.
>> >> >> Also,
>> >> >> > because we require the replicas for the partition assignment to be
>> >> all
>> >> >> > active and not in fenced or controlled shutdown(JR35), we’ll throw
>> >> >> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
>> >> >> >
>> >> >> >
>> >> >> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used
>> by a
>> >> >> > different topic name"
>> >> >> > Should we also return this error when the topic name is used as a
>> >> >> regular
>> >> >> > topic?
>> >> >> >
>> >> >> > There are 2 errors we will throw when topic id <-> topic name
>> >> mismatch.
>> >> >> > Suppose provided topic id: ID_A, topic name: NAME_A
>> >> >> > 1. Destination cluster exists topic id: ID_A, but topic name is:
>> >> NAME_B
>> >> >> > TOPIC_ALREADY_EXISTS will be thrown
>> >> >> > 2. Destination cluster exists topic id: ID_B, but topic name is:
>> >> NAME_A
>> >> >> > INCONSISTENT_TOPIC_ID will be thrown
>> >> >> >
>> >> >> > Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS
>> error
>> >> >> for
>> >> >> > these 2 cases makes more sense. KIP updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR60. CreateClusterMirrorRequest: When will it receive an
>> >> >> > INVALID_CLUSTER_MIRROR_STATES error not covered by
>> >> >> > CLUSTER_MIRROR_ALREADY_EXISTS?
>> >> >> >
>> >> >> >
>> >> >> > Sorry, createClusterMirrorRequest doesn’t throw
>> >> >> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate
>> the
>> >> >> mirror
>> >> >> > states when creating a cluster mirror. We only care if the “mirror
>> >> name”
>> >> >> > exists or not. KIP updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we
>> just
>> >> >> remove
>> >> >> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> >> >> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe
>> >> the
>> >> >> > specific state causing the error.
>> >> >> >
>> >> >> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following
>> >> errors:
>> >> >> > a. MIRROR_TOPIC_ALREADY_PAUSED
>> >> >> > b. MIRROR_TOPIC_NOT_PAUSED,
>> >> >> > c. MIRROR_TOPIC_ALREADY_STOPPED,
>> >> >> > d. CLUSTER_MIRROR_NOT_EMPTY
>> >> >> > e. MIRROR_TOPIC_NOT_STOPPED
>> >> >> >
>> >> >> > KIP is updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
>> >> >> requests
>> >> >> > (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
>> >> Could
>> >> >> we
>> >> >> > make them consistent?
>> >> >> >
>> >> >> >
>> >> >> > Good catch. The updated errors for the new added RPC are this:
>> >> >> >
>> >> >> > CreateClusterMirror: NOT_CONTROLLER
>> >> >> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >> >
>> >> >> >
>> >> >> > > JR63. StopMirrorTopics:
>> >> >> > > JR63.1 "The broker validates that all target topic partitions
>> are
>> >> in
>> >> >> > either
>> >> >> > LOG_TRUNCATION or MIRRORING or FAILED state."
>> >> >> > Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED
>> >> can.
>> >> >> >
>> >> >> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is
>> >> updated.
>> >> >> >
>> >> >> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error
>> since
>> >> >> PAUSED
>> >> >> > -> STOPPED is valid?
>> >> >> >
>> >> >> > Yes, it is not correct. KIP updated to remove this error.
>> >> >> >
>> >> >> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
>> >> stopped
>> >> >> > state"
>> >> >> > This is weird. The reason to issue StopMirrorTopics is that it's
>> not
>> >> in
>> >> >> > stopped state.
>> >> >> >
>> >> >> > Indeed. KIP is updated to only include
>> INVALID_CLUSTER_MIRROR_STATES
>> >> >> > error, and will be thrown when states are not in MIRRORING or
>> PAUSED
>> >> >> state.
>> >> >> >
>> >> >> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
>> >> >> derived
>> >> >> > from SourceOffset and DestinationOffset.
>> >> >> >
>> >> >> > True. Removed.
>> >> >> >
>> >> >> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems
>> >> that we
>> >> >> > could just set the partition level state to STOPPED.
>> >> >> >
>> >> >> > Nice catch. We don’t use it at all. That is some old leftover
>> field
>> >> we
>> >> >> > forgot to remove. KIP updated.
>> >> >> >
>> >> >> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be
>> >> sent
>> >> >> to
>> >> >> > the controller?
>> >> >> >
>> >> >> > No, these 2 only send to the broker. KIP updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and
>> >> StateEpoch?
>> >> >> It
>> >> >> > seems both are used for fencing and fencing based on the latter is
>> >> >> > stronger.
>> >> >> >
>> >> >> > Yes, basically the StateEpoch can fence for most cases. The
>> >> LeaderEpoch
>> >> >> > fencing is used for cases like:
>> >> >> >
>> >> >> > 1. b1 is the current leader for foo-0, preparing write state from
>> >> >> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
>> >> >> > 2. leadership change to b2 for foo-0, LE changes to 2.
>> >> >> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1 }.
>> >> >> > 4. The coordinator receives the read from b2 in step (3), the
>> highest
>> >> >> > lastSeen LE for foo-0 updates to 2 in coordinator.
>> >> >> > 5. The writes state from b1 in step (1) arrives the coordinator,
>> but
>> >> it
>> >> >> > rejects b1 writes because the highest lastSeen LE is 2, which is
>> > 1
>> >> in
>> >> >> the
>> >> >> > request.
>> >> >> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2,
>> stateEpoch: 1
>> >> and
>> >> >> > succeeds.
>> >> >> >
>> >> >> > The leaderEpoch fence is to avoid any stale broker writes after
>> the
>> >> new
>> >> >> > leader epoch read. This cannot be protected by the stateEpoch. The
>> >> >> > stateEpoch can prevent the write with older stateEpoch. So in the
>> >> >> situation
>> >> >> > above, the stateEpoch cannot determine if we should accept b1
>> writes
>> >> or
>> >> >> > not, but leaderEpoch can. By the way, we borrowed this concept
>> from
>> >> >> > KIP-932:
>> >> >> >
>> >> >>
>> >>
>> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
>> <https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!t3M02ge9e-mRqsshxasG_K0qLi9sX21suoBd69nLUl_gWUvn9ZLLekJiQspa_q3gTVSzWyUQBQfQLHk$>
>> >> <
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rX3-NwDU3s0U7POQYH1R-DvASnCPTTtFRWxZTVaS-PQHg-C1TAcu9xMe3MQF4fIXGO67WKOqrlz-lc8$
>> >
>> >> >> <
>> >>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!u8EkPGXm8_VKdltXy8lrIW14mR2kgYIoaBPD7LO2uAN6rjhrfCX37Pfa6r85QPpSwSPh4V6eW_bKlns$
>> >> >
>> >> >> > <
>> >> >>
>> >>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > > JR68. Could we make the usage of topicId and topicName in the
>> RPCs
>> >> >> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> >> >> > BumpLeaderEpochsReponse uses TopicName.
>> >> >> >
>> >> >> > Yes, we have made:
>> >> >> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use TopicName.
>> >> >> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse use
>> >> >> > TopicName.
>> >> >> > 3. MirrorTopicStateChangeRecord uses TopicName because we will do
>> >> >> > validation before writing the metadata log.
>> >> >> >
>> >> >> >
>> >> >> > > JR69. MIRROR_PID_RESET:
>> >> >> > > JR69.1 Could you specify the value of the control record's type?
>> >> >> >
>> >> >> > It’s MIRROR_PID_RESET(7). KIP updated.
>> >> >> >
>> >> >> > > JR69.2 How is the SourceClusterId field used?
>> >> >> >
>> >> >> > I’m also surprised to see this field in the MirrorPidResetRecord.
>> I
>> >> >> traced
>> >> >> > back to the internal PR we added for this field, and confirmed it
>> is
>> >> not
>> >> >> > used since it was added. Sorry for the confusion and nice catch!
>> KIP
>> >> >> > updated.
>> >> >> >
>> >> >> > > JR70. MirrorPartitionState: type should be "data" not
>> >> >> "coordinator-key",
>> >> >> > right? Ditto for LastMirrorEpochs.
>> >> >> > We store both MirrorPartitionState and LastMirrorEpochs records
>> into
>> >> >> > internal __mirror_state topic, and rely on Coordinator Runtime to
>> >> >> > write/read the records, like how `OffsetCommit` and
>> `GroupMetadata`
>> >> >> work.
>> >> >> > So I think we should keep them as
>> >> "coordinator-key"/"coordinator-value"
>> >> >> > type.
>> >> >> >
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Luke
>> >> >> >
>> >> >> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <
>> >> dev@kafka.apache.org>
>> >> >> > wrote:
>> >> >> >
>> >> >> >> Hi, Luke,
>> >> >> >>
>> >> >> >> Thanks for the reply.
>> >> >> >>
>> >> >> >> JR51. "ULE_LOG_TRUNCATION When
>> >> mirror.support.unclean.leader.election
>> >> >> is
>> >> >> >> enabled and the destination leader has log truncation, the state
>> >> will
>> >> >> move
>> >> >> >> from MIRRORING to this state."
>> >> >> >> This still suggests that divergeEpoch is only handled if
>> >> >> >> mirror.support.unclean.leader.election
>> >> >> >> is enabled.
>> >> >> >>
>> >> >> >> JR55. Should the mirror state and the mirror lags always be
>> exposed
>> >> >> >> together? If so, we can remove IncludeMirrorState and
>> >> IncludeMirrorLags
>> >> >> >> from DescribeClusterMirrorsRequest. If not, it's better to use
>> two
>> >> >> >> separate
>> >> >> >> RPCs.
>> >> >> >>
>> >> >> >> DescribeClusterMirrorsRequest should be used in the Describe
>> Mirror
>> >> >> flow,
>> >> >> >> right? Currently it's missing. Also, I assume the admin client
>> >> refers
>> >> >> to
>> >> >> >> the one in the kafka-cluster-mirrors tool. But it seems that
>> >> sometimes
>> >> >> it
>> >> >> >> also refers to the admin client on the broker? Perhaps we can use
>> >> 'the
>> >> >> >> tool' instead to make it clear.
>> >> >> >>
>> >> >> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the
>> broker,
>> >> >> >> instead of just to the controller?
>> >> >> >>
>> >> >> >> JR59. Start Mirror Topics:
>> >> >> >> JR59.1 "During this state, the broker queries the source cluster
>> for
>> >> >> last
>> >> >> >> mirror epochs".
>> >> >> >> Could you describe which RPC is issued to find the LME?
>> >> >> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
>> request
>> >> >> >> doesn't
>> >> >> >> contain a replication factor.
>> >> >> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used
>> by a
>> >> >> >> different topic name"
>> >> >> >> Should we also return this error when the topic name is used as a
>> >> >> regular
>> >> >> >> topic?
>> >> >> >>
>> >> >> >> JR60. CreateClusterMirrorRequest: When will it receive an
>> >> >> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
>> >> >> >> CLUSTER_MIRROR_ALREADY_EXISTS?
>> >> >> >>
>> >> >> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
>> >> >> remove
>> >> >> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> >> >> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to
>> describe
>> >> the
>> >> >> >> specific state causing the error.
>> >> >> >>
>> >> >> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
>> >> >> requests
>> >> >> >> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
>> >> >> Could we
>> >> >> >> make them consistent?
>> >> >> >>
>> >> >> >> JR63. StopMirrorTopics:
>> >> >> >> JR63.1 "The broker validates that all target topic partitions
>> are in
>> >> >> >> either
>> >> >> >> LOG_TRUNCATION or MIRRORING or FAILED state."
>> >> >> >> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED
>> >> can.
>> >> >> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error
>> since
>> >> >> PAUSED
>> >> >> >> -> STOPPED is valid?
>> >> >> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
>> stopped
>> >> >> >> state"
>> >> >> >> This is weird. The reason to issue StopMirrorTopics is that it's
>> >> not in
>> >> >> >> stopped state.
>> >> >> >>
>> >> >> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
>> >> derived
>> >> >> >> from SourceOffset and DestinationOffset.
>> >> >> >>
>> >> >> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems
>> that
>> >> we
>> >> >> >> could just set the partition level state to STOPPED.
>> >> >> >>
>> >> >> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be
>> >> sent
>> >> >> to
>> >> >> >> the controller?
>> >> >> >>
>> >> >> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and
>> >> StateEpoch? It
>> >> >> >> seems both are used for fencing and fencing based on the latter
>> is
>> >> >> >> stronger.
>> >> >> >>
>> >> >> >> JR68. Could we make the usage of topicId and topicName in the
>> RPCs
>> >> >> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> >> >> >> BumpLeaderEpochsReponse uses TopicName.
>> >> >> >>
>> >> >> >> JR69. MIRROR_PID_RESET:
>> >> >> >> JR69.1 Could you specify the value of the control record's type?
>> >> >> >> JR69.2 How is the SourceClusterId field used?
>> >> >> >>
>> >> >> >> JR70. MirrorPartitionState: type should be "data" not
>> >> >> "coordinator-key",
>> >> >> >> right? Ditto for LastMirrorEpochs.
>> >> >> >>
>> >> >> >> Jun
>> >> >> >>
>> >> >> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com>
>> >> wrote:
>> >> >> >>
>> >> >> >> > Hi Jun,
>> >> >> >> >
>> >> >> >> > Thanks for the comments.
>> >> >> >> > Answering them below.
>> >> >> >> >
>> >> >> >> > > JR36. Sounds good. Should we remove
>> >> >> >> MirrorLeaderEpochExceededException?
>> >> >> >> >
>> >> >> >> > Sorry, it’s a confusing name. The
>> >> MirrorLeaderEpochExceededException
>> >> >> is
>> >> >> >> > used and thrown when the fetched mirror batch leader epoch >
>> local
>> >> >> >> leader
>> >> >> >> > epoch in destination cluster. And then we’ll stop mirroring and
>> >> move
>> >> >> the
>> >> >> >> > partition to EPOCH_FENSING state to wait for local leader epoch
>> >> >> bump. I
>> >> >> >> > think since we don’t have the concept of `MirrorLeaderEpoch`
>> >> anymore,
>> >> >> >> the
>> >> >> >> > `MirrorLeaderEpochExceededException` should not cause confusion
>> >> now.
>> >> >> >> I’ll
>> >> >> >> > keep the name for the exception.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR51. Sounds good. Do we plan to always set lastFetchedEpoch
>> in
>> >> the
>> >> >> >> > mirror
>> >> >> >> > fetch request issued to the source cluster, independent of the
>> >> value
>> >> >> of
>> >> >> >> > mirror.support.unclean.leader.election? It's needed to detect
>> >> >> divergence
>> >> >> >> > during failover. However, if it's also set when
>> >> >> >> > mirror.support.unclean.leader.election=false,
>> >> >> >> > an unclean leader election in the source cluster can halt the
>> >> >> mirroring
>> >> >> >> > since we don’t do anything with the returned divergeEpoch in
>> that
>> >> >> mode
>> >> >> >> > and continue
>> >> >> >> > the next fetch.
>> >> >> >> >
>> >> >> >> > > Do we plan to always set lastFetchedEpoch in the mirror
>> >> >> >> > fetch request issued to the source cluster, independent of the
>> >> value
>> >> >> of
>> >> >> >> > Mirror.support.unclean.leader.election?
>> >> >> >> >
>> >> >> >> > Yes, we always set lastFetchedEpoch in the mirror fetch
>> request.
>> >> >> >> >
>> >> >> >> > > However, if it's also set when
>> >> >> >> > mirror.support.unclean.leader.election=false,
>> >> >> >> > an unclean leader election in the source cluster can halt the
>> >> >> mirroring
>> >> >> >> > since we don’t do anything with the returned divergeEpoch in
>> that
>> >> >> mode
>> >> >> >> > and continue the next fetch.
>> >> >> >> >
>> >> >> >> > Sorry I didn’t make it clear. When
>> >> >> >> > “mirror.support.unclean.leader.election=false” and a
>> >> “divergeEpoch”
>> >> >> is
>> >> >> >> set
>> >> >> >> > in fetch response, the leader node will handle the divergeEpoch
>> >> and
>> >> >> then
>> >> >> >> > continue the next fetch as usual. The difference with
>> >> >> >> > “mirror.support.unclean.leader.election=true” is that “should
>> we
>> >> wait
>> >> >> >> for
>> >> >> >> > ALL replicas caught up (complete truncation)”. Without the
>> >> waiting,
>> >> >> it’s
>> >> >> >> > possible the inconsistent log issue in KAFKA-18723 could happen
>> >> >> between
>> >> >> >> > leader and follower nodes in the destination cluster because we
>> >> don’t
>> >> >> >> have
>> >> >> >> > “MirrorLeaderEpoch” concept now to reject higher leader epoch
>> >> >> batches.
>> >> >> >> That
>> >> >> >> > is, we won't halt the mirroring if "divergeEpoch" is set but
>> >> >> >> > “mirror.support.unclean.leader.election=false”.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR55. The DescribeClusterMirrorsRequest still returns both
>> the
>> >> >> mirror
>> >> >> >> > partition state and the mirror partition leader state. The
>> >> problem is
>> >> >> >> that
>> >> >> >> > if a caller only needs one of the state, the other state
>> becomes
>> >> >> >> redundant.
>> >> >> >> > For example, when starting the mirror, to obtain LME, we only
>> need
>> >> >> the
>> >> >> >> > mirror partition state. Here is an alternative design: We
>> expose
>> >> the
>> >> >> >> mirror
>> >> >> >> > partition state through ReadMirrorState. The client will use
>> the
>> >> >> >> > FindCoordinatorRequest
>> >> >> >> > to find the right mirror partition coordinator to issue that
>> >> >> request. We
>> >> >> >> > expose the mirror partition leader state through the
>> >> >> >> ReadClusterMirrorLags
>> >> >> >> > request. The client can use the metadata request to find the
>> >> >> partition
>> >> >> >> > leader to issue that request.
>> >> >> >> >
>> >> >> >> > > In the describing mirror flow, the admin client first issues
>> the
>> >> >> >> > ListMirrorRequest to find all the mirrors and the topics being
>> >> >> >> mirrored. It
>> >> >> >> > then issues the metadata request to find the partitions for
>> those
>> >> >> topics
>> >> >> >> > and issues a FindCoordinatorRequest to find the mirror
>> partition
>> >> >> state
>> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
>> >> >> partition
>> >> >> >> > state coordinator and ReadClusterMirrorLags to the partition
>> >> leader.
>> >> >> >> >
>> >> >> >> > > In the starting mirror flow, the partition leader in the
>> >> >> destination
>> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name
>> based
>> >> on
>> >> >> its
>> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to find the
>> >> >> mirror
>> >> >> >> > partition state coordinators. Finally, it issues
>> ReadMirrorState
>> >> to
>> >> >> the
>> >> >> >> > mirror partition state coordinator to get the LME.
>> >> >> >> >
>> >> >> >> > > We can also improve the ListMirror request to accept a filter
>> >> >> based on
>> >> >> >> > mirror name, source cluster ID and topic name.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Answers:
>> >> >> >> >
>> >> >> >> > > The DescribeClusterMirrorsRequest still returns both the
>> mirror
>> >> >> >> > partition state and the mirror partition leader state. The
>> >> problem is
>> >> >> >> that
>> >> >> >> > if a caller only needs one of the state, the other state
>> becomes
>> >> >> >> redundant.
>> >> >> >> > For example, when starting the mirror, to obtain LME, we only
>> need
>> >> >> the
>> >> >> >> > mirror partition state.
>> >> >> >> >
>> >> >> >> > Good point!
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > Here is an alternative design: We expose the mirror
>> >> >> >> > partition state through ReadMirrorState. The client will use
>> the
>> >> >> >> > FindCoordinatorRequest
>> >> >> >> > to find the right mirror partition coordinator to issue that
>> >> >> request. We
>> >> >> >> > expose the mirror partition leader state through the
>> >> >> >> ReadClusterMirrorLags
>> >> >> >> > request. The client can use the metadata request to find the
>> >> >> partition
>> >> >> >> > leader to issue that request.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Sounds good. But we try not to expose
>> >> >> >> > ReadMirrorState/ReadClusterMirrorLags directly to the admin
>> client
>> >> >> >> because
>> >> >> >> > users might get confused about these 2 requests, and they are
>> also
>> >> >> both
>> >> >> >> > related to describeClusterMirror. I think we can add 2
>> additional
>> >> >> >> fields to
>> >> >> >> > describeClusterMirrorRequest: “includeMirrorStates” and
>> >> >> >> > “includeMirrorLags”. So the admin client can include either
>> one or
>> >> >> both
>> >> >> >> of
>> >> >> >> > them in the DescrbieClusterMirrorRequest based on the provided
>> >> >> options.
>> >> >> >> KIP
>> >> >> >> > is updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > In the describing mirror flow, the admin client first issues
>> the
>> >> >> >> > ListMirrorRequest to find all the mirrors and the topics being
>> >> >> >> mirrored. It
>> >> >> >> > then issues the metadata request to find the partitions for
>> those
>> >> >> topics
>> >> >> >> > and issues a FindCoordinatorRequest to find the mirror
>> partition
>> >> >> state
>> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
>> >> >> partition
>> >> >> >> > state coordinator and ReadClusterMirrorLags to the partition
>> >> leader.
>> >> >> >> >
>> >> >> >> > Makes sense. KIP is updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > In the starting mirror flow, the partition leader in the
>> >> >> destination
>> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name
>> based
>> >> on
>> >> >> its
>> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to find the
>> >> >> mirror
>> >> >> >> > partition state coordinators. Finally, it issues
>> ReadMirrorState
>> >> to
>> >> >> the
>> >> >> >> > mirror partition state coordinator to get the LME.
>> >> >> >> >
>> >> >> >> > Make sense. So we can send DescribeClusterMirrorRequest with
>> >> >> >> > “includeMirrorStates” only.
>> >> >> >> >
>> >> >> >> > > We can also improve the ListMirror request to accept a filter
>> >> >> based on
>> >> >> >> > mirror name, source cluster ID and topic name.
>> >> >> >> >
>> >> >> >> > Good suggestion, just like what ListGroupsRequest does.
>> >> >> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
>> >> >> >> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest.
>> KIP
>> >> >> >> updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR56. Could you list the ACL needed for each new RPC?
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Yes, it’s already listed in “Security Control” section.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR57. In the command workflow, it sometimes says "The user
>> >> sends"
>> >> >> and
>> >> >> >> > some
>> >> >> >> > other times says "The admin client sends". It's better to use
>> "The
>> >> >> admin
>> >> >> >> > client" consistently in all those places.
>> >> >> >> >
>> >> >> >> > Agree. KIP updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Thank you,
>> >> >> >> > Luke
>> >> >> >> >
>> >> >> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <
>> >> dev@kafka.apache.org
>> >> >> >
>> >> >> >> > wrote:
>> >> >> >> >
>> >> >> >> >> Hi, Luke,
>> >> >> >> >>
>> >> >> >> >> Thanks for the reply.
>> >> >> >> >>
>> >> >> >> >> JR36. Sounds good. Should we remove
>> >> >> MirrorLeaderEpochExceededException?
>> >> >> >> >>
>> >> >> >> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch
>> in
>> >> the
>> >> >> >> mirror
>> >> >> >> >> fetch request issued to the source cluster, independent of the
>> >> >> value of
>> >> >> >> >> mirror.support.unclean.leader.election? It's needed to detect
>> >> >> >> divergence
>> >> >> >> >> during failover. However, if it's also set when
>> >> >> >> >> mirror.support.unclean.leader.election=false,
>> >> >> >> >> an unclean leader election in the source cluster can halt the
>> >> >> mirroring
>> >> >> >> >> since we don’t do anything with the returned divergeEpoch in
>> that
>> >> >> mode
>> >> >> >> >> and continue
>> >> >> >> >> the next fetch.
>> >> >> >> >>
>> >> >> >> >> JR55. The DescribeClusterMirrorsRequest still returns both the
>> >> >> mirror
>> >> >> >> >> partition state and the mirror partition leader state. The
>> >> problem
>> >> >> is
>> >> >> >> that
>> >> >> >> >> if a caller only needs one of the state, the other state
>> becomes
>> >> >> >> >> redundant.
>> >> >> >> >> For example, when starting the mirror, to obtain LME, we only
>> >> need
>> >> >> the
>> >> >> >> >> mirror partition state. Here is an alternative design: We
>> expose
>> >> the
>> >> >> >> >> mirror
>> >> >> >> >> partition state through ReadMirrorState. The client will use
>> the
>> >> >> >> >> FindCoordinatorRequest
>> >> >> >> >> to find the right mirror partition coordinator to issue that
>> >> >> request.
>> >> >> >> We
>> >> >> >> >> expose the mirror partition leader state through the
>> >> >> >> ReadClusterMirrorLags
>> >> >> >> >> request. The client can use the metadata request to find the
>> >> >> partition
>> >> >> >> >> leader to issue that request.
>> >> >> >> >>
>> >> >> >> >> In the describing mirror flow, the admin client first issues
>> the
>> >> >> >> >> ListMirrorRequest to find all the mirrors and the topics being
>> >> >> >> mirrored.
>> >> >> >> >> It
>> >> >> >> >> then issues the metadata request to find the partitions for
>> those
>> >> >> >> topics
>> >> >> >> >> and issues a FindCoordinatorRequest to find the mirror
>> partition
>> >> >> state
>> >> >> >> >> coordinators. Finally, it issues ReadMirrorState to the mirror
>> >> >> >> partition
>> >> >> >> >> state coordinator and ReadClusterMirrorLags to the partition
>> >> leader.
>> >> >> >> >>
>> >> >> >> >> In the starting mirror flow, the partition leader in the
>> >> destination
>> >> >> >> >> cluster issues a ListMirrorRequest to find the mirror name
>> based
>> >> on
>> >> >> its
>> >> >> >> >> cluster id. It then issues a FindCoordinatorRequest to find
>> the
>> >> >> mirror
>> >> >> >> >> partition state coordinators. Finally, it issues
>> ReadMirrorState
>> >> to
>> >> >> the
>> >> >> >> >> mirror partition state coordinator to get the LME.
>> >> >> >> >>
>> >> >> >> >> We can also improve the ListMirror request to accept a filter
>> >> based
>> >> >> on
>> >> >> >> >> mirror name, source cluster ID and topic name.
>> >> >> >> >>
>> >> >> >> >> JR56. Could you list the ACL needed for each new RPC?
>> >> >> >> >>
>> >> >> >> >> JR57. In the command workflow, it sometimes says "The user
>> sends"
>> >> >> and
>> >> >> >> some
>> >> >> >> >> other times says "The admin client sends". It's better to use
>> >> "The
>> >> >> >> admin
>> >> >> >> >> client" consistently in all those places.
>> >> >> >> >>
>> >> >> >> >> Jun
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com>
>> >> wrote:
>> >> >> >> >>
>> >> >> >> >> > Hi Jun,
>> >> >> >> >> >
>> >> >> >> >> > Thanks for your comments.
>> >> >> >> >> > Answering them below.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> >> >> >> specifically
>> >> >> >> >> > designed for the unclean leader election support in Cluster
>> >> >> Mirroring
>> >> >> >> >> > because the unclean leader election will break the
>> assumption
>> >> that
>> >> >> >> the
>> >> >> >> >> > committed data won’t be truncated."
>> >> >> >> >> > If the source cluster has an unclean leader election, the
>> >> >> destination
>> >> >> >> >> > cluster will detect the inconsistency and truncate the log
>> at
>> >> the
>> >> >> >> >> > destination leader. We can just
>> >> >> >> >> > extend mirror.support.unclean.leader.election to cover this
>> >> case,
>> >> >> >> i.e.,
>> >> >> >> >> we
>> >> >> >> >> > wait until all replicas' log end offsets are reduced to the
>> >> target
>> >> >> >> >> offset
>> >> >> >> >> > before resuming mirroring. This avoids the inconsistency.
>> >> >> >> >> >
>> >> >> >> >> > If the destination cluster has an unclean leader election,
>> it
>> >> will
>> >> >> >> >> > re-fetch the same data from the source. There won't be any
>> >> >> >> inconsistency
>> >> >> >> >> > either.
>> >> >> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Yes, that works, too. I agree this is better because the
>> >> unclean
>> >> >> >> leader
>> >> >> >> >> > election is rare, we don't want to occupy a field just for
>> it.
>> >> >> >> >> >
>> >> >> >> >> > So to achieve this, we have to:
>> >> >> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
>> >> >> >> >> > 2. When the destination cluster leader completes a log
>> >> truncation
>> >> >> >> after
>> >> >> >> >> a
>> >> >> >> >> > fetch response due to divergeEpoch, we know it must be
>> caused
>> >> by
>> >> >> >> unclean
>> >> >> >> >> > leader election in the source cluster because we only fetch
>> >> >> committed
>> >> >> >> >> > records. If “mirror.support.unclean.leader.election=false”,
>> we
>> >> >> don’t
>> >> >> >> do
>> >> >> >> >> > anything and continue the next fetch. If
>> >> >> >> >> > “mirror.support.unclean.leader.election=true”, we remove the
>> >> >> mirror
>> >> >> >> >> fetcher
>> >> >> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
>> >> >> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas
>> to
>> >> be
>> >> >> >> caught
>> >> >> >> >> up
>> >> >> >> >> > into ISR and reduced to the expected LEO.
>> >> >> >> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
>> >> >> >> >> >
>> >> >> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch
>> >> API.
>> >> >> >> >> > KIP is updated.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR51. Make sense. To truncate based on LME, we need to
>> issue
>> >> >> >> another
>> >> >> >> >> RPC
>> >> >> >> >> > to translate the LME to an offset. It would be useful to
>> >> include
>> >> >> >> that in
>> >> >> >> >> > the workflow description.
>> >> >> >> >> >
>> >> >> >> >> > About the additional RPC to translate the LME to the offset
>> >> (i.e.
>> >> >> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on this
>> >> to do
>> >> >> >> the
>> >> >> >> >> > offset translation. We use the local leader epoch cache in
>> the
>> >> >> >> >> destination
>> >> >> >> >> > cluster to translate the epoch to save the RPC call. This
>> works
>> >> >> >> because
>> >> >> >> >> of
>> >> >> >> >> > an invariant the LME truncation protocol maintains: for any
>> >> leader
>> >> >> >> >> epoch ≤
>> >> >> >> >> > LME, that epoch's data belongs to exactly one cluster,
>> either
>> >> it
>> >> >> >> >> originated
>> >> >> >> >> > there, or it was mirrored. This holds even across repeated
>> >> mirror
>> >> >> ->
>> >> >> >> >> > reverse-mirror -> mirror cycles, since each direction's
>> >> >> >> LOG_TRUNCATION
>> >> >> >> >> > phase re-establishes the invariant before that cluster
>> starts
>> >> >> >> mirroring.
>> >> >> >> >> > With this invariant, the replication protocol can find out
>> the
>> >> >> >> diverge
>> >> >> >> >> > point and truncate the log correctly when mirroring. For
>> >> example:
>> >> >> >> >> >
>> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> >> > Offset 2: A (LE: 0)
>> >> >> >> >> > Offset 3: A (LE: 1)
>> >> >> >> >> >
>> >> >> >> >> > Cluster A crashes while Cluster B mirrors to offset 1.
>> Failover
>> >> >> to B
>> >> >> >> and
>> >> >> >> >> > store the LME=0 in cluster B. When cluster A recovers and
>> >> starts
>> >> >> the
>> >> >> >> >> > reverse mirroring, it’ll query the LME from B and get back
>> >> LME: 0.
>> >> >> >> Then
>> >> >> >> >> > cluster A gets the end offset for epoch 0 from
>> >> leaderEpochCache,
>> >> >> and
>> >> >> >> >> gets
>> >> >> >> >> > offset 2. So offset 3 is truncated. The log is like this
>> now:
>> >> >> >> >> >
>> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
>> >> >> >> >> > Offset 3: B (LE:
>> >> 10)
>> >> >> >> >> >
>> >> >> >> >> > When mirror starts, cluster A sends the fetch request with {
>> >> >> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the
>> >> divergingEpoch
>> >> >> >> check
>> >> >> >> >> in
>> >> >> >> >> > the cluster B queries the endOffset for leader epoch 0, and
>> >> found
>> >> >> the
>> >> >> >> >> end
>> >> >> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so
>> returns
>> >> >> fetch
>> >> >> >> >> > response with divergingEpoch info to cluster A for log
>> >> >> truncation. In
>> >> >> >> >> the
>> >> >> >> >> > end, offset 2 in cluster A will be truncated and then log
>> >> >> converges
>> >> >> >> with
>> >> >> >> >> > cluster B.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit
>> >> >> strange
>> >> >> >> >> > because it mixes querying the mirror partition state with
>> >> querying
>> >> >> >> the
>> >> >> >> >> > mirror partition leader. Another option is to use
>> >> >> >> >> ReadMirrorStatesRequest
>> >> >> >> >> > to query the mirror partition state and a new request to
>> query
>> >> the
>> >> >> >> >> mirror
>> >> >> >> >> > partition leader. If we want the adminClient to query the
>> right
>> >> >> >> >> ClusterMirrorCoordinator
>> >> >> >> >> > directly, we can extend the FindCoordinatorRequest to
>> support
>> >> >> that.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Good suggestion. Yes, we need to fan out to all brokers
>> >> because we
>> >> >> >> have
>> >> >> >> >> to
>> >> >> >> >> > collect info from mirror partition leaders and mirror
>> >> >> coordinators.
>> >> >> >> If
>> >> >> >> >> we
>> >> >> >> >> > create a new API to contact mirror partition leaders, it
>> will
>> >> make
>> >> >> >> the
>> >> >> >> >> API
>> >> >> >> >> > cleaner and clearer. So we will do this:
>> >> >> >> >> >
>> >> >> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields
>> from
>> >> >> >> >> > DescribeClusterMirror request/response
>> >> >> >> >> >
>> >> >> >> >> > b. Add a new API: ReadClusterMirrorLags API to request for
>> >> >> >> SourceOffset,
>> >> >> >> >> > DestinationOffset, and Lag info.
>> >> >> >> >> >
>> >> >> >> >> > The workflow for admin client describeClusterMirror will be
>> >> like
>> >> >> >> this:
>> >> >> >> >> > 1. adminClient sends describeClusterMirror to the
>> >> leastLoadedNode
>> >> >> >> broker
>> >> >> >> >> > 2. The received broker collects the current mirrored
>> >> partitions,
>> >> >> and
>> >> >> >> >> > groups them by coordinator node and by partition leader
>> node.
>> >> >> >> >> > 3. The broker sends ReadMirrorState request to the
>> coordinator
>> >> >> nodes
>> >> >> >> to
>> >> >> >> >> > get the mirror state related info
>> >> >> >> >> > 4. The broker sends ReadClusterMirrorLags request to the
>> mirror
>> >> >> >> >> partition
>> >> >> >> >> > leader nodes to get the mirror lag, source offset,
>> destination
>> >> >> offset
>> >> >> >> >> > (which updates on every fetch response).
>> >> >> >> >> > 5. The broker merge all the response into
>> describeClusterMirror
>> >> >> >> response
>> >> >> >> >> > and send to admin client
>> >> >> >> >> >
>> >> >> >> >> > KIP is updated. Thanks for the suggestion.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR55. For __mirror_state topic, could we describe whether
>> it
>> >> is
>> >> >> >> >> > compacted?
>> >> >> >> >> >
>> >> >> >> >> > Yes, it is a compacted topic. I made it much clearer in the
>> >> >> >> >> > “ClusterMirrorCoordinator” section.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Thank you,
>> >> >> >> >> > Luke
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
>> >> >> dev@kafka.apache.org
>> >> >> >> >
>> >> >> >> >> > wrote:
>> >> >> >> >> >
>> >> >> >> >> >> Hi, Luke,
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks for the reply.
>> >> >> >> >> >>
>> >> >> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> >> >> >> specifically
>> >> >> >> >> >> designed for the unclean leader election support in Cluster
>> >> >> >> Mirroring
>> >> >> >> >> >> because the unclean leader election will break the
>> assumption
>> >> >> that
>> >> >> >> the
>> >> >> >> >> >> committed data won’t be truncated."
>> >> >> >> >> >> If the source cluster has an unclean leader election, the
>> >> >> >> destination
>> >> >> >> >> >> cluster will detect the inconsistency and truncate the log
>> at
>> >> the
>> >> >> >> >> >> destination leader. We can just
>> >> >> >> >> >> extend mirror.support.unclean.leader.election to cover this
>> >> case,
>> >> >> >> >> i.e., we
>> >> >> >> >> >> wait until all replicas' log end offsets are reduced to the
>> >> >> target
>> >> >> >> >> offset
>> >> >> >> >> >> before resuming mirroring. This avoids the inconsistency.
>> >> >> >> >> >>
>> >> >> >> >> >> If the destination cluster has an unclean leader election,
>> it
>> >> >> will
>> >> >> >> >> >> re-fetch
>> >> >> >> >> >> the same data from the source. There won't be any
>> >> inconsistency
>> >> >> >> either.
>> >> >> >> >> >>
>> >> >> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >> >> >> >>
>> >> >> >> >> >> JR51. Make sense. To truncate based on LME, we need to
>> issue
>> >> >> another
>> >> >> >> >> RPC
>> >> >> >> >> >> to
>> >> >> >> >> >> translate the LME to an offset. It would be useful to
>> include
>> >> >> that
>> >> >> >> in
>> >> >> >> >> the
>> >> >> >> >> >> workflow description.
>> >> >> >> >> >>
>> >> >> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit
>> >> strange
>> >> >> >> >> because
>> >> >> >> >> >> it mixes querying the mirror partition state with querying
>> the
>> >> >> >> mirror
>> >> >> >> >> >> partition leader. Another option is to use
>> >> >> ReadMirrorStatesRequest
>> >> >> >> to
>> >> >> >> >> >> query
>> >> >> >> >> >> the mirror partition state and a new request to query the
>> >> mirror
>> >> >> >> >> partition
>> >> >> >> >> >> leader. If we want the adminClient to query the right
>> >> >> >> >> >> ClusterMirrorCoordinator
>> >> >> >> >> >> directly, we can extend the FindCoordinatorRequest to
>> support
>> >> >> that.
>> >> >> >> >> >>
>> >> >> >> >> >> JR55. For __mirror_state topic, could we describe whether
>> it
>> >> is
>> >> >> >> >> compacted?
>> >> >> >> >> >>
>> >> >> >> >> >> Jun
>> >> >> >> >> >>
>> >> >> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <
>> showuon@gmail.com>
>> >> >> wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> > Hi Jun,
>> >> >> >> >> >> >
>> >> >> >> >> >> > Thanks for the comments.
>> >> >> >> >> >> > Answering them below:
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR31.3 It makes sense for the new partition leader to
>> >> query
>> >> >> the
>> >> >> >> >> >> current
>> >> >> >> >> >> > mirror state from the local or remote
>> MirrorMetadataManager
>> >> >> that
>> >> >> >> owns
>> >> >> >> >> >> the
>> >> >> >> >> >> > partition's state. However, the wording in the KIP is
>> >> >> confusing.
>> >> >> >> >> >> > MirrorMetadataManager
>> >> >> >> >> >> > mirror partition state cache should be populated only by
>> one
>> >> >> way,
>> >> >> >> >> which
>> >> >> >> >> >> is
>> >> >> >> >> >> > replaying __mirror_state partitions this broker leads. By
>> >> >> saying
>> >> >> >> >> "and by
>> >> >> >> >> >> > fetching state from remote coordinators via
>> ReadMirrorStates
>> >> >> >> RPCs.",
>> >> >> >> >> >> it's
>> >> >> >> >> >> > as if the mirror partition state cache on each broker
>> stores
>> >> >> the
>> >> >> >> >> state
>> >> >> >> >> >> for
>> >> >> >> >> >> > all mirror partitions.
>> >> >> >> >> >> >
>> >> >> >> >> >> > You’re right. KIP is updated to make it clear.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR36. I understand the intention now, but I still don't
>> >> >> think we
>> >> >> >> >> need
>> >> >> >> >> >> > MirrorLeaderEpoch. Let's
>> >> >> >> >> >> > assume the common case: no unclean leader election in the
>> >> >> source
>> >> >> >> >> cluster
>> >> >> >> >> >> > during mirroring. Mirroring only fetches the committed
>> data
>> >> >> from
>> >> >> >> the
>> >> >> >> >> >> > source, which never changes if obtained a second time. In
>> >> your
>> >> >> >> >> example
>> >> >> >> >> >> > image, follower broker 1 in the destination cluster can
>> >> never
>> >> >> >> obtain
>> >> >> >> >> the
>> >> >> >> >> >> > record B(0) because it's never committed in the source
>> and
>> >> >> thus is
>> >> >> >> >> never
>> >> >> >> >> >> > exposed to the destination leader. It's possible for
>> broker
>> >> 1
>> >> >> to
>> >> >> >> >> first
>> >> >> >> >> >> get
>> >> >> >> >> >> > D(1) which is truncated later when broker 0 follows a new
>> >> >> leader.
>> >> >> >> But
>> >> >> >> >> >> the
>> >> >> >> >> >> > new leader will refetch the same D(1) from the source
>> >> cluster.
>> >> >> So,
>> >> >> >> >> it's
>> >> >> >> >> >> ok
>> >> >> >> >> >> > for broker 1 to keep D(1).
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > Yes, you are correct. Because mirroring only fetches the
>> >> >> committed
>> >> >> >> >> data
>> >> >> >> >> >> > from the source, the MirrorLeaderEpoch is unnecessary for
>> >> >> normal
>> >> >> >> >> >> mirroring.
>> >> >> >> >> >> > Thanks for the clarification. So I think we can say, the
>> >> >> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the
>> unclean
>> >> >> >> leader
>> >> >> >> >> >> > election support in Cluster Mirroring because the unclean
>> >> >> leader
>> >> >> >> >> >> election
>> >> >> >> >> >> > will break the assumption that the committed data won’t
>> be
>> >> >> >> truncated.
>> >> >> >> >> >> >
>> >> >> >> >> >> > Given that, we could gate the MirrorLeaderEpoch field by
>> >> only
>> >> >> >> >> setting it
>> >> >> >> >> >> > in the Fetch request when
>> >> >> >> >> "mirror.support.unclean.leader.election=true"
>> >> >> >> >> >> > (false by default), and skip it entirely for normal
>> >> mirroring.
>> >> >> >> What
>> >> >> >> >> do
>> >> >> >> >> >> you
>> >> >> >> >> >> > think?
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR51. Sounds good. Why do we choose to store the last
>> >> mirror
>> >> >> >> epoch
>> >> >> >> >> >> > instead
>> >> >> >> >> >> > of a last mirror offset?
>> >> >> >> >> >> >
>> >> >> >> >> >> > Yes, originally we stored the “last mirror offset”
>> instead
>> >> of
>> >> >> >> “last
>> >> >> >> >> >> mirror
>> >> >> >> >> >> > epoch”. But later, we found it doesn’t work for unclean
>> >> leader
>> >> >> >> >> election
>> >> >> >> >> >> > cases. (yes, `unclean leader election` again :) ) For
>> >> example:
>> >> >> >> >> >> >
>> >> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> >> >> >
>> >> >> >> >> >> > Cluster B is mirroring from A. And A crashes and
>> failover to
>> >> >> B. B
>> >> >> >> >> stores
>> >> >> >> >> >> > the “last mirror offset” as 1.
>> >> >> >> >> >> > When cluster A recovers, somehow the unclean leader
>> election
>> >> >> >> happens,
>> >> >> >> >> >> log
>> >> >> >> >> >> > is empty in the new leader, and then some more leadership
>> >> >> changes
>> >> >> >> to
>> >> >> >> >> >> make
>> >> >> >> >> >> > the leader epoch become 10. Then it starts the reverse
>> >> mirror
>> >> >> >> from B.
>> >> >> >> >> >> The
>> >> >> >> >> >> > log looks like this:
>> >> >> >> >> >> >
>> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
>> >> >> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
>> >> >> >> >> >> > Offset 2:
>> >> pid_res
>> >> >> (LE:
>> >> >> >> >> 10)
>> >> >> >> >> >> >
>> >> >> >> >> >> > When A starts mirroring from B, log is truncated to last
>> >> mirror
>> >> >> >> >> offset:
>> >> >> >> >> >> 1
>> >> >> >> >> >> > (no-op), and then fetch from cluster B. The 1st fetch
>> >> request
>> >> >> is
>> >> >> >> like
>> >> >> >> >> >> this:
>> >> >> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10,
>> fetchOffset:
>> >> >> 2, …
>> >> >> >> }.
>> >> >> >> >> The
>> >> >> >> >> >> > divergingEpoch check in the cluster B queries the
>> endOffset
>> >> for
>> >> >> >> >> leader
>> >> >> >> >> >> > epoch 10 (in Partition#readRecords), and found the end
>> >> offset
>> >> >> for
>> >> >> >> LE
>> >> >> >> >> 10
>> >> >> >> >> >> is
>> >> >> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch
>> >> validation
>> >> >> >> >> passes
>> >> >> >> >> >> and
>> >> >> >> >> >> > returns offset 2 to cluster A. In short, relying on “last
>> >> >> mirror
>> >> >> >> >> offset”
>> >> >> >> >> >> > cannot resolve the log diverge issue in unclean leader
>> >> election
>> >> >> >> case.
>> >> >> >> >> >> >
>> >> >> >> >> >> > If instead, we truncate until “last mirror epoch”: 0, the
>> >> >> offset 0
>> >> >> >> >> and 1
>> >> >> >> >> >> > in cluster A will be truncated as expected and then log
>> >> >> converge
>> >> >> >> with
>> >> >> >> >> >> > cluster B correctly in the end.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > Also, about JR53.2:
>> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
>> >> >> brokers
>> >> >> >> and
>> >> >> >> >> >> > merges
>> >> >> >> >> >> > the responses client-side."
>> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
>> >> partition.
>> >> >> So
>> >> >> >> >> sending
>> >> >> >> >> >> > the request to every broker is wasteful. It's probably
>> >> better
>> >> >> to
>> >> >> >> send
>> >> >> >> >> >> the
>> >> >> >> >> >> > request to a single broker, which can then contact the
>> right
>> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >> >
>> >> >> >> >> >> > While working on this, I found we need to add one more
>> >> tagged
>> >> >> >> field
>> >> >> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema
>> >> because
>> >> >> >> >> >> otherwise
>> >> >> >> >> >> > the receiving broker doesn’t know if this is the request
>> >> from
>> >> >> >> client
>> >> >> >> >> >> and it
>> >> >> >> >> >> > needs to contact other brokers, or this is just a
>> forwarded
>> >> >> >> request.
>> >> >> >> >> The
>> >> >> >> >> >> > workflow is like this:
>> >> >> >> >> >> > 1. the adminClient sends the
>> DescribeClusterMirrorsRequest
>> >> to a
>> >> >> >> >> >> > leastLoaded broker, which doesn’t contain the "Forwarded"
>> >> >> field.
>> >> >> >> >> >> > 2. The broker receives the request and sends
>> >> >> >> >> >> DescribeClusterMirrorsRequest
>> >> >> >> >> >> > with "Forwarded=true" to the responsible brokers and
>> waits
>> >> for
>> >> >> the
>> >> >> >> >> >> response
>> >> >> >> >> >> > async.
>> >> >> >> >> >> > 3. When all responses are received, merge the results and
>> >> >> respond
>> >> >> >> to
>> >> >> >> >> the
>> >> >> >> >> >> > client.
>> >> >> >> >> >> >
>> >> >> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here because
>> it
>> >> >> >> doesn’t
>> >> >> >> >> >> > contain the lag info in the coordinator.
>> >> >> >> >> >> >
>> >> >> >> >> >> > The alternative solution is the adminClient sends
>> >> >> >> >> >> > DescribeClusterMirrorsRequest to the “responsible brokers
>> >> only”
>> >> >> >> (via
>> >> >> >> >> >> > describeTopics), instead of ALL brokers, and merge the
>> >> results
>> >> >> in
>> >> >> >> the
>> >> >> >> >> >> > client side. This solution doesn’t need to add a new
>> field
>> >> in
>> >> >> the
>> >> >> >> >> >> > DescribeClusterMirrorsRequest and still can avoid
>> >> unnecessary
>> >> >> >> >> requests
>> >> >> >> >> >> > sent.
>> >> >> >> >> >> >
>> >> >> >> >> >> > My preference is to keep the merge logic on the admin
>> client
>> >> >> side,
>> >> >> >> >> which
>> >> >> >> >> >> > saves the overhead in the broker side.
>> >> >> >> >> >> > What do you think?
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > Thank you,
>> >> >> >> >> >> > Luke
>> >> >> >> >> >> >
>> >> >> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
>> >> >> >> dev@kafka.apache.org
>> >> >> >> >> >
>> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >
>> >> >> >> >> >> >> Hi, Luke,
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> Thanks for the reply.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR31.3 It makes sense for the new partition leader to
>> query
>> >> >> the
>> >> >> >> >> current
>> >> >> >> >> >> >> mirror state from the local or remote
>> MirrorMetadataManager
>> >> >> that
>> >> >> >> >> owns
>> >> >> >> >> >> the
>> >> >> >> >> >> >> partition's state. However, the wording in the KIP is
>> >> >> confusing.
>> >> >> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> >> mirror partition state cache should be populated only by
>> >> one
>> >> >> way,
>> >> >> >> >> >> which is
>> >> >> >> >> >> >> replaying __mirror_state partitions this broker leads.
>> By
>> >> >> saying
>> >> >> >> >> "and
>> >> >> >> >> >> by
>> >> >> >> >> >> >> fetching state from remote coordinators via
>> >> ReadMirrorStates
>> >> >> >> RPCs.",
>> >> >> >> >> >> it's
>> >> >> >> >> >> >> as if the mirror partition state cache on each broker
>> >> stores
>> >> >> the
>> >> >> >> >> state
>> >> >> >> >> >> for
>> >> >> >> >> >> >> all mirror partitions.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR36. I understand the intention now, but I still don't
>> >> think
>> >> >> we
>> >> >> >> >> need
>> >> >> >> >> >> >> MirrorLeaderEpoch. Let's
>> >> >> >> >> >> >> assume the common case: no unclean leader election in
>> the
>> >> >> source
>> >> >> >> >> >> cluster
>> >> >> >> >> >> >> during mirroring. Mirroring only fetches the committed
>> data
>> >> >> from
>> >> >> >> the
>> >> >> >> >> >> >> source, which never changes if obtained a second time.
>> In
>> >> your
>> >> >> >> >> example
>> >> >> >> >> >> >> image, follower broker 1 in the destination cluster can
>> >> never
>> >> >> >> obtain
>> >> >> >> >> >> the
>> >> >> >> >> >> >> record B(0) because it's never committed in the source
>> and
>> >> >> thus
>> >> >> >> is
>> >> >> >> >> >> never
>> >> >> >> >> >> >> exposed to the destination leader. It's possible for
>> >> broker 1
>> >> >> to
>> >> >> >> >> first
>> >> >> >> >> >> get
>> >> >> >> >> >> >> D(1) which is truncated later when broker 0 follows a
>> new
>> >> >> leader.
>> >> >> >> >> But
>> >> >> >> >> >> the
>> >> >> >> >> >> >> new leader will refetch the same D(1) from the source
>> >> cluster.
>> >> >> >> So,
>> >> >> >> >> >> it's ok
>> >> >> >> >> >> >> for broker 1 to keep D(1).
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR51. Sounds good. Why do we choose to store the last
>> >> mirror
>> >> >> >> epoch
>> >> >> >> >> >> instead
>> >> >> >> >> >> >> of a last mirror offset?
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> Jun
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <
>> >> showuon@gmail.com>
>> >> >> >> wrote:
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> > Hi Jun,
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Thanks for the great comments again.
>> >> >> >> >> >> >> > Answering your question below.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror
>> partition
>> >> >> cache
>> >> >> >> >> lives
>> >> >> >> >> >> in
>> >> >> >> >> >> >> > MirrorMetadataManager. It is populated in two ways: by
>> >> >> >> replaying
>> >> >> >> >> >> >> > __mirror_state partitions this broker leads, and by
>> >> fetching
>> >> >> >> state
>> >> >> >> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> >> >> >> > Hmm, a given broker only caches the mirror partition
>> >> state
>> >> >> for
>> >> >> >> >> >> >> partitions
>> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted on
>> this
>> >> >> broker,
>> >> >> >> >> >> right?
>> >> >> >> >> >> >> Why
>> >> >> >> >> >> >> > does it need to fetch the state from other remote
>> >> >> coordinators?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > a given broker only caches the mirror partition
>> state
>> >> for
>> >> >> >> >> >> partitions
>> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted on
>> this
>> >> >> broker,
>> >> >> >> >> >> right?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Right.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > Why does it need to fetch the state from other
>> remote
>> >> >> >> >> coordinators?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Because the data partition leader needs to know the
>> >> current
>> >> >> >> mirror
>> >> >> >> >> >> state
>> >> >> >> >> >> >> > of the partitions it leads. On a leadership change,
>> the
>> >> >> >> metadata
>> >> >> >> >> >> delta
>> >> >> >> >> >> >> is
>> >> >> >> >> >> >> > published to the MirrorMetadataManager, which queries
>> the
>> >> >> >> current
>> >> >> >> >> >> mirror
>> >> >> >> >> >> >> > state for its led partitions from the coordinator
>> (local
>> >> or
>> >> >> >> >> remote),
>> >> >> >> >> >> and
>> >> >> >> >> >> >> > processes tasks based on that state. This is the
>> reason
>> >> why
>> >> >> it
>> >> >> >> >> >> fetches
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > state from other remote coordinator. This part of
>> cache
>> >> >> will be
>> >> >> >> >> >> cleared
>> >> >> >> >> >> >> > after the leadership change.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR36. I am still not quite sure why we need
>> >> >> >> MirrorLeaderEpoch in
>> >> >> >> >> >> the
>> >> >> >> >> >> >> > Fetch
>> >> >> >> >> >> >> > request. In the current Fetch request, in addition to
>> >> >> >> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> >> >> > it has another field LastFetchedEpoch, which captures
>> the
>> >> >> last
>> >> >> >> >> leader
>> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> > in the data and is used to resolve the inconsistency
>> >> between
>> >> >> >> the
>> >> >> >> >> >> >> follower
>> >> >> >> >> >> >> > and the leader. Does it achieve the same purpose as
>> >> >> >> >> >> MirrorLeaderEpoch?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to
>> do
>> >> the
>> >> >> >> >> >> validation
>> >> >> >> >> >> >> to
>> >> >> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any
>> fetched
>> >> >> >> batches
>> >> >> >> >> with
>> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> > higher than the LastFetchedEpoch, it means the
>> follower
>> >> can
>> >> >> >> never
>> >> >> >> >> >> accept
>> >> >> >> >> >> >> > new appended batches with higher epoch in the leader
>> >> node.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2
>> >> purposes in
>> >> >> >> the
>> >> >> >> >> >> fetch
>> >> >> >> >> >> >> > API:
>> >> >> >> >> >> >> > 1. The receiver (leader node) uses it to validate it
>> >> matches
>> >> >> >> the
>> >> >> >> >> >> leader
>> >> >> >> >> >> >> > epoch in the leader node
>> >> >> >> >> >> >> > 2. The sender (follower node) uses it to validate the
>> >> epoch
>> >> >> in
>> >> >> >> the
>> >> >> >> >> >> >> fetched
>> >> >> >> >> >> >> > batches are all <= this value.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still
>> serving
>> >> for
>> >> >> >> >> >> followers
>> >> >> >> >> >> >> in
>> >> >> >> >> >> >> > the destination cluster. For purpose (2), my
>> >> understanding
>> >> >> of
>> >> >> >> the
>> >> >> >> >> >> >> > validation for KAFKA-18723 is like: the follower knows
>> >> the
>> >> >> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader
>> node
>> >> >> >> should
>> >> >> >> >> >> >> contain in
>> >> >> >> >> >> >> > the log batches when fetch request is sent. Any log
>> batch
>> >> >> >> beyond
>> >> >> >> >> this
>> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> > in the fetch response means the follower’s metadata is
>> >> >> stale.
>> >> >> >> The
>> >> >> >> >> >> >> batches
>> >> >> >> >> >> >> > need to be rejected and follower needs to update
>> metadata
>> >> >> for
>> >> >> >> next
>> >> >> >> >> >> >> fetch.
>> >> >> >> >> >> >> > In the Cluster Mirroring’s world, the
>> MirrorLeaderEpoch
>> >> >> serves
>> >> >> >> >> this
>> >> >> >> >> >> >> purpose
>> >> >> >> >> >> >> > to let the followers know the highest epoch in the
>> leader
>> >> >> node
>> >> >> >> >> log in
>> >> >> >> >> >> >> each
>> >> >> >> >> >> >> > fetch response. Once the info is stale in the follower
>> >> >> side, it
>> >> >> >> >> >> updates
>> >> >> >> >> >> >> it
>> >> >> >> >> >> >> > from fetch response, and send next fetch request with
>> >> >> correct
>> >> >> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is
>> >> >> possible
>> >> >> >> >> like
>> >> >> >> >> >> >> > KAFKA-18723.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > I tried not to add new field in Fetch API, but this is
>> >> the
>> >> >> best
>> >> >> >> >> >> >> solution I
>> >> >> >> >> >> >> > can think of.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR48. "The NumPartitions field is needed so the
>> >> controller
>> >> >> >> can
>> >> >> >> >> >> create
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > destination topic if it does not already exist. When a
>> >> >> topic is
>> >> >> >> >> being
>> >> >> >> >> >> >> > mirrored for the first time, the destination cluster
>> has
>> >> no
>> >> >> >> >> knowledge
>> >> >> >> >> >> >> > of the source topic's partition count, so the request
>> >> must
>> >> >> >> carry
>> >> >> >> >> it."
>> >> >> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is
>> started,
>> >> the
>> >> >> >> source
>> >> >> >> >> >> topic
>> >> >> >> >> >> >> > may not exist yet. When the source topic is created
>> >> later,
>> >> >> we
>> >> >> >> can
>> >> >> >> >> >> start
>> >> >> >> >> >> >> > mirroring without requiring the user to provide the
>> >> number
>> >> >> of
>> >> >> >> >> >> >> partitions.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > When the source topic is created later, we can start
>> >> >> >> mirroring
>> >> >> >> >> >> without
>> >> >> >> >> >> >> > requiring the user to provide the number of
>> partitions.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > This is because this happens in periodical sync up in
>> the
>> >> >> >> >> destination
>> >> >> >> >> >> >> > cluster, and the numPartitions and other topic info in
>> >> the
>> >> >> >> source
>> >> >> >> >> >> >> cluster
>> >> >> >> >> >> >> > is already retrieved during the sync up process.
>> Also, if
>> >> >> users
>> >> >> >> >> are
>> >> >> >> >> >> >> > creating mirror topics via admin API, this info can
>> also
>> >> be
>> >> >> >> >> retrieved
>> >> >> >> >> >> >> > inside the admin client if not provided.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly need
>> the
>> >> >> last
>> >> >> >> >> mirror
>> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> > for truncation when starting a mirror. When a mirror
>> is
>> >> >> >> stopped at
>> >> >> >> >> >> the
>> >> >> >> >> >> >> > source, we already bump up the leader epoch and write
>> >> some
>> >> >> >> >> additional
>> >> >> >> >> >> >> > records for cleanup. It seems that we can just use the
>> >> >> >> >> DivergingEpoch
>> >> >> >> >> >> >> > returned in the initial mirror Fetch request to detect
>> >> the
>> >> >> >> >> diverging
>> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> > and use that for truncation at the mirror destination.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > That’s a good question. I thought about this before
>> but
>> >> >> >> >> >> unfortunately it
>> >> >> >> >> >> >> > doesn’t work. The reason we need the last mirror epoch
>> >> >> record
>> >> >> >> is
>> >> >> >> >> >> because
>> >> >> >> >> >> >> > the destination cluster might not completely sync with
>> >> the
>> >> >> >> source
>> >> >> >> >> >> >> cluster
>> >> >> >> >> >> >> > before STOPPED. And this un-mirrored data might
>> confuse
>> >> the
>> >> >> >> >> >> replication
>> >> >> >> >> >> >> > protocol when reverse mirror. For example,
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
>> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE:
>> 0)
>> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE:
>> 0)
>> >> >> >> >> >> >> > Offset 2: from A (LE: 10)
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > While cluster B mirrors until offset 1, the cluster A
>> >> >> crashes
>> >> >> >> and
>> >> >> >> >> >> >> failover
>> >> >> >> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
>> >> >> >> >> MIRROR_PID_RESET
>> >> >> >> >> >> >> with
>> >> >> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from
>> >> >> cluster B,
>> >> >> >> >> the
>> >> >> >> >> >> log
>> >> >> >> >> >> >> > will be like this:
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> >> >> > foo-0 leader log: foo-0 leader log:
>> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE:
>> 0)
>> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE:
>> 0)
>> >> >> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE:
>> >> 10)
>> >> >> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE:
>> 10)
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > As you can see, cluster A (new destination cluster)
>> >> doesn’t
>> >> >> >> know
>> >> >> >> >> the
>> >> >> >> >> >> >> > offset 2 is not belonging to the cluster B, which
>> causes
>> >> >> >> >> inconsistent
>> >> >> >> >> >> >> logs.
>> >> >> >> >> >> >> > So like what we described in the KIP, the last mirror
>> >> epoch
>> >> >> is
>> >> >> >> the
>> >> >> >> >> >> >> > synchronization point between source and destination.
>> Any
>> >> >> data
>> >> >> >> >> beyond
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > point is treated as unknown to the source cluster and
>> >> >> should be
>> >> >> >> >> >> >> truncated.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR52. Stop Mirror:
>> >> >> >> >> >> >> > > JR52.1 "3. If patterns are provided, the controller
>> >> >> removes
>> >> >> >> >> >> matching
>> >> >> >> >> >> >> > entries from mirror.topics.include or adds them to
>> >> >> >> >> >> >> mirror.topics.exclude on
>> >> >> >> >> >> >> > the CLUSTER_MIRROR resource."
>> >> >> >> >> >> >> > Why is it 'or' and not 'and'?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> >> > discovers new source topics matching the persisted
>> >> >> >> include/exclude
>> >> >> >> >> >> >> patterns
>> >> >> >> >> >> >> > and repeats steps 3 through 7 for each."
>> >> >> >> >> >> >> > The broker will send a create topic request to the
>> >> >> controller.
>> >> >> >> How
>> >> >> >> >> >> does
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > controller associate it with the mirror name to
>> create a
>> >> >> >> >> >> >> > `MirrorTopicStateChangeRecord`?
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > No, we don’t send a create topic request to the
>> >> controller
>> >> >> >> here.
>> >> >> >> >> >> >> Actually
>> >> >> >> >> >> >> > we send a StartMirrorTopics request to the controller,
>> >> which
>> >> >> >> >> contains
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > mirror name and topic metadata like NumPartitions,
>> >> topicId…
>> >> >> >> etc.
>> >> >> >> >> So
>> >> >> >> >> >> the
>> >> >> >> >> >> >> > controller can create the non-existed topic as
>> explained
>> >> in
>> >> >> >> JR48,
>> >> >> >> >> and
>> >> >> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR53. Describe Mirrors:
>> >> >> >> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors
>> >> request
>> >> >> >> with
>> >> >> >> >> >> >> optional
>> >> >> >> >> >> >> > mirror names (empty means all mirrors)."
>> >> >> >> >> >> >> > In metadata request, we use a null arrary to represent
>> >> all
>> >> >> >> >> topics. It
>> >> >> >> >> >> >> would
>> >> >> >> >> >> >> > be useful to be consistent here.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > OK, KIP updated.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to
>> all
>> >> >> >> brokers
>> >> >> >> >> and
>> >> >> >> >> >> >> > merges
>> >> >> >> >> >> >> > the responses client-side."
>> >> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
>> >> >> partition.
>> >> >> >> So
>> >> >> >> >> >> sending
>> >> >> >> >> >> >> > the request to every broker is wasteful. It's probably
>> >> >> better
>> >> >> >> to
>> >> >> >> >> send
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> > request to a single broker, which can then contact the
>> >> right
>> >> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Agree, KIP updated.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are
>> >> >> >> MIRRORING,
>> >> >> >> >> >> >> PAUSED,
>> >> >> >> >> >> >> > STOPPED the only validate states? The state table has
>> >> other
>> >> >> >> states
>> >> >> >> >> >> too.
>> >> >> >> >> >> >> > Ditto for MirrorPartitionState.
>> >> >> >> >> >> >> > The DesiredState field in MirrorTopicStateChangeRecord
>> >> is to
>> >> >> >> >> define
>> >> >> >> >> >> the
>> >> >> >> >> >> >> > target state the user wants. And the desired state
>> users
>> >> can
>> >> >> >> >> assign
>> >> >> >> >> >> is
>> >> >> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a
>> desired
>> >> >> state
>> >> >> >> as
>> >> >> >> >> >> >> STOPPING
>> >> >> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
>> >> >> intermediate
>> >> >> >> >> >> states.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > About MirrorPartitionState record, it’ll record any
>> >> state in
>> >> >> >> the
>> >> >> >> >> >> state
>> >> >> >> >> >> >> > table that the current partition is. This is the real
>> >> state
>> >> >> for
>> >> >> >> >> each
>> >> >> >> >> >> >> > partition. It’s like in the kubernetes world, when we
>> >> change
>> >> >> >> the
>> >> >> >> >> >> replica
>> >> >> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired
>> >> state), it
>> >> >> >> won’t
>> >> >> >> >> >> >> increase
>> >> >> >> >> >> >> > to 5 immediately. The current replica
>> >> (MirrorPartitionState
>> >> >> >> >> record)
>> >> >> >> >> >> is
>> >> >> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > Thank you,
>> >> >> >> >> >> >> > Luke
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
>> >> >> >> >> dev@kafka.apache.org
>> >> >> >> >> >> >
>> >> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> Hi, Fede,
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> Thanks for the reply. A few more comments.
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror partition
>> >> cache
>> >> >> >> >> lives in
>> >> >> >> >> >> >> >> MirrorMetadataManager. It is populated in two ways:
>> by
>> >> >> >> replaying
>> >> >> >> >> >> >> >> __mirror_state partitions this broker leads, and by
>> >> >> fetching
>> >> >> >> >> state
>> >> >> >> >> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> >> >> >> >> Hmm, a given broker only caches the mirror partition
>> >> state
>> >> >> for
>> >> >> >> >> >> >> partitions
>> >> >> >> >> >> >> >> hashed into the __mirror_state partition hosted on
>> this
>> >> >> >> broker,
>> >> >> >> >> >> right?
>> >> >> >> >> >> >> Why
>> >> >> >> >> >> >> >> does it need to fetch the state from other remote
>> >> >> >> coordinators?
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR36. I am still not quite sure why we need
>> >> >> MirrorLeaderEpoch
>> >> >> >> in
>> >> >> >> >> the
>> >> >> >> >> >> >> Fetch
>> >> >> >> >> >> >> >> request. In the current Fetch request, in addition to
>> >> >> >> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> >> >> >> it has another field LastFetchedEpoch, which captures
>> >> the
>> >> >> last
>> >> >> >> >> >> leader
>> >> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> in the data and is used to resolve the inconsistency
>> >> >> between
>> >> >> >> the
>> >> >> >> >> >> >> follower
>> >> >> >> >> >> >> >> and the leader. Does it achieve the same purpose as
>> >> >> >> >> >> MirrorLeaderEpoch?
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR48. "The NumPartitions field is needed so the
>> >> controller
>> >> >> can
>> >> >> >> >> >> create
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> destination topic if it does not already exist. When
>> a
>> >> >> topic
>> >> >> >> is
>> >> >> >> >> >> being
>> >> >> >> >> >> >> >> mirrored for the first time, the destination cluster
>> >> has no
>> >> >> >> >> >> knowledge
>> >> >> >> >> >> >> >> of the source topic's partition count, so the request
>> >> must
>> >> >> >> carry
>> >> >> >> >> >> it."
>> >> >> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is
>> started,
>> >> the
>> >> >> >> >> source
>> >> >> >> >> >> >> topic
>> >> >> >> >> >> >> >> may not exist yet. When the source topic is created
>> >> later,
>> >> >> we
>> >> >> >> can
>> >> >> >> >> >> start
>> >> >> >> >> >> >> >> mirroring without requiring the user to provide the
>> >> number
>> >> >> of
>> >> >> >> >> >> >> partitions.
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need
>> the
>> >> >> last
>> >> >> >> >> mirror
>> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> for truncation when starting a mirror. When a mirror
>> is
>> >> >> >> stopped
>> >> >> >> >> at
>> >> >> >> >> >> the
>> >> >> >> >> >> >> >> source, we already bump up the leader epoch and write
>> >> some
>> >> >> >> >> >> additional
>> >> >> >> >> >> >> >> records for cleanup. It seems that we can just use
>> the
>> >> >> >> >> >> DivergingEpoch
>> >> >> >> >> >> >> >> returned in the initial mirror Fetch request to
>> detect
>> >> the
>> >> >> >> >> diverging
>> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> >> and use that for truncation at the mirror
>> destination.
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR52. Stop Mirror:
>> >> >> >> >> >> >> >> JR52.1 "3. If patterns are provided, the controller
>> >> removes
>> >> >> >> >> matching
>> >> >> >> >> >> >> >> entries from mirror.topics.include or adds them to
>> >> >> >> >> >> >> mirror.topics.exclude
>> >> >> >> >> >> >> >> on
>> >> >> >> >> >> >> >> the CLUSTER_MIRROR resource."
>> >> >> >> >> >> >> >> Why is it 'or' and not 'and'?
>> >> >> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> >> >> discovers new source topics matching the persisted
>> >> >> >> >> include/exclude
>> >> >> >> >> >> >> >> patterns
>> >> >> >> >> >> >> >> and repeats steps 3 through 7 for each."
>> >> >> >> >> >> >> >> The broker will send a create topic request to the
>> >> >> controller.
>> >> >> >> >> How
>> >> >> >> >> >> does
>> >> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> controller associate it with the mirror name to
>> create a
>> >> >> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR53. Describe Mirrors:
>> >> >> >> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors
>> >> request
>> >> >> >> with
>> >> >> >> >> >> >> optional
>> >> >> >> >> >> >> >> mirror names (empty means all mirrors)."
>> >> >> >> >> >> >> >> In metadata request, we use a null arrary to
>> represent
>> >> all
>> >> >> >> >> topics.
>> >> >> >> >> >> It
>> >> >> >> >> >> >> >> would
>> >> >> >> >> >> >> >> be useful to be consistent here.
>> >> >> >> >> >> >> >> JR53.2 "2. The admin client fans out the request to
>> all
>> >> >> >> brokers
>> >> >> >> >> and
>> >> >> >> >> >> >> merges
>> >> >> >> >> >> >> >> the responses client-side."
>> >> >> >> >> >> >> >> This is not ideal. Not every broker hosts a mirror
>> >> >> partition.
>> >> >> >> So
>> >> >> >> >> >> >> sending
>> >> >> >> >> >> >> >> the request to every broker is wasteful. It's
>> probably
>> >> >> better
>> >> >> >> to
>> >> >> >> >> >> send
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> >> request to a single broker, which can then contact
>> the
>> >> >> right
>> >> >> >> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are
>> >> >> >> MIRRORING,
>> >> >> >> >> >> PAUSED,
>> >> >> >> >> >> >> >> STOPPED the only validate states? The state table has
>> >> other
>> >> >> >> >> states
>> >> >> >> >> >> too.
>> >> >> >> >> >> >> >> Ditto for MirrorPartitionState.
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> Jun
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
>> >> >> >> >> >> fedevaleri@gmail.com>
>> >> >> >> >> >> >> >> wrote:
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> > Hi Andrew, yes, validation is called at the start
>> of
>> >> >> >> >> createTopic.
>> >> >> >> >> >> If
>> >> >> >> >> >> >> >> > the topic ID is invalid, the request fails with
>> >> >> >> >> INVALID_REQUEST.
>> >> >> >> >> >> If
>> >> >> >> >> >> >> >> > the topic ID is already used by a different topic
>> >> name,
>> >> >> it
>> >> >> >> >> fails
>> >> >> >> >> >> with
>> >> >> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the
>> >> individual
>> >> >> >> topic
>> >> >> >> >> in
>> >> >> >> >> >> the
>> >> >> >> >> >> >> >> > batch fails; other topics in the same CreateTopics
>> >> >> request
>> >> >> >> >> >> continue
>> >> >> >> >> >> >> >> > processing normally. Added this information to the
>> >> KIP.
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
>> >> >> >> >> >> >> >> schofielaj@apache.org>
>> >> >> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> >> > > Hi Fede,
>> >> >> >> >> >> >> >> > > One more small question.
>> >> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> >> > > AS37: If the controller fails validation for the
>> >> >> >> >> >> MirrorInfo.TopicId
>> >> >> >> >> >> >> >> > added to the CreateTopic request, what does it do?
>> I
>> >> >> suspect
>> >> >> >> >> that
>> >> >> >> >> >> it
>> >> >> >> >> >> >> >> fails
>> >> >> >> >> >> >> >> > the request with a particular error code.
>> >> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> >> > > Thanks,
>> >> >> >> >> >> >> >> > > Andrew
>> >> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> >> >> >> >> >> >> >> > > > Hi all,
>> >> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> >> > > > We updated the KIP to include Coordinator
>> Runtime
>> >> >> >> >> >> configurations
>> >> >> >> >> >> >> and
>> >> >> >> >> >> >> >> > > > the coordinator state transition validation
>> >> >> paragraph.
>> >> >> >> The
>> >> >> >> >> >> latter
>> >> >> >> >> >> >> >> > > > covers the per-partition epoch fencing
>> mechanism
>> >> >> (leader
>> >> >> >> >> epoch
>> >> >> >> >> >> >> and
>> >> >> >> >> >> >> >> > > > state epoch) that complements the existing
>> >> per-topic
>> >> >> >> >> >> StateOffset
>> >> >> >> >> >> >> >> > > > fencing for lifecycle operations.
>> >> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> >> > > > Thanks,
>> >> >> >> >> >> >> >> > > > Fede
>> >> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >>
>> >> >> >> >> >> >
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for the good comments.
Answering them below.
> JR80. In that case, it seems that there is no need for
AdminClient.startMirrorTopics() to take topics. The adminClient only uses
it to register the included topics, which can be passed in as
includePatterns (specific names or regex). The broker will use
StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
through the AdminClient.
Make sense. So when users invoke admin.startMirrorTopics, they only need to
set the topic patterns. And the receiving broker will resolve the patterns
into the actual topics using the listTopic response from the source
cluster, and get the topic id, numPartitions, … via the describeTopics
request from the source cluster if the topic is non-existent in the
destination cluster. KIP is updated.
> JR82. If “IncludeMirrorOffset” is disabled,
DescribeClusterMirrorsResponse.SourceOffset is still returned in the
response and what is its value in this case?
It will be -1 because when the `IncludeMirrorOffset` is disabled, we won’t
query the mirror partition leader. I’ve updated the description to make it
clear. Thanks.
> JR88. Why is partition-load-time a Meter instead of a Gauge?
You’re right. It is not the Meter type. It’s the SampledStat state like
“batch-size-avg” and “batch-size-max” in producer. I’ve updated the KIP.
> JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
You’re right. It’s the rate type. KIP is updated
> JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
Meter.
Yes, it makes more sense to make it as Meter type. I’ve updated these
error-related metrics to Meter type:
1. MetadataRefreshError
2. TopicConfigMetadataSyncError
3. ConsumerGroupOffsetSyncError
4. ShareGroupOffsetSyncError
5. AclSyncError
Thank you,
Luke
On Sat, Aug 15, 2026 at 8:10 AM Jun Rao via dev <de...@kafka.apache.org>
wrote:
> Hi, Luke,
>
> Thanks for the reply.
>
> JR80. In that case, it seems that there is no need for
> AdminClient.startMirrorTopics() to take topics. The adminClient only uses
> it to register the included topics, which can be passed in as
> includePatterns (specific names or regex). The broker will use
> StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
> through the AdminClient.
>
> JR82. If “IncludeMirrorOffset” is disabled,
> DescribeClusterMirrorsResponse.SourceOffset is still returned in the
> response and what is its value in this case?
>
> JR88. Why is partition-load-time a Meter instead of a Gauge?
>
> JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
>
> JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
> Meter.
>
> Jun
>
> On Fri, Aug 14, 2026 at 4:32 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the good comments,
> > Answering them below.
> >
> > > JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
> > supports regex in the ClusterMirrorComamnd, but we don’t need to add the
> > “patterns”."
> > So, in this case, we use topics to support both specific topics and
> regex.
> > However, startMirrorTopicsOptions uses topics for specific topics and
> > includePatterns for regex. Could we make them consistent?
> >
> >
> > The purpose of the “includePatterns” field is for the
> > “mirror.topics.include” update, so that if there is any newly created
> topic
> > in the source cluster that matches the regex, we can create it. And the
> > “Topics” field in the StartMirrorTopicsRequest is to specify the topicId
> > and numPartitions in the source cluster. So I think “includePatterns” and
> > “Topics” both are necessary in the StartMirrorTopicsRequest. On the other
> > hand, in PauseMirrorTopics, because it won’t change the
> > “mirror.topics.include”/”mirror.topics.exclude” configs, we don’t need to
> > provide the regex in the request.
> >
> >
> >
> > > JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also -1
> > if
> > it's not included in the request?
> >
> > If “IncludeMirrorOffset” is disabled, we won’t query the mirror partition
> > leader, so we can’t get the sourceOffset.
> >
> > I assume you want to ask `if it’s still -1 if not actively mirroring,
> > while “IncludeMirrorOffset” is enabled`. That’s a good question. I think
> > our purpose of this command is to show the lag to users. So if the state
> is
> > not MIRRORING (ex: STOPPED/PAUSED), we think it’s not helpful to users to
> > show only the sourceOffset. Of course users can use
> `kafka-get-offsets.sh`
> > to get the offset if they really want to know.
> >
> >
> > > JR83. Start Mirror Topics:
> > > JR83.1 Should the controller verify the expected state for the topic?
> >
> > As replied in JR84, the desired state validation should happen in the
> > broker. And the controller validates the StateOffset. I’ve updated the
> > workflow to make it clear.
> >
> > > JR83.2 "d. truncates the log accordingly,"
> > Could you add a description of the truncation offset?
> >
> > OK, added one more sentence:
> > d. based on the last mirror epoch response in (b), the broker translates
> > the epoch into the offset by querying the leaderEpochCache
> > (endOffsetForEpoch), and truncates the log accordingly, and waits for all
> > ISR members…
> >
> >
> > > JR84. Stop Mirror Topics: "The controller verifies that the desired
> state
> > of all topics are in MIRRORING or PAUSED state."
> > Should the broker verfy the topic level state? I thought the controller
> > will use StateOffset in the request to make sure the validation on the
> > broker is stil valid. If the controller performs the verification, is
> > StateOffset still useful?
> >
> > Oh, you’re right. We’re doing duplicated work here. Yes, the broker
> > already verifies the “desired state” of all topics are in MIRRORING or
> > PAUSED state, as well as the “actual state” is in MIRRORING or PAUSED
> > state. If all validation passes, we’ll forward it to the controller. So
> the
> > controller only needs to validate the StateOffset to make sure the
> > validation on the broker is still valid. So, yes, we don’t need the
> desired
> > state validation on the controller side. I’ve updated the KIP to make it
> > clear.
> >
> > One more thing to add is that, I’ve thought about not adding StateOffset,
> > and relying on the desired state validation to make sure we’re safe to do
> > the deletion or other operations. The problem is that, even if the broker
> > sees the desired state for topic A is STOPPED, and then the controller
> also
> > sees the desired state is STOPPED, it still cannot make sure the
> > controller’s STOPPED state equals the broker’s STOPPED state. For
> example:
> > 1. Broker sees STOPPED, then forward to controller
> > 2. User changes the desired state to MIRRORING
> > 3. User changes the desired state to STOPPED
> > 4. Controller sees the desired state is STOPPED
> > So I think StateOffset is required.
> >
> > Thanks for catching it BTW.
> >
> >
> > > JR85. There are still a few references to includeMirrorLags. They
> should
> > be
> > IncludeMirrorOffset.
> >
> > Sorry, just updated.
> >
> >
> > > JR86. Metrics:
> > > JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
> >
> > KIP is updated. I also group the metrics into new added metrics and other
> > inherited metrics.
> >
> > > JR86.2 Could you describe a bit more what MinFetchRate represents?
> >
> > It means the lowest fetch request rate among all of this mirror fetcher
> > manager's fetcher threads. It inherits from the AbstractFetcherManager’s
> > metric.
> >
> >
> > > JR87. Version Compatibility: "For migration or DR use cases where
> > failback
> > is not required, this proposal supports source brokers up to version 2.1
> > included"
> > This KIP needs topicId and KIP-516 was introduced later than 2.1.
> > Yes, when mirroring from a cluster without topic id support, we will
> > assign a random topic ID in the destination cluster and then fetch
> (without
> > topic id in the fetch request using the old version of Fetch API) as
> usual.
> > Of course this is for the one-way mirroring use case. It can’t start the
> > reverse mirror(failback) under this setup. You can check the
> “Compatibility
> > Matrix” section for more detail.
> >
> >
> > Thank you,
> > Luke
> >
> > On Fri, Aug 14, 2026 at 7:09 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Luke,
> >>
> >> Thanks for the reply.
> >>
> >> JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
> >> supports regex in the ClusterMirrorComamnd, but we don’t need to add the
> >> “patterns”."
> >> So, in this case, we use topics to support both specific topics and
> regex.
> >> However, startMirrorTopicsOptions uses topics for specific topics and
> >> includePatterns for regex. Could we make them consistent?
> >>
> >> JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also -1
> if
> >> it's not included in the request?
> >>
> >> JR83. Start Mirror Topics:
> >> JR83.1 Should the controller verify the expected state for the topic?
> >> JR83.2 "d. truncates the log accordingly,"
> >> Could you add a description of the truncation offset?
> >>
> >> JR84. Stop Mirror Topics: "The controller verifies that the desired
> state
> >> of all topics are in MIRRORING or PAUSED state."
> >> Should the broker verfy the topic level state? I thought the controller
> >> will use StateOffset in the request to make sure the validation on the
> >> broker is stil valid. If the controller performs the verification, is
> >> StateOffset still useful?
> >>
> >> JR85. There are still a few references to includeMirrorLags. They should
> >> be
> >> IncludeMirrorOffset.
> >>
> >> JR86. Metrics:
> >> JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
> >> JR86.2 Could you describe a bit more what MinFetchRate represents?
> >>
> >> JR87. Version Compatibility: "For migration or DR use cases where
> failback
> >> is not required, this proposal supports source brokers up to version 2.1
> >> included"
> >> This KIP needs topicId and KIP-516 was introduced later than 2.1.
> >>
> >> Jun
> >>
> >>
> >> On Thu, Aug 13, 2026 at 5:14 AM Luke Chen <sh...@gmail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thanks for the good comments.
> >> > Answering them below.
> >> >
> >> > > JR59.1 Could you add the RPC to the workflow? The broker first needs
> >> to
> >> > issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
> >> > mirror name and then issue DescribeClusterMirrors to get the LME.
> >> >
> >> > Yes, you’re right. The workflow is updated to make them clear.
> >> >
> >> > > JR71. mirror.support.unclean.leader.election : The name is
> confusing.
> >> > It's
> >> > also used in LOG_TRUNCATION, which is not related to unclean leader
> >> > election. Could we use a better name?
> >> >
> >> > I personally think the name should be fine because even though we use
> in
> >> > “LOG_TRUNCATION” state, the config is to allow us to know if users
> want
> >> to
> >> > support unclean leader election, if so, we wait for all replicas to
> >> catch
> >> > up. If we want to explicitly show the intention of the config, how
> about
> >> > “mirror.full.replica.convergence.required”? I’m open to other
> >> suggestions.
> >> >
> >> >
> >> > > JR72. DescribeClusterMirrors
> >> > > JR72.1 A user may specify just the topic name when describing a
> >> mirror.
> >> > Should we include the topic name in the request?
> >> >
> >> > Hmm.. we didn’t think about the use case. But yes, if users only care
> >> > about the state of a topic, they don’t need the info for other topics.
> >> I’ve
> >> > combined with JR72.2, added “TopicPartitions” filed into
> >> > DescribeClusterMirrorsRequest, and by default is null, which means all
> >> > topic partitions under the requested mirrors will be returned. If set,
> >> > we’ll query the mirrors and return the topic partitions that users
> >> > expected. KIP is updated.
> >> >
> >> >
> >> > > JR72.2 LastMirrorEpochLookups is a bit weird since the response
> >> returns
> >> > either mirror state or mirror lag and LastMirrorEpochLookups doesn't
> >> match
> >> > either.
> >> >
> >> > Yes, after the recent refactor, I think the LastMirrorEpochLookups can
> >> be
> >> > merged into one level up, combining with the topic name filter in
> above
> >> > comment, which will allow users to query down to the partition level.
> >> The
> >> > LastMirrorEpochLookups will query along with normal mirror state by
> the
> >> > readMirrorState from the coordinator. The difference is that after
> >> > receiving the response, it will filter out the mirrors that don’t
> match
> >> the
> >> > requested sourceClusterId, and then add the lastMirrorEpoch into
> >> > DescribeClusterMirrorResponse. KIP is updated.
> >> >
> >> >
> >> > > JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >
> >> > Right. Added.
> >> >
> >> >
> >> > > JR72.4 "includeMirrorLags: source offset, destination offset, lag"
> >> > DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
> >> > should be omitted.
> >> >
> >> > Sorry, I removed the “SourceOffset” and “DestinationOffset” fields
> from
> >> > DescribeClusterMirrorsResponse because I added them into
> >> ReadMirrorOffsets
> >> > (ex-ReadMirrorLags) API. But I forgot we still need them in the
> >> > DescribeClusterMirrorsResponse so that the adminClient can receive the
> >> > results. KIP is updated.
> >> >
> >> >
> >> > > JR73. MirrorPartitionState record, LastMirrorEpochs record and
> >> > MirrorPartitionStateRequest: The values for type and name are
> incorrect.
> >> >
> >> > Ah, that’s my bad. Fixed.
> >> >
> >> > > JR74. ListClusterMirrorsRequest
> >> > > JR74.1 When is DesiredMirrorStateFilter used? There is no option to
> >> > specify
> >> > the desired state in the mirror-topic command.
> >> >
> >> > Yes, referred to kafka-consumer-groups.sh, I added the `--state`
> option
> >> to
> >> > allow users to set the state when listMirror.
> >> >
> >> > > JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
> >> >
> >> > Yes, KIP updated.
> >> >
> >> > > JR75. ReadMirrorLags
> >> > > JR75.1 Should it be named ReadMirrorOffsets since the response
> doesn't
> >> > include lag?
> >> >
> >> > Indeed. Updated.
> >> >
> >> > > JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >
> >> > Yes, added
> >> >
> >> > > JR75.3 Some places refer to this as ReadClusterMirrorLags. We need
> to
> >> be
> >> > consistent.
> >> >
> >> > Thanks for pointing it out. KIP is updated.
> >> >
> >> > > JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR
> too?
> >> >
> >> > Yes, we should validate the mirror name before writing the data. KIP
> is
> >> > updated.
> >> >
> >> > > JR77. Delete Mirror: "3. The controller verifies that no topics are
> in
> >> > MIRRORING or PAUSED state."
> >> > Is this necessary since step 2 has verified all partitions are in
> >> STOPPED
> >> > state, which can only happen if the topics are in STOPPED state.
> >> >
> >> > It’s true, but there could be some delay between “actual state” and
> >> > “desired state”. In step (2), we verify all partitions’ “actual state”
> >> are
> >> > in STOPPED state. But the current desired state could be MIRRORING,
> it’s
> >> > just that the mirror partition leaders haven’t got the metadata update
> >> yet.
> >> > So, the controller still needs to verify the desired state is not in
> >> > MIRRORING or PAUSED. I found we didn’t specify that the controller is
> >> > actually verifying the “desired state”, not “actual state”. And this
> >> > controller’s desired state validation also applies in
> >> >
> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
> >> > I’ve updated the KIP to make it clear.
> >> >
> >> > > JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
> >> > bumped,
> >> > ABORT markers are appended for all ongoing transactions, last mirror
> >> epochs
> >> > are persisted, and a MIRROR_PID_RESET record appended."
> >> > This describes the state for STOPPED, not STOPPING.
> >> >
> >> > No, these are the tasks during STOPPING state, like what we described
> in
> >> > “Stop Mirror Topics” workflow. If we do these in STOPPED state, we
> can’t
> >> > safely move the readonly topic into writable because not all required
> >> tasks
> >> > are done. I think I know why it confuses you, it’s because all the
> tasks
> >> > are in the completed state. I’ve rephrased it to make it clear.
> >> >
> >> >
> >> > > JR79. RPCs like DescribeClusterMirrors can take time. Should we add
> a
> >> > timeout field?
> >> >
> >> > Make sense. Added into `DescribeClusterMirrors` request, as well as
> >> >
> >>
> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
> >> > because we will query the partition states from coordinator for state
> >> > validation in these APIs.
> >> >
> >> >
> >> > > JR80. PauseMirrorTopicsOptions: Should it support patterns? The
> >> command
> >> > line seems to suppor it.
> >> >
> >> > Sorry, it’s a little confusing here. The “pattern” in
> >> > startMirrorTopicsOptions or stopMirrorTopicsOptions is to update the
> >> > “mirror.topics.include” and “mirror.topics.exclude” configuration for
> >> topic
> >> > pattern discovery periodically. For the PauseMirrorTopics and
> >> > ResumeMirrorTopics in the clusterMirrorCommand, users can provide
> regex
> >> in
> >> > the “--topics” option, and the clusterMirrorCommand will listTopic in
> >> the
> >> > destination cluster and match with the patterns users provided, and
> then
> >> > feeds the matched topics into the PauseMirrorTopics/ResumeMirrorTopics
> >> > requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics
> >> supports
> >> > regex in the ClusterMirrorComamnd, but we don’t need to add the
> >> “patterns”.
> >> >
> >> >
> >> > > JR81. AdminClient
> >> > > JR81.1 ListClusterMirrors: We need to add the option to specify
> >> different
> >> > filters supported in the request.
> >> >
> >> > We already have the `--mirror` option. Added `--state` and
> >> > `--sourceClusterId` options to support the other filters.
> >> >
> >> > > JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations:
> The
> >> > request doesn't support this option. Could we add other options
> >> supported
> >> > in the request?
> >> >
> >> > Currently, we have “includeAuthorizedOperations” in DescribeCluster,
> >> > DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
> >> > DescribeStreamsGroups requests. But none of them include the option in
> >> the
> >> > tool to allow users to enable it. I think the reason is that we don’t
> >> want
> >> > to show this to users on the tool output. But having this field, we
> >> allow
> >> > the advanced users to invoke admin API with
> includeAuthorizedOperations
> >> > enabled to get the results they want.
> >> >
> >> >
> >> > > JR81.3 LeaderStateDescription: Should it be named
> >> > MirrorPartitionStateDescription? Also, could we make the fields
> >> consistent
> >> > with those in the response?
> >> >
> >> > OK. KIP is updated.
> >> >
> >> >
> >> > Thank you,
> >> > Luke
> >> >
> >> > On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <dev@kafka.apache.org
> >
> >> > wrote:
> >> >
> >> >> Hi, Luke,
> >> >>
> >> >> Thanks for the reply.
> >> >>
> >> >> JR59.1 Could you add the RPC to the workflow? The broker first needs
> to
> >> >> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
> >> >> mirror name and then issue DescribeClusterMirrors to get the LME.
> >> >>
> >> >> JR71. mirror.support.unclean.leader.election : The name is confusing.
> >> It's
> >> >> also used in LOG_TRUNCATION, which is not related to unclean leader
> >> >> election. Could we use a better name?
> >> >>
> >> >> JR72. DescribeClusterMirrors
> >> >> JR72.1 A user may specify just the topic name when describing a
> mirror.
> >> >> Should we include the topic name in the request?
> >> >> JR72.2 LastMirrorEpochLookups is a bit weird since the response
> returns
> >> >> either mirror state or mirror lag and LastMirrorEpochLookups doesn't
> >> match
> >> >> either.
> >> >> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> JR72.4 "includeMirrorLags: source offset, destination offset, lag"
> >> >> DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
> >> >> should be omitted.
> >> >>
> >> >> JR73. MirrorPartitionState record, LastMirrorEpochs record and
> >> >> MirrorPartitionStateRequest: The values for type and name are
> >> incorrect.
> >> >>
> >> >> JR74. ListClusterMirrorsRequest
> >> >> JR74.1 When is DesiredMirrorStateFilter used? There is no option to
> >> >> specify
> >> >> the desired state in the mirror-topic command.
> >> >> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
> >> >>
> >> >> JR75. ReadMirrorLags
> >> >> JR75.1 Should it be named ReadMirrorOffsets since the response
> doesn't
> >> >> include lag?
> >> >> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >> JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to
> >> be
> >> >> consistent.
> >> >>
> >> >> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> >>
> >> >> JR77. Delete Mirror: "3. The controller verifies that no topics are
> in
> >> >> MIRRORING or PAUSED state."
> >> >> Is this necessary since step 2 has verified all partitions are in
> >> STOPPED
> >> >> state, which can only happen if the topics are in STOPPED state.
> >> >>
> >> >> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
> >> >> bumped,
> >> >> ABORT markers are appended for all ongoing transactions, last mirror
> >> >> epochs
> >> >> are persisted, and a MIRROR_PID_RESET record appended."
> >> >> This describes the state for STOPPED, not STOPPING.
> >> >>
> >> >> JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
> >> >> timeout field?
> >> >>
> >> >> JR80. PauseMirrorTopicsOptions: Should it support patterns? The
> command
> >> >> line seems to suppor it.
> >> >>
> >> >> JR81. AdminClient
> >> >> JR81.1 ListClusterMirrors: We need to add the option to specify
> >> different
> >> >> filters supported in the request.
> >> >> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
> >> >> request doesn't support this option. Could we add other options
> >> supported
> >> >> in the request?
> >> >> JR81.3 LeaderStateDescription: Should it be named
> >> >> MirrorPartitionStateDescription? Also, could we make the fields
> >> consistent
> >> >> with those in the response?
> >> >>
> >> >> Jun
> >> >>
> >> >> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com> wrote:
> >> >>
> >> >> > Hi Jun,
> >> >> >
> >> >> > Thanks for the good comments again.
> >> >> > Answering them below.
> >> >> >
> >> >> > > JR51. "ULE_LOG_TRUNCATION When
> >> mirror.support.unclean.leader.election
> >> >> is
> >> >> > enabled and the destination leader has log truncation, the state
> will
> >> >> move
> >> >> > from MIRRORING to this state."
> >> >> > This still suggests that divergeEpoch is only handled if
> >> >> > mirror.support.unclean.leader.election
> >> >> > is enabled.
> >> >> >
> >> >> >
> >> >> > OK, KIP is updated to make it clear.
> >> >> >
> >> >> > > JR55. Should the mirror state and the mirror lags always be
> exposed
> >> >> > together? If so, we can remove IncludeMirrorState and
> >> IncludeMirrorLags
> >> >> > from DescribeClusterMirrorsRequest. If not, it's better to use two
> >> >> separate
> >> >> > RPCs.
> >> >> >
> >> >> >
> >> >> > No, we sometimes only need MirrorState, and sometimes need both. So
> >> I’ll
> >> >> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
> >> >> > DescribeClusterMirrorsRequest. And we will use ReadMirrorStates and
> >> >> > ReadMirrorLags to get each info.
> >> >> >
> >> >> >
> >> >> > > DescribeClusterMirrorsRequest should be used in the Describe
> Mirror
> >> >> flow,
> >> >> > right? Currently it's missing. Also, I assume the admin client
> >> refers to
> >> >> > the one in the kafka-cluster-mirrors tool. But it seems that
> >> sometimes
> >> >> it
> >> >> > also refers to the admin client on the broker? Perhaps we can use
> >> 'the
> >> >> > tool' instead to make it clear.
> >> >> >
> >> >> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
> >> >> > And sorry for the confusion. I’d like to keep the admin client as
> the
> >> >> one
> >> >> > invoking from the tool (or user’s direct admin client API
> >> invocation).
> >> >> On
> >> >> > the broker side, we actually use InterBrokerSendThread to invoke
> >> these
> >> >> RPCs
> >> >> > or get directly from MetadataImage. I’ve updated the KIP to make it
> >> >> clear.
> >> >> >
> >> >> >
> >> >> > > JR58. BumpLeaderEpochs: Why does it need to be issued to the
> >> broker,
> >> >> > instead of just to the controller?
> >> >> >
> >> >> > Yes, we only need to issue to the controller. Updated to the
> >> >> > BumpLeaderEpochs API.
> >> >> >
> >> >> >
> >> >> > > JR59. Start Mirror Topics:
> >> >> > > JR59.1 "During this state, the broker queries the source cluster
> >> for
> >> >> last
> >> >> > mirror epochs".
> >> >> > Could you describe which RPC is issued to find the LME?
> >> >> >
> >> >> > We’re using describeClusterMirror API via admin client here.
> >> >> >
> >> >> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
> request
> >> >> > doesn't
> >> >> > contain a replication factor.
> >> >> >
> >> >> > We use “default.replication.factor” when topic creation, so the
> >> >> > INVALID_REPLICATION_FACTOR will be thrown like normal topic
> creation.
> >> >> Also,
> >> >> > because we require the replicas for the partition assignment to be
> >> all
> >> >> > active and not in fenced or controlled shutdown(JR35), we’ll throw
> >> >> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
> >> >> >
> >> >> >
> >> >> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used
> by a
> >> >> > different topic name"
> >> >> > Should we also return this error when the topic name is used as a
> >> >> regular
> >> >> > topic?
> >> >> >
> >> >> > There are 2 errors we will throw when topic id <-> topic name
> >> mismatch.
> >> >> > Suppose provided topic id: ID_A, topic name: NAME_A
> >> >> > 1. Destination cluster exists topic id: ID_A, but topic name is:
> >> NAME_B
> >> >> > TOPIC_ALREADY_EXISTS will be thrown
> >> >> > 2. Destination cluster exists topic id: ID_B, but topic name is:
> >> NAME_A
> >> >> > INCONSISTENT_TOPIC_ID will be thrown
> >> >> >
> >> >> > Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS
> error
> >> >> for
> >> >> > these 2 cases makes more sense. KIP updated.
> >> >> >
> >> >> >
> >> >> > > JR60. CreateClusterMirrorRequest: When will it receive an
> >> >> > INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> >> > CLUSTER_MIRROR_ALREADY_EXISTS?
> >> >> >
> >> >> >
> >> >> > Sorry, createClusterMirrorRequest doesn’t throw
> >> >> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate the
> >> >> mirror
> >> >> > states when creating a cluster mirror. We only care if the “mirror
> >> name”
> >> >> > exists or not. KIP updated.
> >> >> >
> >> >> >
> >> >> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
> >> >> remove
> >> >> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> >> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe
> >> the
> >> >> > specific state causing the error.
> >> >> >
> >> >> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following
> >> errors:
> >> >> > a. MIRROR_TOPIC_ALREADY_PAUSED
> >> >> > b. MIRROR_TOPIC_NOT_PAUSED,
> >> >> > c. MIRROR_TOPIC_ALREADY_STOPPED,
> >> >> > d. CLUSTER_MIRROR_NOT_EMPTY
> >> >> > e. MIRROR_TOPIC_NOT_STOPPED
> >> >> >
> >> >> > KIP is updated.
> >> >> >
> >> >> >
> >> >> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
> >> >> requests
> >> >> > (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
> >> Could
> >> >> we
> >> >> > make them consistent?
> >> >> >
> >> >> >
> >> >> > Good catch. The updated errors for the new added RPC are this:
> >> >> >
> >> >> > CreateClusterMirror: NOT_CONTROLLER
> >> >> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >> >
> >> >> >
> >> >> > > JR63. StopMirrorTopics:
> >> >> > > JR63.1 "The broker validates that all target topic partitions are
> >> in
> >> >> > either
> >> >> > LOG_TRUNCATION or MIRRORING or FAILED state."
> >> >> > Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED
> >> can.
> >> >> >
> >> >> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is
> >> updated.
> >> >> >
> >> >> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error
> since
> >> >> PAUSED
> >> >> > -> STOPPED is valid?
> >> >> >
> >> >> > Yes, it is not correct. KIP updated to remove this error.
> >> >> >
> >> >> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
> >> stopped
> >> >> > state"
> >> >> > This is weird. The reason to issue StopMirrorTopics is that it's
> not
> >> in
> >> >> > stopped state.
> >> >> >
> >> >> > Indeed. KIP is updated to only include
> INVALID_CLUSTER_MIRROR_STATES
> >> >> > error, and will be thrown when states are not in MIRRORING or
> PAUSED
> >> >> state.
> >> >> >
> >> >> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
> >> >> derived
> >> >> > from SourceOffset and DestinationOffset.
> >> >> >
> >> >> > True. Removed.
> >> >> >
> >> >> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems
> >> that we
> >> >> > could just set the partition level state to STOPPED.
> >> >> >
> >> >> > Nice catch. We don’t use it at all. That is some old leftover field
> >> we
> >> >> > forgot to remove. KIP updated.
> >> >> >
> >> >> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be
> >> sent
> >> >> to
> >> >> > the controller?
> >> >> >
> >> >> > No, these 2 only send to the broker. KIP updated.
> >> >> >
> >> >> >
> >> >> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and
> >> StateEpoch?
> >> >> It
> >> >> > seems both are used for fencing and fencing based on the latter is
> >> >> > stronger.
> >> >> >
> >> >> > Yes, basically the StateEpoch can fence for most cases. The
> >> LeaderEpoch
> >> >> > fencing is used for cases like:
> >> >> >
> >> >> > 1. b1 is the current leader for foo-0, preparing write state from
> >> >> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
> >> >> > 2. leadership change to b2 for foo-0, LE changes to 2.
> >> >> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1 }.
> >> >> > 4. The coordinator receives the read from b2 in step (3), the
> highest
> >> >> > lastSeen LE for foo-0 updates to 2 in coordinator.
> >> >> > 5. The writes state from b1 in step (1) arrives the coordinator,
> but
> >> it
> >> >> > rejects b1 writes because the highest lastSeen LE is 2, which is >
> 1
> >> in
> >> >> the
> >> >> > request.
> >> >> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2, stateEpoch:
> 1
> >> and
> >> >> > succeeds.
> >> >> >
> >> >> > The leaderEpoch fence is to avoid any stale broker writes after the
> >> new
> >> >> > leader epoch read. This cannot be protected by the stateEpoch. The
> >> >> > stateEpoch can prevent the write with older stateEpoch. So in the
> >> >> situation
> >> >> > above, the stateEpoch cannot determine if we should accept b1
> writes
> >> or
> >> >> > not, but leaderEpoch can. By the way, we borrowed this concept from
> >> >> > KIP-932:
> >> >> >
> >> >>
> >>
> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
> >> <
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rX3-NwDU3s0U7POQYH1R-DvASnCPTTtFRWxZTVaS-PQHg-C1TAcu9xMe3MQF4fIXGO67WKOqrlz-lc8$
> >
> >> >> <
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!u8EkPGXm8_VKdltXy8lrIW14mR2kgYIoaBPD7LO2uAN6rjhrfCX37Pfa6r85QPpSwSPh4V6eW_bKlns$
> >> >
> >> >> > <
> >> >>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
> >> >> >
> >> >> >
> >> >> >
> >> >> > > JR68. Could we make the usage of topicId and topicName in the
> RPCs
> >> >> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> >> > BumpLeaderEpochsReponse uses TopicName.
> >> >> >
> >> >> > Yes, we have made:
> >> >> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use TopicName.
> >> >> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse use
> >> >> > TopicName.
> >> >> > 3. MirrorTopicStateChangeRecord uses TopicName because we will do
> >> >> > validation before writing the metadata log.
> >> >> >
> >> >> >
> >> >> > > JR69. MIRROR_PID_RESET:
> >> >> > > JR69.1 Could you specify the value of the control record's type?
> >> >> >
> >> >> > It’s MIRROR_PID_RESET(7). KIP updated.
> >> >> >
> >> >> > > JR69.2 How is the SourceClusterId field used?
> >> >> >
> >> >> > I’m also surprised to see this field in the MirrorPidResetRecord. I
> >> >> traced
> >> >> > back to the internal PR we added for this field, and confirmed it
> is
> >> not
> >> >> > used since it was added. Sorry for the confusion and nice catch!
> KIP
> >> >> > updated.
> >> >> >
> >> >> > > JR70. MirrorPartitionState: type should be "data" not
> >> >> "coordinator-key",
> >> >> > right? Ditto for LastMirrorEpochs.
> >> >> > We store both MirrorPartitionState and LastMirrorEpochs records
> into
> >> >> > internal __mirror_state topic, and rely on Coordinator Runtime to
> >> >> > write/read the records, like how `OffsetCommit` and `GroupMetadata`
> >> >> work.
> >> >> > So I think we should keep them as
> >> "coordinator-key"/"coordinator-value"
> >> >> > type.
> >> >> >
> >> >> >
> >> >> > Thank you,
> >> >> > Luke
> >> >> >
> >> >> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <
> >> dev@kafka.apache.org>
> >> >> > wrote:
> >> >> >
> >> >> >> Hi, Luke,
> >> >> >>
> >> >> >> Thanks for the reply.
> >> >> >>
> >> >> >> JR51. "ULE_LOG_TRUNCATION When
> >> mirror.support.unclean.leader.election
> >> >> is
> >> >> >> enabled and the destination leader has log truncation, the state
> >> will
> >> >> move
> >> >> >> from MIRRORING to this state."
> >> >> >> This still suggests that divergeEpoch is only handled if
> >> >> >> mirror.support.unclean.leader.election
> >> >> >> is enabled.
> >> >> >>
> >> >> >> JR55. Should the mirror state and the mirror lags always be
> exposed
> >> >> >> together? If so, we can remove IncludeMirrorState and
> >> IncludeMirrorLags
> >> >> >> from DescribeClusterMirrorsRequest. If not, it's better to use two
> >> >> >> separate
> >> >> >> RPCs.
> >> >> >>
> >> >> >> DescribeClusterMirrorsRequest should be used in the Describe
> Mirror
> >> >> flow,
> >> >> >> right? Currently it's missing. Also, I assume the admin client
> >> refers
> >> >> to
> >> >> >> the one in the kafka-cluster-mirrors tool. But it seems that
> >> sometimes
> >> >> it
> >> >> >> also refers to the admin client on the broker? Perhaps we can use
> >> 'the
> >> >> >> tool' instead to make it clear.
> >> >> >>
> >> >> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the
> broker,
> >> >> >> instead of just to the controller?
> >> >> >>
> >> >> >> JR59. Start Mirror Topics:
> >> >> >> JR59.1 "During this state, the broker queries the source cluster
> for
> >> >> last
> >> >> >> mirror epochs".
> >> >> >> Could you describe which RPC is issued to find the LME?
> >> >> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The
> request
> >> >> >> doesn't
> >> >> >> contain a replication factor.
> >> >> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by
> a
> >> >> >> different topic name"
> >> >> >> Should we also return this error when the topic name is used as a
> >> >> regular
> >> >> >> topic?
> >> >> >>
> >> >> >> JR60. CreateClusterMirrorRequest: When will it receive an
> >> >> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> >> >> CLUSTER_MIRROR_ALREADY_EXISTS?
> >> >> >>
> >> >> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
> >> >> remove
> >> >> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> >> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe
> >> the
> >> >> >> specific state causing the error.
> >> >> >>
> >> >> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
> >> >> requests
> >> >> >> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
> >> >> Could we
> >> >> >> make them consistent?
> >> >> >>
> >> >> >> JR63. StopMirrorTopics:
> >> >> >> JR63.1 "The broker validates that all target topic partitions are
> in
> >> >> >> either
> >> >> >> LOG_TRUNCATION or MIRRORING or FAILED state."
> >> >> >> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED
> >> can.
> >> >> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
> >> >> PAUSED
> >> >> >> -> STOPPED is valid?
> >> >> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
> stopped
> >> >> >> state"
> >> >> >> This is weird. The reason to issue StopMirrorTopics is that it's
> >> not in
> >> >> >> stopped state.
> >> >> >>
> >> >> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
> >> derived
> >> >> >> from SourceOffset and DestinationOffset.
> >> >> >>
> >> >> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems
> that
> >> we
> >> >> >> could just set the partition level state to STOPPED.
> >> >> >>
> >> >> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be
> >> sent
> >> >> to
> >> >> >> the controller?
> >> >> >>
> >> >> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and
> >> StateEpoch? It
> >> >> >> seems both are used for fencing and fencing based on the latter is
> >> >> >> stronger.
> >> >> >>
> >> >> >> JR68. Could we make the usage of topicId and topicName in the RPCs
> >> >> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> >> >> BumpLeaderEpochsReponse uses TopicName.
> >> >> >>
> >> >> >> JR69. MIRROR_PID_RESET:
> >> >> >> JR69.1 Could you specify the value of the control record's type?
> >> >> >> JR69.2 How is the SourceClusterId field used?
> >> >> >>
> >> >> >> JR70. MirrorPartitionState: type should be "data" not
> >> >> "coordinator-key",
> >> >> >> right? Ditto for LastMirrorEpochs.
> >> >> >>
> >> >> >> Jun
> >> >> >>
> >> >> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com>
> >> wrote:
> >> >> >>
> >> >> >> > Hi Jun,
> >> >> >> >
> >> >> >> > Thanks for the comments.
> >> >> >> > Answering them below.
> >> >> >> >
> >> >> >> > > JR36. Sounds good. Should we remove
> >> >> >> MirrorLeaderEpochExceededException?
> >> >> >> >
> >> >> >> > Sorry, it’s a confusing name. The
> >> MirrorLeaderEpochExceededException
> >> >> is
> >> >> >> > used and thrown when the fetched mirror batch leader epoch >
> local
> >> >> >> leader
> >> >> >> > epoch in destination cluster. And then we’ll stop mirroring and
> >> move
> >> >> the
> >> >> >> > partition to EPOCH_FENSING state to wait for local leader epoch
> >> >> bump. I
> >> >> >> > think since we don’t have the concept of `MirrorLeaderEpoch`
> >> anymore,
> >> >> >> the
> >> >> >> > `MirrorLeaderEpochExceededException` should not cause confusion
> >> now.
> >> >> >> I’ll
> >> >> >> > keep the name for the exception.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR51. Sounds good. Do we plan to always set lastFetchedEpoch
> in
> >> the
> >> >> >> > mirror
> >> >> >> > fetch request issued to the source cluster, independent of the
> >> value
> >> >> of
> >> >> >> > mirror.support.unclean.leader.election? It's needed to detect
> >> >> divergence
> >> >> >> > during failover. However, if it's also set when
> >> >> >> > mirror.support.unclean.leader.election=false,
> >> >> >> > an unclean leader election in the source cluster can halt the
> >> >> mirroring
> >> >> >> > since we don’t do anything with the returned divergeEpoch in
> that
> >> >> mode
> >> >> >> > and continue
> >> >> >> > the next fetch.
> >> >> >> >
> >> >> >> > > Do we plan to always set lastFetchedEpoch in the mirror
> >> >> >> > fetch request issued to the source cluster, independent of the
> >> value
> >> >> of
> >> >> >> > Mirror.support.unclean.leader.election?
> >> >> >> >
> >> >> >> > Yes, we always set lastFetchedEpoch in the mirror fetch request.
> >> >> >> >
> >> >> >> > > However, if it's also set when
> >> >> >> > mirror.support.unclean.leader.election=false,
> >> >> >> > an unclean leader election in the source cluster can halt the
> >> >> mirroring
> >> >> >> > since we don’t do anything with the returned divergeEpoch in
> that
> >> >> mode
> >> >> >> > and continue the next fetch.
> >> >> >> >
> >> >> >> > Sorry I didn’t make it clear. When
> >> >> >> > “mirror.support.unclean.leader.election=false” and a
> >> “divergeEpoch”
> >> >> is
> >> >> >> set
> >> >> >> > in fetch response, the leader node will handle the divergeEpoch
> >> and
> >> >> then
> >> >> >> > continue the next fetch as usual. The difference with
> >> >> >> > “mirror.support.unclean.leader.election=true” is that “should we
> >> wait
> >> >> >> for
> >> >> >> > ALL replicas caught up (complete truncation)”. Without the
> >> waiting,
> >> >> it’s
> >> >> >> > possible the inconsistent log issue in KAFKA-18723 could happen
> >> >> between
> >> >> >> > leader and follower nodes in the destination cluster because we
> >> don’t
> >> >> >> have
> >> >> >> > “MirrorLeaderEpoch” concept now to reject higher leader epoch
> >> >> batches.
> >> >> >> That
> >> >> >> > is, we won't halt the mirroring if "divergeEpoch" is set but
> >> >> >> > “mirror.support.unclean.leader.election=false”.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR55. The DescribeClusterMirrorsRequest still returns both the
> >> >> mirror
> >> >> >> > partition state and the mirror partition leader state. The
> >> problem is
> >> >> >> that
> >> >> >> > if a caller only needs one of the state, the other state becomes
> >> >> >> redundant.
> >> >> >> > For example, when starting the mirror, to obtain LME, we only
> need
> >> >> the
> >> >> >> > mirror partition state. Here is an alternative design: We expose
> >> the
> >> >> >> mirror
> >> >> >> > partition state through ReadMirrorState. The client will use the
> >> >> >> > FindCoordinatorRequest
> >> >> >> > to find the right mirror partition coordinator to issue that
> >> >> request. We
> >> >> >> > expose the mirror partition leader state through the
> >> >> >> ReadClusterMirrorLags
> >> >> >> > request. The client can use the metadata request to find the
> >> >> partition
> >> >> >> > leader to issue that request.
> >> >> >> >
> >> >> >> > > In the describing mirror flow, the admin client first issues
> the
> >> >> >> > ListMirrorRequest to find all the mirrors and the topics being
> >> >> >> mirrored. It
> >> >> >> > then issues the metadata request to find the partitions for
> those
> >> >> topics
> >> >> >> > and issues a FindCoordinatorRequest to find the mirror partition
> >> >> state
> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
> >> >> partition
> >> >> >> > state coordinator and ReadClusterMirrorLags to the partition
> >> leader.
> >> >> >> >
> >> >> >> > > In the starting mirror flow, the partition leader in the
> >> >> destination
> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name based
> >> on
> >> >> its
> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to find the
> >> >> mirror
> >> >> >> > partition state coordinators. Finally, it issues ReadMirrorState
> >> to
> >> >> the
> >> >> >> > mirror partition state coordinator to get the LME.
> >> >> >> >
> >> >> >> > > We can also improve the ListMirror request to accept a filter
> >> >> based on
> >> >> >> > mirror name, source cluster ID and topic name.
> >> >> >> >
> >> >> >> >
> >> >> >> > Answers:
> >> >> >> >
> >> >> >> > > The DescribeClusterMirrorsRequest still returns both the
> mirror
> >> >> >> > partition state and the mirror partition leader state. The
> >> problem is
> >> >> >> that
> >> >> >> > if a caller only needs one of the state, the other state becomes
> >> >> >> redundant.
> >> >> >> > For example, when starting the mirror, to obtain LME, we only
> need
> >> >> the
> >> >> >> > mirror partition state.
> >> >> >> >
> >> >> >> > Good point!
> >> >> >> >
> >> >> >> >
> >> >> >> > > Here is an alternative design: We expose the mirror
> >> >> >> > partition state through ReadMirrorState. The client will use the
> >> >> >> > FindCoordinatorRequest
> >> >> >> > to find the right mirror partition coordinator to issue that
> >> >> request. We
> >> >> >> > expose the mirror partition leader state through the
> >> >> >> ReadClusterMirrorLags
> >> >> >> > request. The client can use the metadata request to find the
> >> >> partition
> >> >> >> > leader to issue that request.
> >> >> >> >
> >> >> >> >
> >> >> >> > Sounds good. But we try not to expose
> >> >> >> > ReadMirrorState/ReadClusterMirrorLags directly to the admin
> client
> >> >> >> because
> >> >> >> > users might get confused about these 2 requests, and they are
> also
> >> >> both
> >> >> >> > related to describeClusterMirror. I think we can add 2
> additional
> >> >> >> fields to
> >> >> >> > describeClusterMirrorRequest: “includeMirrorStates” and
> >> >> >> > “includeMirrorLags”. So the admin client can include either one
> or
> >> >> both
> >> >> >> of
> >> >> >> > them in the DescrbieClusterMirrorRequest based on the provided
> >> >> options.
> >> >> >> KIP
> >> >> >> > is updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > In the describing mirror flow, the admin client first issues
> the
> >> >> >> > ListMirrorRequest to find all the mirrors and the topics being
> >> >> >> mirrored. It
> >> >> >> > then issues the metadata request to find the partitions for
> those
> >> >> topics
> >> >> >> > and issues a FindCoordinatorRequest to find the mirror partition
> >> >> state
> >> >> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
> >> >> partition
> >> >> >> > state coordinator and ReadClusterMirrorLags to the partition
> >> leader.
> >> >> >> >
> >> >> >> > Makes sense. KIP is updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > In the starting mirror flow, the partition leader in the
> >> >> destination
> >> >> >> > cluster issues a ListMirrorRequest to find the mirror name based
> >> on
> >> >> its
> >> >> >> > cluster id. It then issues a FindCoordinatorRequest to find the
> >> >> mirror
> >> >> >> > partition state coordinators. Finally, it issues ReadMirrorState
> >> to
> >> >> the
> >> >> >> > mirror partition state coordinator to get the LME.
> >> >> >> >
> >> >> >> > Make sense. So we can send DescribeClusterMirrorRequest with
> >> >> >> > “includeMirrorStates” only.
> >> >> >> >
> >> >> >> > > We can also improve the ListMirror request to accept a filter
> >> >> based on
> >> >> >> > mirror name, source cluster ID and topic name.
> >> >> >> >
> >> >> >> > Good suggestion, just like what ListGroupsRequest does.
> >> >> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
> >> >> >> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP
> >> >> >> updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR56. Could you list the ACL needed for each new RPC?
> >> >> >> >
> >> >> >> >
> >> >> >> > Yes, it’s already listed in “Security Control” section.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR57. In the command workflow, it sometimes says "The user
> >> sends"
> >> >> and
> >> >> >> > some
> >> >> >> > other times says "The admin client sends". It's better to use
> "The
> >> >> admin
> >> >> >> > client" consistently in all those places.
> >> >> >> >
> >> >> >> > Agree. KIP updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > Thank you,
> >> >> >> > Luke
> >> >> >> >
> >> >> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <
> >> dev@kafka.apache.org
> >> >> >
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Hi, Luke,
> >> >> >> >>
> >> >> >> >> Thanks for the reply.
> >> >> >> >>
> >> >> >> >> JR36. Sounds good. Should we remove
> >> >> MirrorLeaderEpochExceededException?
> >> >> >> >>
> >> >> >> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in
> >> the
> >> >> >> mirror
> >> >> >> >> fetch request issued to the source cluster, independent of the
> >> >> value of
> >> >> >> >> mirror.support.unclean.leader.election? It's needed to detect
> >> >> >> divergence
> >> >> >> >> during failover. However, if it's also set when
> >> >> >> >> mirror.support.unclean.leader.election=false,
> >> >> >> >> an unclean leader election in the source cluster can halt the
> >> >> mirroring
> >> >> >> >> since we don’t do anything with the returned divergeEpoch in
> that
> >> >> mode
> >> >> >> >> and continue
> >> >> >> >> the next fetch.
> >> >> >> >>
> >> >> >> >> JR55. The DescribeClusterMirrorsRequest still returns both the
> >> >> mirror
> >> >> >> >> partition state and the mirror partition leader state. The
> >> problem
> >> >> is
> >> >> >> that
> >> >> >> >> if a caller only needs one of the state, the other state
> becomes
> >> >> >> >> redundant.
> >> >> >> >> For example, when starting the mirror, to obtain LME, we only
> >> need
> >> >> the
> >> >> >> >> mirror partition state. Here is an alternative design: We
> expose
> >> the
> >> >> >> >> mirror
> >> >> >> >> partition state through ReadMirrorState. The client will use
> the
> >> >> >> >> FindCoordinatorRequest
> >> >> >> >> to find the right mirror partition coordinator to issue that
> >> >> request.
> >> >> >> We
> >> >> >> >> expose the mirror partition leader state through the
> >> >> >> ReadClusterMirrorLags
> >> >> >> >> request. The client can use the metadata request to find the
> >> >> partition
> >> >> >> >> leader to issue that request.
> >> >> >> >>
> >> >> >> >> In the describing mirror flow, the admin client first issues
> the
> >> >> >> >> ListMirrorRequest to find all the mirrors and the topics being
> >> >> >> mirrored.
> >> >> >> >> It
> >> >> >> >> then issues the metadata request to find the partitions for
> those
> >> >> >> topics
> >> >> >> >> and issues a FindCoordinatorRequest to find the mirror
> partition
> >> >> state
> >> >> >> >> coordinators. Finally, it issues ReadMirrorState to the mirror
> >> >> >> partition
> >> >> >> >> state coordinator and ReadClusterMirrorLags to the partition
> >> leader.
> >> >> >> >>
> >> >> >> >> In the starting mirror flow, the partition leader in the
> >> destination
> >> >> >> >> cluster issues a ListMirrorRequest to find the mirror name
> based
> >> on
> >> >> its
> >> >> >> >> cluster id. It then issues a FindCoordinatorRequest to find the
> >> >> mirror
> >> >> >> >> partition state coordinators. Finally, it issues
> ReadMirrorState
> >> to
> >> >> the
> >> >> >> >> mirror partition state coordinator to get the LME.
> >> >> >> >>
> >> >> >> >> We can also improve the ListMirror request to accept a filter
> >> based
> >> >> on
> >> >> >> >> mirror name, source cluster ID and topic name.
> >> >> >> >>
> >> >> >> >> JR56. Could you list the ACL needed for each new RPC?
> >> >> >> >>
> >> >> >> >> JR57. In the command workflow, it sometimes says "The user
> sends"
> >> >> and
> >> >> >> some
> >> >> >> >> other times says "The admin client sends". It's better to use
> >> "The
> >> >> >> admin
> >> >> >> >> client" consistently in all those places.
> >> >> >> >>
> >> >> >> >> Jun
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com>
> >> wrote:
> >> >> >> >>
> >> >> >> >> > Hi Jun,
> >> >> >> >> >
> >> >> >> >> > Thanks for your comments.
> >> >> >> >> > Answering them below.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> >> >> >> specifically
> >> >> >> >> > designed for the unclean leader election support in Cluster
> >> >> Mirroring
> >> >> >> >> > because the unclean leader election will break the assumption
> >> that
> >> >> >> the
> >> >> >> >> > committed data won’t be truncated."
> >> >> >> >> > If the source cluster has an unclean leader election, the
> >> >> destination
> >> >> >> >> > cluster will detect the inconsistency and truncate the log at
> >> the
> >> >> >> >> > destination leader. We can just
> >> >> >> >> > extend mirror.support.unclean.leader.election to cover this
> >> case,
> >> >> >> i.e.,
> >> >> >> >> we
> >> >> >> >> > wait until all replicas' log end offsets are reduced to the
> >> target
> >> >> >> >> offset
> >> >> >> >> > before resuming mirroring. This avoids the inconsistency.
> >> >> >> >> >
> >> >> >> >> > If the destination cluster has an unclean leader election, it
> >> will
> >> >> >> >> > re-fetch the same data from the source. There won't be any
> >> >> >> inconsistency
> >> >> >> >> > either.
> >> >> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Yes, that works, too. I agree this is better because the
> >> unclean
> >> >> >> leader
> >> >> >> >> > election is rare, we don't want to occupy a field just for
> it.
> >> >> >> >> >
> >> >> >> >> > So to achieve this, we have to:
> >> >> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
> >> >> >> >> > 2. When the destination cluster leader completes a log
> >> truncation
> >> >> >> after
> >> >> >> >> a
> >> >> >> >> > fetch response due to divergeEpoch, we know it must be caused
> >> by
> >> >> >> unclean
> >> >> >> >> > leader election in the source cluster because we only fetch
> >> >> committed
> >> >> >> >> > records. If “mirror.support.unclean.leader.election=false”,
> we
> >> >> don’t
> >> >> >> do
> >> >> >> >> > anything and continue the next fetch. If
> >> >> >> >> > “mirror.support.unclean.leader.election=true”, we remove the
> >> >> mirror
> >> >> >> >> fetcher
> >> >> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
> >> >> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to
> >> be
> >> >> >> caught
> >> >> >> >> up
> >> >> >> >> > into ISR and reduced to the expected LEO.
> >> >> >> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
> >> >> >> >> >
> >> >> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch
> >> API.
> >> >> >> >> > KIP is updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR51. Make sense. To truncate based on LME, we need to
> issue
> >> >> >> another
> >> >> >> >> RPC
> >> >> >> >> > to translate the LME to an offset. It would be useful to
> >> include
> >> >> >> that in
> >> >> >> >> > the workflow description.
> >> >> >> >> >
> >> >> >> >> > About the additional RPC to translate the LME to the offset
> >> (i.e.
> >> >> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on this
> >> to do
> >> >> >> the
> >> >> >> >> > offset translation. We use the local leader epoch cache in
> the
> >> >> >> >> destination
> >> >> >> >> > cluster to translate the epoch to save the RPC call. This
> works
> >> >> >> because
> >> >> >> >> of
> >> >> >> >> > an invariant the LME truncation protocol maintains: for any
> >> leader
> >> >> >> >> epoch ≤
> >> >> >> >> > LME, that epoch's data belongs to exactly one cluster, either
> >> it
> >> >> >> >> originated
> >> >> >> >> > there, or it was mirrored. This holds even across repeated
> >> mirror
> >> >> ->
> >> >> >> >> > reverse-mirror -> mirror cycles, since each direction's
> >> >> >> LOG_TRUNCATION
> >> >> >> >> > phase re-establishes the invariant before that cluster starts
> >> >> >> mirroring.
> >> >> >> >> > With this invariant, the replication protocol can find out
> the
> >> >> >> diverge
> >> >> >> >> > point and truncate the log correctly when mirroring. For
> >> example:
> >> >> >> >> >
> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> > Offset 2: A (LE: 0)
> >> >> >> >> > Offset 3: A (LE: 1)
> >> >> >> >> >
> >> >> >> >> > Cluster A crashes while Cluster B mirrors to offset 1.
> Failover
> >> >> to B
> >> >> >> and
> >> >> >> >> > store the LME=0 in cluster B. When cluster A recovers and
> >> starts
> >> >> the
> >> >> >> >> > reverse mirroring, it’ll query the LME from B and get back
> >> LME: 0.
> >> >> >> Then
> >> >> >> >> > cluster A gets the end offset for epoch 0 from
> >> leaderEpochCache,
> >> >> and
> >> >> >> >> gets
> >> >> >> >> > offset 2. So offset 3 is truncated. The log is like this now:
> >> >> >> >> >
> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
> >> >> >> >> > Offset 3: B (LE:
> >> 10)
> >> >> >> >> >
> >> >> >> >> > When mirror starts, cluster A sends the fetch request with {
> >> >> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the
> >> divergingEpoch
> >> >> >> check
> >> >> >> >> in
> >> >> >> >> > the cluster B queries the endOffset for leader epoch 0, and
> >> found
> >> >> the
> >> >> >> >> end
> >> >> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so
> returns
> >> >> fetch
> >> >> >> >> > response with divergingEpoch info to cluster A for log
> >> >> truncation. In
> >> >> >> >> the
> >> >> >> >> > end, offset 2 in cluster A will be truncated and then log
> >> >> converges
> >> >> >> with
> >> >> >> >> > cluster B.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit
> >> >> strange
> >> >> >> >> > because it mixes querying the mirror partition state with
> >> querying
> >> >> >> the
> >> >> >> >> > mirror partition leader. Another option is to use
> >> >> >> >> ReadMirrorStatesRequest
> >> >> >> >> > to query the mirror partition state and a new request to
> query
> >> the
> >> >> >> >> mirror
> >> >> >> >> > partition leader. If we want the adminClient to query the
> right
> >> >> >> >> ClusterMirrorCoordinator
> >> >> >> >> > directly, we can extend the FindCoordinatorRequest to support
> >> >> that.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Good suggestion. Yes, we need to fan out to all brokers
> >> because we
> >> >> >> have
> >> >> >> >> to
> >> >> >> >> > collect info from mirror partition leaders and mirror
> >> >> coordinators.
> >> >> >> If
> >> >> >> >> we
> >> >> >> >> > create a new API to contact mirror partition leaders, it will
> >> make
> >> >> >> the
> >> >> >> >> API
> >> >> >> >> > cleaner and clearer. So we will do this:
> >> >> >> >> >
> >> >> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields
> from
> >> >> >> >> > DescribeClusterMirror request/response
> >> >> >> >> >
> >> >> >> >> > b. Add a new API: ReadClusterMirrorLags API to request for
> >> >> >> SourceOffset,
> >> >> >> >> > DestinationOffset, and Lag info.
> >> >> >> >> >
> >> >> >> >> > The workflow for admin client describeClusterMirror will be
> >> like
> >> >> >> this:
> >> >> >> >> > 1. adminClient sends describeClusterMirror to the
> >> leastLoadedNode
> >> >> >> broker
> >> >> >> >> > 2. The received broker collects the current mirrored
> >> partitions,
> >> >> and
> >> >> >> >> > groups them by coordinator node and by partition leader node.
> >> >> >> >> > 3. The broker sends ReadMirrorState request to the
> coordinator
> >> >> nodes
> >> >> >> to
> >> >> >> >> > get the mirror state related info
> >> >> >> >> > 4. The broker sends ReadClusterMirrorLags request to the
> mirror
> >> >> >> >> partition
> >> >> >> >> > leader nodes to get the mirror lag, source offset,
> destination
> >> >> offset
> >> >> >> >> > (which updates on every fetch response).
> >> >> >> >> > 5. The broker merge all the response into
> describeClusterMirror
> >> >> >> response
> >> >> >> >> > and send to admin client
> >> >> >> >> >
> >> >> >> >> > KIP is updated. Thanks for the suggestion.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR55. For __mirror_state topic, could we describe whether
> it
> >> is
> >> >> >> >> > compacted?
> >> >> >> >> >
> >> >> >> >> > Yes, it is a compacted topic. I made it much clearer in the
> >> >> >> >> > “ClusterMirrorCoordinator” section.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Thank you,
> >> >> >> >> > Luke
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
> >> >> dev@kafka.apache.org
> >> >> >> >
> >> >> >> >> > wrote:
> >> >> >> >> >
> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >>
> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >>
> >> >> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> >> >> >> specifically
> >> >> >> >> >> designed for the unclean leader election support in Cluster
> >> >> >> Mirroring
> >> >> >> >> >> because the unclean leader election will break the
> assumption
> >> >> that
> >> >> >> the
> >> >> >> >> >> committed data won’t be truncated."
> >> >> >> >> >> If the source cluster has an unclean leader election, the
> >> >> >> destination
> >> >> >> >> >> cluster will detect the inconsistency and truncate the log
> at
> >> the
> >> >> >> >> >> destination leader. We can just
> >> >> >> >> >> extend mirror.support.unclean.leader.election to cover this
> >> case,
> >> >> >> >> i.e., we
> >> >> >> >> >> wait until all replicas' log end offsets are reduced to the
> >> >> target
> >> >> >> >> offset
> >> >> >> >> >> before resuming mirroring. This avoids the inconsistency.
> >> >> >> >> >>
> >> >> >> >> >> If the destination cluster has an unclean leader election,
> it
> >> >> will
> >> >> >> >> >> re-fetch
> >> >> >> >> >> the same data from the source. There won't be any
> >> inconsistency
> >> >> >> either.
> >> >> >> >> >>
> >> >> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >> >> >>
> >> >> >> >> >> JR51. Make sense. To truncate based on LME, we need to issue
> >> >> another
> >> >> >> >> RPC
> >> >> >> >> >> to
> >> >> >> >> >> translate the LME to an offset. It would be useful to
> include
> >> >> that
> >> >> >> in
> >> >> >> >> the
> >> >> >> >> >> workflow description.
> >> >> >> >> >>
> >> >> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit
> >> strange
> >> >> >> >> because
> >> >> >> >> >> it mixes querying the mirror partition state with querying
> the
> >> >> >> mirror
> >> >> >> >> >> partition leader. Another option is to use
> >> >> ReadMirrorStatesRequest
> >> >> >> to
> >> >> >> >> >> query
> >> >> >> >> >> the mirror partition state and a new request to query the
> >> mirror
> >> >> >> >> partition
> >> >> >> >> >> leader. If we want the adminClient to query the right
> >> >> >> >> >> ClusterMirrorCoordinator
> >> >> >> >> >> directly, we can extend the FindCoordinatorRequest to
> support
> >> >> that.
> >> >> >> >> >>
> >> >> >> >> >> JR55. For __mirror_state topic, could we describe whether it
> >> is
> >> >> >> >> compacted?
> >> >> >> >> >>
> >> >> >> >> >> Jun
> >> >> >> >> >>
> >> >> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <showuon@gmail.com
> >
> >> >> wrote:
> >> >> >> >> >>
> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >
> >> >> >> >> >> > Thanks for the comments.
> >> >> >> >> >> > Answering them below:
> >> >> >> >> >> >
> >> >> >> >> >> > > JR31.3 It makes sense for the new partition leader to
> >> query
> >> >> the
> >> >> >> >> >> current
> >> >> >> >> >> > mirror state from the local or remote
> MirrorMetadataManager
> >> >> that
> >> >> >> owns
> >> >> >> >> >> the
> >> >> >> >> >> > partition's state. However, the wording in the KIP is
> >> >> confusing.
> >> >> >> >> >> > MirrorMetadataManager
> >> >> >> >> >> > mirror partition state cache should be populated only by
> one
> >> >> way,
> >> >> >> >> which
> >> >> >> >> >> is
> >> >> >> >> >> > replaying __mirror_state partitions this broker leads. By
> >> >> saying
> >> >> >> >> "and by
> >> >> >> >> >> > fetching state from remote coordinators via
> ReadMirrorStates
> >> >> >> RPCs.",
> >> >> >> >> >> it's
> >> >> >> >> >> > as if the mirror partition state cache on each broker
> stores
> >> >> the
> >> >> >> >> state
> >> >> >> >> >> for
> >> >> >> >> >> > all mirror partitions.
> >> >> >> >> >> >
> >> >> >> >> >> > You’re right. KIP is updated to make it clear.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR36. I understand the intention now, but I still don't
> >> >> think we
> >> >> >> >> need
> >> >> >> >> >> > MirrorLeaderEpoch. Let's
> >> >> >> >> >> > assume the common case: no unclean leader election in the
> >> >> source
> >> >> >> >> cluster
> >> >> >> >> >> > during mirroring. Mirroring only fetches the committed
> data
> >> >> from
> >> >> >> the
> >> >> >> >> >> > source, which never changes if obtained a second time. In
> >> your
> >> >> >> >> example
> >> >> >> >> >> > image, follower broker 1 in the destination cluster can
> >> never
> >> >> >> obtain
> >> >> >> >> the
> >> >> >> >> >> > record B(0) because it's never committed in the source and
> >> >> thus is
> >> >> >> >> never
> >> >> >> >> >> > exposed to the destination leader. It's possible for
> broker
> >> 1
> >> >> to
> >> >> >> >> first
> >> >> >> >> >> get
> >> >> >> >> >> > D(1) which is truncated later when broker 0 follows a new
> >> >> leader.
> >> >> >> But
> >> >> >> >> >> the
> >> >> >> >> >> > new leader will refetch the same D(1) from the source
> >> cluster.
> >> >> So,
> >> >> >> >> it's
> >> >> >> >> >> ok
> >> >> >> >> >> > for broker 1 to keep D(1).
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Yes, you are correct. Because mirroring only fetches the
> >> >> committed
> >> >> >> >> data
> >> >> >> >> >> > from the source, the MirrorLeaderEpoch is unnecessary for
> >> >> normal
> >> >> >> >> >> mirroring.
> >> >> >> >> >> > Thanks for the clarification. So I think we can say, the
> >> >> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the
> unclean
> >> >> >> leader
> >> >> >> >> >> > election support in Cluster Mirroring because the unclean
> >> >> leader
> >> >> >> >> >> election
> >> >> >> >> >> > will break the assumption that the committed data won’t be
> >> >> >> truncated.
> >> >> >> >> >> >
> >> >> >> >> >> > Given that, we could gate the MirrorLeaderEpoch field by
> >> only
> >> >> >> >> setting it
> >> >> >> >> >> > in the Fetch request when
> >> >> >> >> "mirror.support.unclean.leader.election=true"
> >> >> >> >> >> > (false by default), and skip it entirely for normal
> >> mirroring.
> >> >> >> What
> >> >> >> >> do
> >> >> >> >> >> you
> >> >> >> >> >> > think?
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR51. Sounds good. Why do we choose to store the last
> >> mirror
> >> >> >> epoch
> >> >> >> >> >> > instead
> >> >> >> >> >> > of a last mirror offset?
> >> >> >> >> >> >
> >> >> >> >> >> > Yes, originally we stored the “last mirror offset” instead
> >> of
> >> >> >> “last
> >> >> >> >> >> mirror
> >> >> >> >> >> > epoch”. But later, we found it doesn’t work for unclean
> >> leader
> >> >> >> >> election
> >> >> >> >> >> > cases. (yes, `unclean leader election` again :) ) For
> >> example:
> >> >> >> >> >> >
> >> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> >> >
> >> >> >> >> >> > Cluster B is mirroring from A. And A crashes and failover
> to
> >> >> B. B
> >> >> >> >> stores
> >> >> >> >> >> > the “last mirror offset” as 1.
> >> >> >> >> >> > When cluster A recovers, somehow the unclean leader
> election
> >> >> >> happens,
> >> >> >> >> >> log
> >> >> >> >> >> > is empty in the new leader, and then some more leadership
> >> >> changes
> >> >> >> to
> >> >> >> >> >> make
> >> >> >> >> >> > the leader epoch become 10. Then it starts the reverse
> >> mirror
> >> >> >> from B.
> >> >> >> >> >> The
> >> >> >> >> >> > log looks like this:
> >> >> >> >> >> >
> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> >> >> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> >> >> >> >> >> > Offset 2:
> >> pid_res
> >> >> (LE:
> >> >> >> >> 10)
> >> >> >> >> >> >
> >> >> >> >> >> > When A starts mirroring from B, log is truncated to last
> >> mirror
> >> >> >> >> offset:
> >> >> >> >> >> 1
> >> >> >> >> >> > (no-op), and then fetch from cluster B. The 1st fetch
> >> request
> >> >> is
> >> >> >> like
> >> >> >> >> >> this:
> >> >> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10,
> fetchOffset:
> >> >> 2, …
> >> >> >> }.
> >> >> >> >> The
> >> >> >> >> >> > divergingEpoch check in the cluster B queries the
> endOffset
> >> for
> >> >> >> >> leader
> >> >> >> >> >> > epoch 10 (in Partition#readRecords), and found the end
> >> offset
> >> >> for
> >> >> >> LE
> >> >> >> >> 10
> >> >> >> >> >> is
> >> >> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch
> >> validation
> >> >> >> >> passes
> >> >> >> >> >> and
> >> >> >> >> >> > returns offset 2 to cluster A. In short, relying on “last
> >> >> mirror
> >> >> >> >> offset”
> >> >> >> >> >> > cannot resolve the log diverge issue in unclean leader
> >> election
> >> >> >> case.
> >> >> >> >> >> >
> >> >> >> >> >> > If instead, we truncate until “last mirror epoch”: 0, the
> >> >> offset 0
> >> >> >> >> and 1
> >> >> >> >> >> > in cluster A will be truncated as expected and then log
> >> >> converge
> >> >> >> with
> >> >> >> >> >> > cluster B correctly in the end.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Also, about JR53.2:
> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
> >> >> brokers
> >> >> >> and
> >> >> >> >> >> > merges
> >> >> >> >> >> > the responses client-side."
> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
> >> partition.
> >> >> So
> >> >> >> >> sending
> >> >> >> >> >> > the request to every broker is wasteful. It's probably
> >> better
> >> >> to
> >> >> >> send
> >> >> >> >> >> the
> >> >> >> >> >> > request to a single broker, which can then contact the
> right
> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >
> >> >> >> >> >> > While working on this, I found we need to add one more
> >> tagged
> >> >> >> field
> >> >> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema
> >> because
> >> >> >> >> >> otherwise
> >> >> >> >> >> > the receiving broker doesn’t know if this is the request
> >> from
> >> >> >> client
> >> >> >> >> >> and it
> >> >> >> >> >> > needs to contact other brokers, or this is just a
> forwarded
> >> >> >> request.
> >> >> >> >> The
> >> >> >> >> >> > workflow is like this:
> >> >> >> >> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest
> >> to a
> >> >> >> >> >> > leastLoaded broker, which doesn’t contain the "Forwarded"
> >> >> field.
> >> >> >> >> >> > 2. The broker receives the request and sends
> >> >> >> >> >> DescribeClusterMirrorsRequest
> >> >> >> >> >> > with "Forwarded=true" to the responsible brokers and waits
> >> for
> >> >> the
> >> >> >> >> >> response
> >> >> >> >> >> > async.
> >> >> >> >> >> > 3. When all responses are received, merge the results and
> >> >> respond
> >> >> >> to
> >> >> >> >> the
> >> >> >> >> >> > client.
> >> >> >> >> >> >
> >> >> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here because
> it
> >> >> >> doesn’t
> >> >> >> >> >> > contain the lag info in the coordinator.
> >> >> >> >> >> >
> >> >> >> >> >> > The alternative solution is the adminClient sends
> >> >> >> >> >> > DescribeClusterMirrorsRequest to the “responsible brokers
> >> only”
> >> >> >> (via
> >> >> >> >> >> > describeTopics), instead of ALL brokers, and merge the
> >> results
> >> >> in
> >> >> >> the
> >> >> >> >> >> > client side. This solution doesn’t need to add a new field
> >> in
> >> >> the
> >> >> >> >> >> > DescribeClusterMirrorsRequest and still can avoid
> >> unnecessary
> >> >> >> >> requests
> >> >> >> >> >> > sent.
> >> >> >> >> >> >
> >> >> >> >> >> > My preference is to keep the merge logic on the admin
> client
> >> >> side,
> >> >> >> >> which
> >> >> >> >> >> > saves the overhead in the broker side.
> >> >> >> >> >> > What do you think?
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Thank you,
> >> >> >> >> >> > Luke
> >> >> >> >> >> >
> >> >> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
> >> >> >> dev@kafka.apache.org
> >> >> >> >> >
> >> >> >> >> >> > wrote:
> >> >> >> >> >> >
> >> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >> >>
> >> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR31.3 It makes sense for the new partition leader to
> query
> >> >> the
> >> >> >> >> current
> >> >> >> >> >> >> mirror state from the local or remote
> MirrorMetadataManager
> >> >> that
> >> >> >> >> owns
> >> >> >> >> >> the
> >> >> >> >> >> >> partition's state. However, the wording in the KIP is
> >> >> confusing.
> >> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> mirror partition state cache should be populated only by
> >> one
> >> >> way,
> >> >> >> >> >> which is
> >> >> >> >> >> >> replaying __mirror_state partitions this broker leads. By
> >> >> saying
> >> >> >> >> "and
> >> >> >> >> >> by
> >> >> >> >> >> >> fetching state from remote coordinators via
> >> ReadMirrorStates
> >> >> >> RPCs.",
> >> >> >> >> >> it's
> >> >> >> >> >> >> as if the mirror partition state cache on each broker
> >> stores
> >> >> the
> >> >> >> >> state
> >> >> >> >> >> for
> >> >> >> >> >> >> all mirror partitions.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR36. I understand the intention now, but I still don't
> >> think
> >> >> we
> >> >> >> >> need
> >> >> >> >> >> >> MirrorLeaderEpoch. Let's
> >> >> >> >> >> >> assume the common case: no unclean leader election in the
> >> >> source
> >> >> >> >> >> cluster
> >> >> >> >> >> >> during mirroring. Mirroring only fetches the committed
> data
> >> >> from
> >> >> >> the
> >> >> >> >> >> >> source, which never changes if obtained a second time. In
> >> your
> >> >> >> >> example
> >> >> >> >> >> >> image, follower broker 1 in the destination cluster can
> >> never
> >> >> >> obtain
> >> >> >> >> >> the
> >> >> >> >> >> >> record B(0) because it's never committed in the source
> and
> >> >> thus
> >> >> >> is
> >> >> >> >> >> never
> >> >> >> >> >> >> exposed to the destination leader. It's possible for
> >> broker 1
> >> >> to
> >> >> >> >> first
> >> >> >> >> >> get
> >> >> >> >> >> >> D(1) which is truncated later when broker 0 follows a new
> >> >> leader.
> >> >> >> >> But
> >> >> >> >> >> the
> >> >> >> >> >> >> new leader will refetch the same D(1) from the source
> >> cluster.
> >> >> >> So,
> >> >> >> >> >> it's ok
> >> >> >> >> >> >> for broker 1 to keep D(1).
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR51. Sounds good. Why do we choose to store the last
> >> mirror
> >> >> >> epoch
> >> >> >> >> >> instead
> >> >> >> >> >> >> of a last mirror offset?
> >> >> >> >> >> >>
> >> >> >> >> >> >> Jun
> >> >> >> >> >> >>
> >> >> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <
> >> showuon@gmail.com>
> >> >> >> wrote:
> >> >> >> >> >> >>
> >> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Thanks for the great comments again.
> >> >> >> >> >> >> > Answering your question below.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror partition
> >> >> cache
> >> >> >> >> lives
> >> >> >> >> >> in
> >> >> >> >> >> >> > MirrorMetadataManager. It is populated in two ways: by
> >> >> >> replaying
> >> >> >> >> >> >> > __mirror_state partitions this broker leads, and by
> >> fetching
> >> >> >> state
> >> >> >> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> >> >> >> > Hmm, a given broker only caches the mirror partition
> >> state
> >> >> for
> >> >> >> >> >> >> partitions
> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted on this
> >> >> broker,
> >> >> >> >> >> right?
> >> >> >> >> >> >> Why
> >> >> >> >> >> >> > does it need to fetch the state from other remote
> >> >> coordinators?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > a given broker only caches the mirror partition state
> >> for
> >> >> >> >> >> partitions
> >> >> >> >> >> >> > hashed into the __mirror_state partition hosted on this
> >> >> broker,
> >> >> >> >> >> right?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Right.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > Why does it need to fetch the state from other remote
> >> >> >> >> coordinators?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Because the data partition leader needs to know the
> >> current
> >> >> >> mirror
> >> >> >> >> >> state
> >> >> >> >> >> >> > of the partitions it leads. On a leadership change, the
> >> >> >> metadata
> >> >> >> >> >> delta
> >> >> >> >> >> >> is
> >> >> >> >> >> >> > published to the MirrorMetadataManager, which queries
> the
> >> >> >> current
> >> >> >> >> >> mirror
> >> >> >> >> >> >> > state for its led partitions from the coordinator
> (local
> >> or
> >> >> >> >> remote),
> >> >> >> >> >> and
> >> >> >> >> >> >> > processes tasks based on that state. This is the reason
> >> why
> >> >> it
> >> >> >> >> >> fetches
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > state from other remote coordinator. This part of cache
> >> >> will be
> >> >> >> >> >> cleared
> >> >> >> >> >> >> > after the leadership change.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR36. I am still not quite sure why we need
> >> >> >> MirrorLeaderEpoch in
> >> >> >> >> >> the
> >> >> >> >> >> >> > Fetch
> >> >> >> >> >> >> > request. In the current Fetch request, in addition to
> >> >> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> >> > it has another field LastFetchedEpoch, which captures
> the
> >> >> last
> >> >> >> >> leader
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> > in the data and is used to resolve the inconsistency
> >> between
> >> >> >> the
> >> >> >> >> >> >> follower
> >> >> >> >> >> >> > and the leader. Does it achieve the same purpose as
> >> >> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to
> do
> >> the
> >> >> >> >> >> validation
> >> >> >> >> >> >> to
> >> >> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any
> fetched
> >> >> >> batches
> >> >> >> >> with
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> > higher than the LastFetchedEpoch, it means the follower
> >> can
> >> >> >> never
> >> >> >> >> >> accept
> >> >> >> >> >> >> > new appended batches with higher epoch in the leader
> >> node.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2
> >> purposes in
> >> >> >> the
> >> >> >> >> >> fetch
> >> >> >> >> >> >> > API:
> >> >> >> >> >> >> > 1. The receiver (leader node) uses it to validate it
> >> matches
> >> >> >> the
> >> >> >> >> >> leader
> >> >> >> >> >> >> > epoch in the leader node
> >> >> >> >> >> >> > 2. The sender (follower node) uses it to validate the
> >> epoch
> >> >> in
> >> >> >> the
> >> >> >> >> >> >> fetched
> >> >> >> >> >> >> > batches are all <= this value.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still
> serving
> >> for
> >> >> >> >> >> followers
> >> >> >> >> >> >> in
> >> >> >> >> >> >> > the destination cluster. For purpose (2), my
> >> understanding
> >> >> of
> >> >> >> the
> >> >> >> >> >> >> > validation for KAFKA-18723 is like: the follower knows
> >> the
> >> >> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader
> node
> >> >> >> should
> >> >> >> >> >> >> contain in
> >> >> >> >> >> >> > the log batches when fetch request is sent. Any log
> batch
> >> >> >> beyond
> >> >> >> >> this
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> > in the fetch response means the follower’s metadata is
> >> >> stale.
> >> >> >> The
> >> >> >> >> >> >> batches
> >> >> >> >> >> >> > need to be rejected and follower needs to update
> metadata
> >> >> for
> >> >> >> next
> >> >> >> >> >> >> fetch.
> >> >> >> >> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch
> >> >> serves
> >> >> >> >> this
> >> >> >> >> >> >> purpose
> >> >> >> >> >> >> > to let the followers know the highest epoch in the
> leader
> >> >> node
> >> >> >> >> log in
> >> >> >> >> >> >> each
> >> >> >> >> >> >> > fetch response. Once the info is stale in the follower
> >> >> side, it
> >> >> >> >> >> updates
> >> >> >> >> >> >> it
> >> >> >> >> >> >> > from fetch response, and send next fetch request with
> >> >> correct
> >> >> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is
> >> >> possible
> >> >> >> >> like
> >> >> >> >> >> >> > KAFKA-18723.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > I tried not to add new field in Fetch API, but this is
> >> the
> >> >> best
> >> >> >> >> >> >> solution I
> >> >> >> >> >> >> > can think of.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR48. "The NumPartitions field is needed so the
> >> controller
> >> >> >> can
> >> >> >> >> >> create
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > destination topic if it does not already exist. When a
> >> >> topic is
> >> >> >> >> being
> >> >> >> >> >> >> > mirrored for the first time, the destination cluster
> has
> >> no
> >> >> >> >> knowledge
> >> >> >> >> >> >> > of the source topic's partition count, so the request
> >> must
> >> >> >> carry
> >> >> >> >> it."
> >> >> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is started,
> >> the
> >> >> >> source
> >> >> >> >> >> topic
> >> >> >> >> >> >> > may not exist yet. When the source topic is created
> >> later,
> >> >> we
> >> >> >> can
> >> >> >> >> >> start
> >> >> >> >> >> >> > mirroring without requiring the user to provide the
> >> number
> >> >> of
> >> >> >> >> >> >> partitions.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > When the source topic is created later, we can start
> >> >> >> mirroring
> >> >> >> >> >> without
> >> >> >> >> >> >> > requiring the user to provide the number of partitions.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > This is because this happens in periodical sync up in
> the
> >> >> >> >> destination
> >> >> >> >> >> >> > cluster, and the numPartitions and other topic info in
> >> the
> >> >> >> source
> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> > is already retrieved during the sync up process. Also,
> if
> >> >> users
> >> >> >> >> are
> >> >> >> >> >> >> > creating mirror topics via admin API, this info can
> also
> >> be
> >> >> >> >> retrieved
> >> >> >> >> >> >> > inside the admin client if not provided.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly need
> the
> >> >> last
> >> >> >> >> mirror
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> > for truncation when starting a mirror. When a mirror is
> >> >> >> stopped at
> >> >> >> >> >> the
> >> >> >> >> >> >> > source, we already bump up the leader epoch and write
> >> some
> >> >> >> >> additional
> >> >> >> >> >> >> > records for cleanup. It seems that we can just use the
> >> >> >> >> DivergingEpoch
> >> >> >> >> >> >> > returned in the initial mirror Fetch request to detect
> >> the
> >> >> >> >> diverging
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> > and use that for truncation at the mirror destination.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > That’s a good question. I thought about this before but
> >> >> >> >> >> unfortunately it
> >> >> >> >> >> >> > doesn’t work. The reason we need the last mirror epoch
> >> >> record
> >> >> >> is
> >> >> >> >> >> because
> >> >> >> >> >> >> > the destination cluster might not completely sync with
> >> the
> >> >> >> source
> >> >> >> >> >> >> cluster
> >> >> >> >> >> >> > before STOPPED. And this un-mirrored data might confuse
> >> the
> >> >> >> >> >> replication
> >> >> >> >> >> >> > protocol when reverse mirror. For example,
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
> >> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> >> >> >> >> > Offset 2: from A (LE: 10)
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > While cluster B mirrors until offset 1, the cluster A
> >> >> crashes
> >> >> >> and
> >> >> >> >> >> >> failover
> >> >> >> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
> >> >> >> >> MIRROR_PID_RESET
> >> >> >> >> >> >> with
> >> >> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from
> >> >> cluster B,
> >> >> >> >> the
> >> >> >> >> >> log
> >> >> >> >> >> >> > will be like this:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> >> >> > foo-0 leader log: foo-0 leader log:
> >> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE:
> >> 10)
> >> >> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > As you can see, cluster A (new destination cluster)
> >> doesn’t
> >> >> >> know
> >> >> >> >> the
> >> >> >> >> >> >> > offset 2 is not belonging to the cluster B, which
> causes
> >> >> >> >> inconsistent
> >> >> >> >> >> >> logs.
> >> >> >> >> >> >> > So like what we described in the KIP, the last mirror
> >> epoch
> >> >> is
> >> >> >> the
> >> >> >> >> >> >> > synchronization point between source and destination.
> Any
> >> >> data
> >> >> >> >> beyond
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > point is treated as unknown to the source cluster and
> >> >> should be
> >> >> >> >> >> >> truncated.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR52. Stop Mirror:
> >> >> >> >> >> >> > > JR52.1 "3. If patterns are provided, the controller
> >> >> removes
> >> >> >> >> >> matching
> >> >> >> >> >> >> > entries from mirror.topics.include or adds them to
> >> >> >> >> >> >> mirror.topics.exclude on
> >> >> >> >> >> >> > the CLUSTER_MIRROR resource."
> >> >> >> >> >> >> > Why is it 'or' and not 'and'?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> > discovers new source topics matching the persisted
> >> >> >> include/exclude
> >> >> >> >> >> >> patterns
> >> >> >> >> >> >> > and repeats steps 3 through 7 for each."
> >> >> >> >> >> >> > The broker will send a create topic request to the
> >> >> controller.
> >> >> >> How
> >> >> >> >> >> does
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > controller associate it with the mirror name to create
> a
> >> >> >> >> >> >> > `MirrorTopicStateChangeRecord`?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > No, we don’t send a create topic request to the
> >> controller
> >> >> >> here.
> >> >> >> >> >> >> Actually
> >> >> >> >> >> >> > we send a StartMirrorTopics request to the controller,
> >> which
> >> >> >> >> contains
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > mirror name and topic metadata like NumPartitions,
> >> topicId…
> >> >> >> etc.
> >> >> >> >> So
> >> >> >> >> >> the
> >> >> >> >> >> >> > controller can create the non-existed topic as
> explained
> >> in
> >> >> >> JR48,
> >> >> >> >> and
> >> >> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR53. Describe Mirrors:
> >> >> >> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors
> >> request
> >> >> >> with
> >> >> >> >> >> >> optional
> >> >> >> >> >> >> > mirror names (empty means all mirrors)."
> >> >> >> >> >> >> > In metadata request, we use a null arrary to represent
> >> all
> >> >> >> >> topics. It
> >> >> >> >> >> >> would
> >> >> >> >> >> >> > be useful to be consistent here.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > OK, KIP updated.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to
> all
> >> >> >> brokers
> >> >> >> >> and
> >> >> >> >> >> >> > merges
> >> >> >> >> >> >> > the responses client-side."
> >> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
> >> >> partition.
> >> >> >> So
> >> >> >> >> >> sending
> >> >> >> >> >> >> > the request to every broker is wasteful. It's probably
> >> >> better
> >> >> >> to
> >> >> >> >> send
> >> >> >> >> >> >> the
> >> >> >> >> >> >> > request to a single broker, which can then contact the
> >> right
> >> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Agree, KIP updated.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are
> >> >> >> MIRRORING,
> >> >> >> >> >> >> PAUSED,
> >> >> >> >> >> >> > STOPPED the only validate states? The state table has
> >> other
> >> >> >> states
> >> >> >> >> >> too.
> >> >> >> >> >> >> > Ditto for MirrorPartitionState.
> >> >> >> >> >> >> > The DesiredState field in MirrorTopicStateChangeRecord
> >> is to
> >> >> >> >> define
> >> >> >> >> >> the
> >> >> >> >> >> >> > target state the user wants. And the desired state
> users
> >> can
> >> >> >> >> assign
> >> >> >> >> >> is
> >> >> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired
> >> >> state
> >> >> >> as
> >> >> >> >> >> >> STOPPING
> >> >> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
> >> >> intermediate
> >> >> >> >> >> states.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > About MirrorPartitionState record, it’ll record any
> >> state in
> >> >> >> the
> >> >> >> >> >> state
> >> >> >> >> >> >> > table that the current partition is. This is the real
> >> state
> >> >> for
> >> >> >> >> each
> >> >> >> >> >> >> > partition. It’s like in the kubernetes world, when we
> >> change
> >> >> >> the
> >> >> >> >> >> replica
> >> >> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired
> >> state), it
> >> >> >> won’t
> >> >> >> >> >> >> increase
> >> >> >> >> >> >> > to 5 immediately. The current replica
> >> (MirrorPartitionState
> >> >> >> >> record)
> >> >> >> >> >> is
> >> >> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Thank you,
> >> >> >> >> >> >> > Luke
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
> >> >> >> >> dev@kafka.apache.org
> >> >> >> >> >> >
> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> Hi, Fede,
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> Thanks for the reply. A few more comments.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror partition
> >> cache
> >> >> >> >> lives in
> >> >> >> >> >> >> >> MirrorMetadataManager. It is populated in two ways: by
> >> >> >> replaying
> >> >> >> >> >> >> >> __mirror_state partitions this broker leads, and by
> >> >> fetching
> >> >> >> >> state
> >> >> >> >> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> >> >> >> >> Hmm, a given broker only caches the mirror partition
> >> state
> >> >> for
> >> >> >> >> >> >> partitions
> >> >> >> >> >> >> >> hashed into the __mirror_state partition hosted on
> this
> >> >> >> broker,
> >> >> >> >> >> right?
> >> >> >> >> >> >> Why
> >> >> >> >> >> >> >> does it need to fetch the state from other remote
> >> >> >> coordinators?
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR36. I am still not quite sure why we need
> >> >> MirrorLeaderEpoch
> >> >> >> in
> >> >> >> >> the
> >> >> >> >> >> >> Fetch
> >> >> >> >> >> >> >> request. In the current Fetch request, in addition to
> >> >> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> >> >> it has another field LastFetchedEpoch, which captures
> >> the
> >> >> last
> >> >> >> >> >> leader
> >> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> in the data and is used to resolve the inconsistency
> >> >> between
> >> >> >> the
> >> >> >> >> >> >> follower
> >> >> >> >> >> >> >> and the leader. Does it achieve the same purpose as
> >> >> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR48. "The NumPartitions field is needed so the
> >> controller
> >> >> can
> >> >> >> >> >> create
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> destination topic if it does not already exist. When a
> >> >> topic
> >> >> >> is
> >> >> >> >> >> being
> >> >> >> >> >> >> >> mirrored for the first time, the destination cluster
> >> has no
> >> >> >> >> >> knowledge
> >> >> >> >> >> >> >> of the source topic's partition count, so the request
> >> must
> >> >> >> carry
> >> >> >> >> >> it."
> >> >> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is
> started,
> >> the
> >> >> >> >> source
> >> >> >> >> >> >> topic
> >> >> >> >> >> >> >> may not exist yet. When the source topic is created
> >> later,
> >> >> we
> >> >> >> can
> >> >> >> >> >> start
> >> >> >> >> >> >> >> mirroring without requiring the user to provide the
> >> number
> >> >> of
> >> >> >> >> >> >> partitions.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need
> the
> >> >> last
> >> >> >> >> mirror
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> for truncation when starting a mirror. When a mirror
> is
> >> >> >> stopped
> >> >> >> >> at
> >> >> >> >> >> the
> >> >> >> >> >> >> >> source, we already bump up the leader epoch and write
> >> some
> >> >> >> >> >> additional
> >> >> >> >> >> >> >> records for cleanup. It seems that we can just use the
> >> >> >> >> >> DivergingEpoch
> >> >> >> >> >> >> >> returned in the initial mirror Fetch request to detect
> >> the
> >> >> >> >> diverging
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> >> and use that for truncation at the mirror destination.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR52. Stop Mirror:
> >> >> >> >> >> >> >> JR52.1 "3. If patterns are provided, the controller
> >> removes
> >> >> >> >> matching
> >> >> >> >> >> >> >> entries from mirror.topics.include or adds them to
> >> >> >> >> >> >> mirror.topics.exclude
> >> >> >> >> >> >> >> on
> >> >> >> >> >> >> >> the CLUSTER_MIRROR resource."
> >> >> >> >> >> >> >> Why is it 'or' and not 'and'?
> >> >> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> >> discovers new source topics matching the persisted
> >> >> >> >> include/exclude
> >> >> >> >> >> >> >> patterns
> >> >> >> >> >> >> >> and repeats steps 3 through 7 for each."
> >> >> >> >> >> >> >> The broker will send a create topic request to the
> >> >> controller.
> >> >> >> >> How
> >> >> >> >> >> does
> >> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> controller associate it with the mirror name to
> create a
> >> >> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR53. Describe Mirrors:
> >> >> >> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors
> >> request
> >> >> >> with
> >> >> >> >> >> >> optional
> >> >> >> >> >> >> >> mirror names (empty means all mirrors)."
> >> >> >> >> >> >> >> In metadata request, we use a null arrary to represent
> >> all
> >> >> >> >> topics.
> >> >> >> >> >> It
> >> >> >> >> >> >> >> would
> >> >> >> >> >> >> >> be useful to be consistent here.
> >> >> >> >> >> >> >> JR53.2 "2. The admin client fans out the request to
> all
> >> >> >> brokers
> >> >> >> >> and
> >> >> >> >> >> >> merges
> >> >> >> >> >> >> >> the responses client-side."
> >> >> >> >> >> >> >> This is not ideal. Not every broker hosts a mirror
> >> >> partition.
> >> >> >> So
> >> >> >> >> >> >> sending
> >> >> >> >> >> >> >> the request to every broker is wasteful. It's probably
> >> >> better
> >> >> >> to
> >> >> >> >> >> send
> >> >> >> >> >> >> the
> >> >> >> >> >> >> >> request to a single broker, which can then contact the
> >> >> right
> >> >> >> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are
> >> >> >> MIRRORING,
> >> >> >> >> >> PAUSED,
> >> >> >> >> >> >> >> STOPPED the only validate states? The state table has
> >> other
> >> >> >> >> states
> >> >> >> >> >> too.
> >> >> >> >> >> >> >> Ditto for MirrorPartitionState.
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> Jun
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
> >> >> >> >> >> fedevaleri@gmail.com>
> >> >> >> >> >> >> >> wrote:
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >> > Hi Andrew, yes, validation is called at the start of
> >> >> >> >> createTopic.
> >> >> >> >> >> If
> >> >> >> >> >> >> >> > the topic ID is invalid, the request fails with
> >> >> >> >> INVALID_REQUEST.
> >> >> >> >> >> If
> >> >> >> >> >> >> >> > the topic ID is already used by a different topic
> >> name,
> >> >> it
> >> >> >> >> fails
> >> >> >> >> >> with
> >> >> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the
> >> individual
> >> >> >> topic
> >> >> >> >> in
> >> >> >> >> >> the
> >> >> >> >> >> >> >> > batch fails; other topics in the same CreateTopics
> >> >> request
> >> >> >> >> >> continue
> >> >> >> >> >> >> >> > processing normally. Added this information to the
> >> KIP.
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
> >> >> >> >> >> >> >> schofielaj@apache.org>
> >> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> > > Hi Fede,
> >> >> >> >> >> >> >> > > One more small question.
> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> > > AS37: If the controller fails validation for the
> >> >> >> >> >> MirrorInfo.TopicId
> >> >> >> >> >> >> >> > added to the CreateTopic request, what does it do? I
> >> >> suspect
> >> >> >> >> that
> >> >> >> >> >> it
> >> >> >> >> >> >> >> fails
> >> >> >> >> >> >> >> > the request with a particular error code.
> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> > > Thanks,
> >> >> >> >> >> >> >> > > Andrew
> >> >> >> >> >> >> >> > >
> >> >> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> >> >> >> >> >> >> >> > > > Hi all,
> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> > > > We updated the KIP to include Coordinator
> Runtime
> >> >> >> >> >> configurations
> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> > > > the coordinator state transition validation
> >> >> paragraph.
> >> >> >> The
> >> >> >> >> >> latter
> >> >> >> >> >> >> >> > > > covers the per-partition epoch fencing mechanism
> >> >> (leader
> >> >> >> >> epoch
> >> >> >> >> >> >> and
> >> >> >> >> >> >> >> > > > state epoch) that complements the existing
> >> per-topic
> >> >> >> >> >> StateOffset
> >> >> >> >> >> >> >> > > > fencing for lifecycle operations.
> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> > > > Thanks,
> >> >> >> >> >> >> >> > > > Fede
> >> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >> >
> >> >> >> >> >> >> >>
> >> >> >> >> >> >> >
> >> >> >> >> >> >>
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply.
JR80. In that case, it seems that there is no need for
AdminClient.startMirrorTopics() to take topics. The adminClient only uses
it to register the included topics, which can be passed in as
includePatterns (specific names or regex). The broker will use
StartMirrorTopicsRequest to set topic and topicId, but it doesn't go
through the AdminClient.
JR82. If “IncludeMirrorOffset” is disabled,
DescribeClusterMirrorsResponse.SourceOffset is still returned in the
response and what is its value in this case?
JR88. Why is partition-load-time a Meter instead of a Gauge?
JR89. Why is thread-idle-ratio a Histogram? Typically, it's a Rate.
JR90. MetadataRefreshError : Is that cumulative? If so, it should be a
Meter.
Jun
On Fri, Aug 14, 2026 at 4:32 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the good comments,
> Answering them below.
>
> > JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
> supports regex in the ClusterMirrorComamnd, but we don’t need to add the
> “patterns”."
> So, in this case, we use topics to support both specific topics and regex.
> However, startMirrorTopicsOptions uses topics for specific topics and
> includePatterns for regex. Could we make them consistent?
>
>
> The purpose of the “includePatterns” field is for the
> “mirror.topics.include” update, so that if there is any newly created topic
> in the source cluster that matches the regex, we can create it. And the
> “Topics” field in the StartMirrorTopicsRequest is to specify the topicId
> and numPartitions in the source cluster. So I think “includePatterns” and
> “Topics” both are necessary in the StartMirrorTopicsRequest. On the other
> hand, in PauseMirrorTopics, because it won’t change the
> “mirror.topics.include”/”mirror.topics.exclude” configs, we don’t need to
> provide the regex in the request.
>
>
>
> > JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also -1
> if
> it's not included in the request?
>
> If “IncludeMirrorOffset” is disabled, we won’t query the mirror partition
> leader, so we can’t get the sourceOffset.
>
> I assume you want to ask `if it’s still -1 if not actively mirroring,
> while “IncludeMirrorOffset” is enabled`. That’s a good question. I think
> our purpose of this command is to show the lag to users. So if the state is
> not MIRRORING (ex: STOPPED/PAUSED), we think it’s not helpful to users to
> show only the sourceOffset. Of course users can use `kafka-get-offsets.sh`
> to get the offset if they really want to know.
>
>
> > JR83. Start Mirror Topics:
> > JR83.1 Should the controller verify the expected state for the topic?
>
> As replied in JR84, the desired state validation should happen in the
> broker. And the controller validates the StateOffset. I’ve updated the
> workflow to make it clear.
>
> > JR83.2 "d. truncates the log accordingly,"
> Could you add a description of the truncation offset?
>
> OK, added one more sentence:
> d. based on the last mirror epoch response in (b), the broker translates
> the epoch into the offset by querying the leaderEpochCache
> (endOffsetForEpoch), and truncates the log accordingly, and waits for all
> ISR members…
>
>
> > JR84. Stop Mirror Topics: "The controller verifies that the desired state
> of all topics are in MIRRORING or PAUSED state."
> Should the broker verfy the topic level state? I thought the controller
> will use StateOffset in the request to make sure the validation on the
> broker is stil valid. If the controller performs the verification, is
> StateOffset still useful?
>
> Oh, you’re right. We’re doing duplicated work here. Yes, the broker
> already verifies the “desired state” of all topics are in MIRRORING or
> PAUSED state, as well as the “actual state” is in MIRRORING or PAUSED
> state. If all validation passes, we’ll forward it to the controller. So the
> controller only needs to validate the StateOffset to make sure the
> validation on the broker is still valid. So, yes, we don’t need the desired
> state validation on the controller side. I’ve updated the KIP to make it
> clear.
>
> One more thing to add is that, I’ve thought about not adding StateOffset,
> and relying on the desired state validation to make sure we’re safe to do
> the deletion or other operations. The problem is that, even if the broker
> sees the desired state for topic A is STOPPED, and then the controller also
> sees the desired state is STOPPED, it still cannot make sure the
> controller’s STOPPED state equals the broker’s STOPPED state. For example:
> 1. Broker sees STOPPED, then forward to controller
> 2. User changes the desired state to MIRRORING
> 3. User changes the desired state to STOPPED
> 4. Controller sees the desired state is STOPPED
> So I think StateOffset is required.
>
> Thanks for catching it BTW.
>
>
> > JR85. There are still a few references to includeMirrorLags. They should
> be
> IncludeMirrorOffset.
>
> Sorry, just updated.
>
>
> > JR86. Metrics:
> > JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
>
> KIP is updated. I also group the metrics into new added metrics and other
> inherited metrics.
>
> > JR86.2 Could you describe a bit more what MinFetchRate represents?
>
> It means the lowest fetch request rate among all of this mirror fetcher
> manager's fetcher threads. It inherits from the AbstractFetcherManager’s
> metric.
>
>
> > JR87. Version Compatibility: "For migration or DR use cases where
> failback
> is not required, this proposal supports source brokers up to version 2.1
> included"
> This KIP needs topicId and KIP-516 was introduced later than 2.1.
> Yes, when mirroring from a cluster without topic id support, we will
> assign a random topic ID in the destination cluster and then fetch (without
> topic id in the fetch request using the old version of Fetch API) as usual.
> Of course this is for the one-way mirroring use case. It can’t start the
> reverse mirror(failback) under this setup. You can check the “Compatibility
> Matrix” section for more detail.
>
>
> Thank you,
> Luke
>
> On Fri, Aug 14, 2026 at 7:09 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Luke,
>>
>> Thanks for the reply.
>>
>> JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
>> supports regex in the ClusterMirrorComamnd, but we don’t need to add the
>> “patterns”."
>> So, in this case, we use topics to support both specific topics and regex.
>> However, startMirrorTopicsOptions uses topics for specific topics and
>> includePatterns for regex. Could we make them consistent?
>>
>> JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also -1 if
>> it's not included in the request?
>>
>> JR83. Start Mirror Topics:
>> JR83.1 Should the controller verify the expected state for the topic?
>> JR83.2 "d. truncates the log accordingly,"
>> Could you add a description of the truncation offset?
>>
>> JR84. Stop Mirror Topics: "The controller verifies that the desired state
>> of all topics are in MIRRORING or PAUSED state."
>> Should the broker verfy the topic level state? I thought the controller
>> will use StateOffset in the request to make sure the validation on the
>> broker is stil valid. If the controller performs the verification, is
>> StateOffset still useful?
>>
>> JR85. There are still a few references to includeMirrorLags. They should
>> be
>> IncludeMirrorOffset.
>>
>> JR86. Metrics:
>> JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
>> JR86.2 Could you describe a bit more what MinFetchRate represents?
>>
>> JR87. Version Compatibility: "For migration or DR use cases where failback
>> is not required, this proposal supports source brokers up to version 2.1
>> included"
>> This KIP needs topicId and KIP-516 was introduced later than 2.1.
>>
>> Jun
>>
>>
>> On Thu, Aug 13, 2026 at 5:14 AM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for the good comments.
>> > Answering them below.
>> >
>> > > JR59.1 Could you add the RPC to the workflow? The broker first needs
>> to
>> > issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
>> > mirror name and then issue DescribeClusterMirrors to get the LME.
>> >
>> > Yes, you’re right. The workflow is updated to make them clear.
>> >
>> > > JR71. mirror.support.unclean.leader.election : The name is confusing.
>> > It's
>> > also used in LOG_TRUNCATION, which is not related to unclean leader
>> > election. Could we use a better name?
>> >
>> > I personally think the name should be fine because even though we use in
>> > “LOG_TRUNCATION” state, the config is to allow us to know if users want
>> to
>> > support unclean leader election, if so, we wait for all replicas to
>> catch
>> > up. If we want to explicitly show the intention of the config, how about
>> > “mirror.full.replica.convergence.required”? I’m open to other
>> suggestions.
>> >
>> >
>> > > JR72. DescribeClusterMirrors
>> > > JR72.1 A user may specify just the topic name when describing a
>> mirror.
>> > Should we include the topic name in the request?
>> >
>> > Hmm.. we didn’t think about the use case. But yes, if users only care
>> > about the state of a topic, they don’t need the info for other topics.
>> I’ve
>> > combined with JR72.2, added “TopicPartitions” filed into
>> > DescribeClusterMirrorsRequest, and by default is null, which means all
>> > topic partitions under the requested mirrors will be returned. If set,
>> > we’ll query the mirrors and return the topic partitions that users
>> > expected. KIP is updated.
>> >
>> >
>> > > JR72.2 LastMirrorEpochLookups is a bit weird since the response
>> returns
>> > either mirror state or mirror lag and LastMirrorEpochLookups doesn't
>> match
>> > either.
>> >
>> > Yes, after the recent refactor, I think the LastMirrorEpochLookups can
>> be
>> > merged into one level up, combining with the topic name filter in above
>> > comment, which will allow users to query down to the partition level.
>> The
>> > LastMirrorEpochLookups will query along with normal mirror state by the
>> > readMirrorState from the coordinator. The difference is that after
>> > receiving the response, it will filter out the mirrors that don’t match
>> the
>> > requested sourceClusterId, and then add the lastMirrorEpoch into
>> > DescribeClusterMirrorResponse. KIP is updated.
>> >
>> >
>> > > JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >
>> > Right. Added.
>> >
>> >
>> > > JR72.4 "includeMirrorLags: source offset, destination offset, lag"
>> > DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
>> > should be omitted.
>> >
>> > Sorry, I removed the “SourceOffset” and “DestinationOffset” fields from
>> > DescribeClusterMirrorsResponse because I added them into
>> ReadMirrorOffsets
>> > (ex-ReadMirrorLags) API. But I forgot we still need them in the
>> > DescribeClusterMirrorsResponse so that the adminClient can receive the
>> > results. KIP is updated.
>> >
>> >
>> > > JR73. MirrorPartitionState record, LastMirrorEpochs record and
>> > MirrorPartitionStateRequest: The values for type and name are incorrect.
>> >
>> > Ah, that’s my bad. Fixed.
>> >
>> > > JR74. ListClusterMirrorsRequest
>> > > JR74.1 When is DesiredMirrorStateFilter used? There is no option to
>> > specify
>> > the desired state in the mirror-topic command.
>> >
>> > Yes, referred to kafka-consumer-groups.sh, I added the `--state` option
>> to
>> > allow users to set the state when listMirror.
>> >
>> > > JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>> >
>> > Yes, KIP updated.
>> >
>> > > JR75. ReadMirrorLags
>> > > JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
>> > include lag?
>> >
>> > Indeed. Updated.
>> >
>> > > JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >
>> > Yes, added
>> >
>> > > JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to
>> be
>> > consistent.
>> >
>> > Thanks for pointing it out. KIP is updated.
>> >
>> > > JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >
>> > Yes, we should validate the mirror name before writing the data. KIP is
>> > updated.
>> >
>> > > JR77. Delete Mirror: "3. The controller verifies that no topics are in
>> > MIRRORING or PAUSED state."
>> > Is this necessary since step 2 has verified all partitions are in
>> STOPPED
>> > state, which can only happen if the topics are in STOPPED state.
>> >
>> > It’s true, but there could be some delay between “actual state” and
>> > “desired state”. In step (2), we verify all partitions’ “actual state”
>> are
>> > in STOPPED state. But the current desired state could be MIRRORING, it’s
>> > just that the mirror partition leaders haven’t got the metadata update
>> yet.
>> > So, the controller still needs to verify the desired state is not in
>> > MIRRORING or PAUSED. I found we didn’t specify that the controller is
>> > actually verifying the “desired state”, not “actual state”. And this
>> > controller’s desired state validation also applies in
>> > startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
>> > I’ve updated the KIP to make it clear.
>> >
>> > > JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
>> > bumped,
>> > ABORT markers are appended for all ongoing transactions, last mirror
>> epochs
>> > are persisted, and a MIRROR_PID_RESET record appended."
>> > This describes the state for STOPPED, not STOPPING.
>> >
>> > No, these are the tasks during STOPPING state, like what we described in
>> > “Stop Mirror Topics” workflow. If we do these in STOPPED state, we can’t
>> > safely move the readonly topic into writable because not all required
>> tasks
>> > are done. I think I know why it confuses you, it’s because all the tasks
>> > are in the completed state. I’ve rephrased it to make it clear.
>> >
>> >
>> > > JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
>> > timeout field?
>> >
>> > Make sense. Added into `DescribeClusterMirrors` request, as well as
>> >
>> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
>> > because we will query the partition states from coordinator for state
>> > validation in these APIs.
>> >
>> >
>> > > JR80. PauseMirrorTopicsOptions: Should it support patterns? The
>> command
>> > line seems to suppor it.
>> >
>> > Sorry, it’s a little confusing here. The “pattern” in
>> > startMirrorTopicsOptions or stopMirrorTopicsOptions is to update the
>> > “mirror.topics.include” and “mirror.topics.exclude” configuration for
>> topic
>> > pattern discovery periodically. For the PauseMirrorTopics and
>> > ResumeMirrorTopics in the clusterMirrorCommand, users can provide regex
>> in
>> > the “--topics” option, and the clusterMirrorCommand will listTopic in
>> the
>> > destination cluster and match with the patterns users provided, and then
>> > feeds the matched topics into the PauseMirrorTopics/ResumeMirrorTopics
>> > requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics
>> supports
>> > regex in the ClusterMirrorComamnd, but we don’t need to add the
>> “patterns”.
>> >
>> >
>> > > JR81. AdminClient
>> > > JR81.1 ListClusterMirrors: We need to add the option to specify
>> different
>> > filters supported in the request.
>> >
>> > We already have the `--mirror` option. Added `--state` and
>> > `--sourceClusterId` options to support the other filters.
>> >
>> > > JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
>> > request doesn't support this option. Could we add other options
>> supported
>> > in the request?
>> >
>> > Currently, we have “includeAuthorizedOperations” in DescribeCluster,
>> > DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
>> > DescribeStreamsGroups requests. But none of them include the option in
>> the
>> > tool to allow users to enable it. I think the reason is that we don’t
>> want
>> > to show this to users on the tool output. But having this field, we
>> allow
>> > the advanced users to invoke admin API with includeAuthorizedOperations
>> > enabled to get the results they want.
>> >
>> >
>> > > JR81.3 LeaderStateDescription: Should it be named
>> > MirrorPartitionStateDescription? Also, could we make the fields
>> consistent
>> > with those in the response?
>> >
>> > OK. KIP is updated.
>> >
>> >
>> > Thank you,
>> > Luke
>> >
>> > On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <de...@kafka.apache.org>
>> > wrote:
>> >
>> >> Hi, Luke,
>> >>
>> >> Thanks for the reply.
>> >>
>> >> JR59.1 Could you add the RPC to the workflow? The broker first needs to
>> >> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
>> >> mirror name and then issue DescribeClusterMirrors to get the LME.
>> >>
>> >> JR71. mirror.support.unclean.leader.election : The name is confusing.
>> It's
>> >> also used in LOG_TRUNCATION, which is not related to unclean leader
>> >> election. Could we use a better name?
>> >>
>> >> JR72. DescribeClusterMirrors
>> >> JR72.1 A user may specify just the topic name when describing a mirror.
>> >> Should we include the topic name in the request?
>> >> JR72.2 LastMirrorEpochLookups is a bit weird since the response returns
>> >> either mirror state or mirror lag and LastMirrorEpochLookups doesn't
>> match
>> >> either.
>> >> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> JR72.4 "includeMirrorLags: source offset, destination offset, lag"
>> >> DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
>> >> should be omitted.
>> >>
>> >> JR73. MirrorPartitionState record, LastMirrorEpochs record and
>> >> MirrorPartitionStateRequest: The values for type and name are
>> incorrect.
>> >>
>> >> JR74. ListClusterMirrorsRequest
>> >> JR74.1 When is DesiredMirrorStateFilter used? There is no option to
>> >> specify
>> >> the desired state in the mirror-topic command.
>> >> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>> >>
>> >> JR75. ReadMirrorLags
>> >> JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
>> >> include lag?
>> >> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >> JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to
>> be
>> >> consistent.
>> >>
>> >> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
>> >>
>> >> JR77. Delete Mirror: "3. The controller verifies that no topics are in
>> >> MIRRORING or PAUSED state."
>> >> Is this necessary since step 2 has verified all partitions are in
>> STOPPED
>> >> state, which can only happen if the topics are in STOPPED state.
>> >>
>> >> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
>> >> bumped,
>> >> ABORT markers are appended for all ongoing transactions, last mirror
>> >> epochs
>> >> are persisted, and a MIRROR_PID_RESET record appended."
>> >> This describes the state for STOPPED, not STOPPING.
>> >>
>> >> JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
>> >> timeout field?
>> >>
>> >> JR80. PauseMirrorTopicsOptions: Should it support patterns? The command
>> >> line seems to suppor it.
>> >>
>> >> JR81. AdminClient
>> >> JR81.1 ListClusterMirrors: We need to add the option to specify
>> different
>> >> filters supported in the request.
>> >> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
>> >> request doesn't support this option. Could we add other options
>> supported
>> >> in the request?
>> >> JR81.3 LeaderStateDescription: Should it be named
>> >> MirrorPartitionStateDescription? Also, could we make the fields
>> consistent
>> >> with those in the response?
>> >>
>> >> Jun
>> >>
>> >> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com> wrote:
>> >>
>> >> > Hi Jun,
>> >> >
>> >> > Thanks for the good comments again.
>> >> > Answering them below.
>> >> >
>> >> > > JR51. "ULE_LOG_TRUNCATION When
>> mirror.support.unclean.leader.election
>> >> is
>> >> > enabled and the destination leader has log truncation, the state will
>> >> move
>> >> > from MIRRORING to this state."
>> >> > This still suggests that divergeEpoch is only handled if
>> >> > mirror.support.unclean.leader.election
>> >> > is enabled.
>> >> >
>> >> >
>> >> > OK, KIP is updated to make it clear.
>> >> >
>> >> > > JR55. Should the mirror state and the mirror lags always be exposed
>> >> > together? If so, we can remove IncludeMirrorState and
>> IncludeMirrorLags
>> >> > from DescribeClusterMirrorsRequest. If not, it's better to use two
>> >> separate
>> >> > RPCs.
>> >> >
>> >> >
>> >> > No, we sometimes only need MirrorState, and sometimes need both. So
>> I’ll
>> >> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
>> >> > DescribeClusterMirrorsRequest. And we will use ReadMirrorStates and
>> >> > ReadMirrorLags to get each info.
>> >> >
>> >> >
>> >> > > DescribeClusterMirrorsRequest should be used in the Describe Mirror
>> >> flow,
>> >> > right? Currently it's missing. Also, I assume the admin client
>> refers to
>> >> > the one in the kafka-cluster-mirrors tool. But it seems that
>> sometimes
>> >> it
>> >> > also refers to the admin client on the broker? Perhaps we can use
>> 'the
>> >> > tool' instead to make it clear.
>> >> >
>> >> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
>> >> > And sorry for the confusion. I’d like to keep the admin client as the
>> >> one
>> >> > invoking from the tool (or user’s direct admin client API
>> invocation).
>> >> On
>> >> > the broker side, we actually use InterBrokerSendThread to invoke
>> these
>> >> RPCs
>> >> > or get directly from MetadataImage. I’ve updated the KIP to make it
>> >> clear.
>> >> >
>> >> >
>> >> > > JR58. BumpLeaderEpochs: Why does it need to be issued to the
>> broker,
>> >> > instead of just to the controller?
>> >> >
>> >> > Yes, we only need to issue to the controller. Updated to the
>> >> > BumpLeaderEpochs API.
>> >> >
>> >> >
>> >> > > JR59. Start Mirror Topics:
>> >> > > JR59.1 "During this state, the broker queries the source cluster
>> for
>> >> last
>> >> > mirror epochs".
>> >> > Could you describe which RPC is issued to find the LME?
>> >> >
>> >> > We’re using describeClusterMirror API via admin client here.
>> >> >
>> >> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
>> >> > doesn't
>> >> > contain a replication factor.
>> >> >
>> >> > We use “default.replication.factor” when topic creation, so the
>> >> > INVALID_REPLICATION_FACTOR will be thrown like normal topic creation.
>> >> Also,
>> >> > because we require the replicas for the partition assignment to be
>> all
>> >> > active and not in fenced or controlled shutdown(JR35), we’ll throw
>> >> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
>> >> >
>> >> >
>> >> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
>> >> > different topic name"
>> >> > Should we also return this error when the topic name is used as a
>> >> regular
>> >> > topic?
>> >> >
>> >> > There are 2 errors we will throw when topic id <-> topic name
>> mismatch.
>> >> > Suppose provided topic id: ID_A, topic name: NAME_A
>> >> > 1. Destination cluster exists topic id: ID_A, but topic name is:
>> NAME_B
>> >> > TOPIC_ALREADY_EXISTS will be thrown
>> >> > 2. Destination cluster exists topic id: ID_B, but topic name is:
>> NAME_A
>> >> > INCONSISTENT_TOPIC_ID will be thrown
>> >> >
>> >> > Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS error
>> >> for
>> >> > these 2 cases makes more sense. KIP updated.
>> >> >
>> >> >
>> >> > > JR60. CreateClusterMirrorRequest: When will it receive an
>> >> > INVALID_CLUSTER_MIRROR_STATES error not covered by
>> >> > CLUSTER_MIRROR_ALREADY_EXISTS?
>> >> >
>> >> >
>> >> > Sorry, createClusterMirrorRequest doesn’t throw
>> >> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate the
>> >> mirror
>> >> > states when creating a cluster mirror. We only care if the “mirror
>> name”
>> >> > exists or not. KIP updated.
>> >> >
>> >> >
>> >> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
>> >> remove
>> >> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> >> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe
>> the
>> >> > specific state causing the error.
>> >> >
>> >> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following
>> errors:
>> >> > a. MIRROR_TOPIC_ALREADY_PAUSED
>> >> > b. MIRROR_TOPIC_NOT_PAUSED,
>> >> > c. MIRROR_TOPIC_ALREADY_STOPPED,
>> >> > d. CLUSTER_MIRROR_NOT_EMPTY
>> >> > e. MIRROR_TOPIC_NOT_STOPPED
>> >> >
>> >> > KIP is updated.
>> >> >
>> >> >
>> >> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
>> >> requests
>> >> > (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
>> Could
>> >> we
>> >> > make them consistent?
>> >> >
>> >> >
>> >> > Good catch. The updated errors for the new added RPC are this:
>> >> >
>> >> > CreateClusterMirror: NOT_CONTROLLER
>> >> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >> >
>> >> >
>> >> > > JR63. StopMirrorTopics:
>> >> > > JR63.1 "The broker validates that all target topic partitions are
>> in
>> >> > either
>> >> > LOG_TRUNCATION or MIRRORING or FAILED state."
>> >> > Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED
>> can.
>> >> >
>> >> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is
>> updated.
>> >> >
>> >> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
>> >> PAUSED
>> >> > -> STOPPED is valid?
>> >> >
>> >> > Yes, it is not correct. KIP updated to remove this error.
>> >> >
>> >> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in
>> stopped
>> >> > state"
>> >> > This is weird. The reason to issue StopMirrorTopics is that it's not
>> in
>> >> > stopped state.
>> >> >
>> >> > Indeed. KIP is updated to only include INVALID_CLUSTER_MIRROR_STATES
>> >> > error, and will be thrown when states are not in MIRRORING or PAUSED
>> >> state.
>> >> >
>> >> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
>> >> derived
>> >> > from SourceOffset and DestinationOffset.
>> >> >
>> >> > True. Removed.
>> >> >
>> >> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems
>> that we
>> >> > could just set the partition level state to STOPPED.
>> >> >
>> >> > Nice catch. We don’t use it at all. That is some old leftover field
>> we
>> >> > forgot to remove. KIP updated.
>> >> >
>> >> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be
>> sent
>> >> to
>> >> > the controller?
>> >> >
>> >> > No, these 2 only send to the broker. KIP updated.
>> >> >
>> >> >
>> >> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and
>> StateEpoch?
>> >> It
>> >> > seems both are used for fencing and fencing based on the latter is
>> >> > stronger.
>> >> >
>> >> > Yes, basically the StateEpoch can fence for most cases. The
>> LeaderEpoch
>> >> > fencing is used for cases like:
>> >> >
>> >> > 1. b1 is the current leader for foo-0, preparing write state from
>> >> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
>> >> > 2. leadership change to b2 for foo-0, LE changes to 2.
>> >> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1 }.
>> >> > 4. The coordinator receives the read from b2 in step (3), the highest
>> >> > lastSeen LE for foo-0 updates to 2 in coordinator.
>> >> > 5. The writes state from b1 in step (1) arrives the coordinator, but
>> it
>> >> > rejects b1 writes because the highest lastSeen LE is 2, which is > 1
>> in
>> >> the
>> >> > request.
>> >> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2, stateEpoch: 1
>> and
>> >> > succeeds.
>> >> >
>> >> > The leaderEpoch fence is to avoid any stale broker writes after the
>> new
>> >> > leader epoch read. This cannot be protected by the stateEpoch. The
>> >> > stateEpoch can prevent the write with older stateEpoch. So in the
>> >> situation
>> >> > above, the stateEpoch cannot determine if we should accept b1 writes
>> or
>> >> > not, but leaderEpoch can. By the way, we borrowed this concept from
>> >> > KIP-932:
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
>> <https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rX3-NwDU3s0U7POQYH1R-DvASnCPTTtFRWxZTVaS-PQHg-C1TAcu9xMe3MQF4fIXGO67WKOqrlz-lc8$>
>> >> <
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!u8EkPGXm8_VKdltXy8lrIW14mR2kgYIoaBPD7LO2uAN6rjhrfCX37Pfa6r85QPpSwSPh4V6eW_bKlns$
>> >
>> >> > <
>> >>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
>> >> >
>> >> >
>> >> >
>> >> > > JR68. Could we make the usage of topicId and topicName in the RPCs
>> >> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> >> > BumpLeaderEpochsReponse uses TopicName.
>> >> >
>> >> > Yes, we have made:
>> >> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use TopicName.
>> >> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse use
>> >> > TopicName.
>> >> > 3. MirrorTopicStateChangeRecord uses TopicName because we will do
>> >> > validation before writing the metadata log.
>> >> >
>> >> >
>> >> > > JR69. MIRROR_PID_RESET:
>> >> > > JR69.1 Could you specify the value of the control record's type?
>> >> >
>> >> > It’s MIRROR_PID_RESET(7). KIP updated.
>> >> >
>> >> > > JR69.2 How is the SourceClusterId field used?
>> >> >
>> >> > I’m also surprised to see this field in the MirrorPidResetRecord. I
>> >> traced
>> >> > back to the internal PR we added for this field, and confirmed it is
>> not
>> >> > used since it was added. Sorry for the confusion and nice catch! KIP
>> >> > updated.
>> >> >
>> >> > > JR70. MirrorPartitionState: type should be "data" not
>> >> "coordinator-key",
>> >> > right? Ditto for LastMirrorEpochs.
>> >> > We store both MirrorPartitionState and LastMirrorEpochs records into
>> >> > internal __mirror_state topic, and rely on Coordinator Runtime to
>> >> > write/read the records, like how `OffsetCommit` and `GroupMetadata`
>> >> work.
>> >> > So I think we should keep them as
>> "coordinator-key"/"coordinator-value"
>> >> > type.
>> >> >
>> >> >
>> >> > Thank you,
>> >> > Luke
>> >> >
>> >> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <
>> dev@kafka.apache.org>
>> >> > wrote:
>> >> >
>> >> >> Hi, Luke,
>> >> >>
>> >> >> Thanks for the reply.
>> >> >>
>> >> >> JR51. "ULE_LOG_TRUNCATION When
>> mirror.support.unclean.leader.election
>> >> is
>> >> >> enabled and the destination leader has log truncation, the state
>> will
>> >> move
>> >> >> from MIRRORING to this state."
>> >> >> This still suggests that divergeEpoch is only handled if
>> >> >> mirror.support.unclean.leader.election
>> >> >> is enabled.
>> >> >>
>> >> >> JR55. Should the mirror state and the mirror lags always be exposed
>> >> >> together? If so, we can remove IncludeMirrorState and
>> IncludeMirrorLags
>> >> >> from DescribeClusterMirrorsRequest. If not, it's better to use two
>> >> >> separate
>> >> >> RPCs.
>> >> >>
>> >> >> DescribeClusterMirrorsRequest should be used in the Describe Mirror
>> >> flow,
>> >> >> right? Currently it's missing. Also, I assume the admin client
>> refers
>> >> to
>> >> >> the one in the kafka-cluster-mirrors tool. But it seems that
>> sometimes
>> >> it
>> >> >> also refers to the admin client on the broker? Perhaps we can use
>> 'the
>> >> >> tool' instead to make it clear.
>> >> >>
>> >> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
>> >> >> instead of just to the controller?
>> >> >>
>> >> >> JR59. Start Mirror Topics:
>> >> >> JR59.1 "During this state, the broker queries the source cluster for
>> >> last
>> >> >> mirror epochs".
>> >> >> Could you describe which RPC is issued to find the LME?
>> >> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
>> >> >> doesn't
>> >> >> contain a replication factor.
>> >> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
>> >> >> different topic name"
>> >> >> Should we also return this error when the topic name is used as a
>> >> regular
>> >> >> topic?
>> >> >>
>> >> >> JR60. CreateClusterMirrorRequest: When will it receive an
>> >> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
>> >> >> CLUSTER_MIRROR_ALREADY_EXISTS?
>> >> >>
>> >> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
>> >> remove
>> >> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> >> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe
>> the
>> >> >> specific state causing the error.
>> >> >>
>> >> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
>> >> requests
>> >> >> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
>> >> Could we
>> >> >> make them consistent?
>> >> >>
>> >> >> JR63. StopMirrorTopics:
>> >> >> JR63.1 "The broker validates that all target topic partitions are in
>> >> >> either
>> >> >> LOG_TRUNCATION or MIRRORING or FAILED state."
>> >> >> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED
>> can.
>> >> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
>> >> PAUSED
>> >> >> -> STOPPED is valid?
>> >> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
>> >> >> state"
>> >> >> This is weird. The reason to issue StopMirrorTopics is that it's
>> not in
>> >> >> stopped state.
>> >> >>
>> >> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
>> derived
>> >> >> from SourceOffset and DestinationOffset.
>> >> >>
>> >> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that
>> we
>> >> >> could just set the partition level state to STOPPED.
>> >> >>
>> >> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be
>> sent
>> >> to
>> >> >> the controller?
>> >> >>
>> >> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and
>> StateEpoch? It
>> >> >> seems both are used for fencing and fencing based on the latter is
>> >> >> stronger.
>> >> >>
>> >> >> JR68. Could we make the usage of topicId and topicName in the RPCs
>> >> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> >> >> BumpLeaderEpochsReponse uses TopicName.
>> >> >>
>> >> >> JR69. MIRROR_PID_RESET:
>> >> >> JR69.1 Could you specify the value of the control record's type?
>> >> >> JR69.2 How is the SourceClusterId field used?
>> >> >>
>> >> >> JR70. MirrorPartitionState: type should be "data" not
>> >> "coordinator-key",
>> >> >> right? Ditto for LastMirrorEpochs.
>> >> >>
>> >> >> Jun
>> >> >>
>> >> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com>
>> wrote:
>> >> >>
>> >> >> > Hi Jun,
>> >> >> >
>> >> >> > Thanks for the comments.
>> >> >> > Answering them below.
>> >> >> >
>> >> >> > > JR36. Sounds good. Should we remove
>> >> >> MirrorLeaderEpochExceededException?
>> >> >> >
>> >> >> > Sorry, it’s a confusing name. The
>> MirrorLeaderEpochExceededException
>> >> is
>> >> >> > used and thrown when the fetched mirror batch leader epoch > local
>> >> >> leader
>> >> >> > epoch in destination cluster. And then we’ll stop mirroring and
>> move
>> >> the
>> >> >> > partition to EPOCH_FENSING state to wait for local leader epoch
>> >> bump. I
>> >> >> > think since we don’t have the concept of `MirrorLeaderEpoch`
>> anymore,
>> >> >> the
>> >> >> > `MirrorLeaderEpochExceededException` should not cause confusion
>> now.
>> >> >> I’ll
>> >> >> > keep the name for the exception.
>> >> >> >
>> >> >> >
>> >> >> > > JR51. Sounds good. Do we plan to always set lastFetchedEpoch in
>> the
>> >> >> > mirror
>> >> >> > fetch request issued to the source cluster, independent of the
>> value
>> >> of
>> >> >> > mirror.support.unclean.leader.election? It's needed to detect
>> >> divergence
>> >> >> > during failover. However, if it's also set when
>> >> >> > mirror.support.unclean.leader.election=false,
>> >> >> > an unclean leader election in the source cluster can halt the
>> >> mirroring
>> >> >> > since we don’t do anything with the returned divergeEpoch in that
>> >> mode
>> >> >> > and continue
>> >> >> > the next fetch.
>> >> >> >
>> >> >> > > Do we plan to always set lastFetchedEpoch in the mirror
>> >> >> > fetch request issued to the source cluster, independent of the
>> value
>> >> of
>> >> >> > Mirror.support.unclean.leader.election?
>> >> >> >
>> >> >> > Yes, we always set lastFetchedEpoch in the mirror fetch request.
>> >> >> >
>> >> >> > > However, if it's also set when
>> >> >> > mirror.support.unclean.leader.election=false,
>> >> >> > an unclean leader election in the source cluster can halt the
>> >> mirroring
>> >> >> > since we don’t do anything with the returned divergeEpoch in that
>> >> mode
>> >> >> > and continue the next fetch.
>> >> >> >
>> >> >> > Sorry I didn’t make it clear. When
>> >> >> > “mirror.support.unclean.leader.election=false” and a
>> “divergeEpoch”
>> >> is
>> >> >> set
>> >> >> > in fetch response, the leader node will handle the divergeEpoch
>> and
>> >> then
>> >> >> > continue the next fetch as usual. The difference with
>> >> >> > “mirror.support.unclean.leader.election=true” is that “should we
>> wait
>> >> >> for
>> >> >> > ALL replicas caught up (complete truncation)”. Without the
>> waiting,
>> >> it’s
>> >> >> > possible the inconsistent log issue in KAFKA-18723 could happen
>> >> between
>> >> >> > leader and follower nodes in the destination cluster because we
>> don’t
>> >> >> have
>> >> >> > “MirrorLeaderEpoch” concept now to reject higher leader epoch
>> >> batches.
>> >> >> That
>> >> >> > is, we won't halt the mirroring if "divergeEpoch" is set but
>> >> >> > “mirror.support.unclean.leader.election=false”.
>> >> >> >
>> >> >> >
>> >> >> > > JR55. The DescribeClusterMirrorsRequest still returns both the
>> >> mirror
>> >> >> > partition state and the mirror partition leader state. The
>> problem is
>> >> >> that
>> >> >> > if a caller only needs one of the state, the other state becomes
>> >> >> redundant.
>> >> >> > For example, when starting the mirror, to obtain LME, we only need
>> >> the
>> >> >> > mirror partition state. Here is an alternative design: We expose
>> the
>> >> >> mirror
>> >> >> > partition state through ReadMirrorState. The client will use the
>> >> >> > FindCoordinatorRequest
>> >> >> > to find the right mirror partition coordinator to issue that
>> >> request. We
>> >> >> > expose the mirror partition leader state through the
>> >> >> ReadClusterMirrorLags
>> >> >> > request. The client can use the metadata request to find the
>> >> partition
>> >> >> > leader to issue that request.
>> >> >> >
>> >> >> > > In the describing mirror flow, the admin client first issues the
>> >> >> > ListMirrorRequest to find all the mirrors and the topics being
>> >> >> mirrored. It
>> >> >> > then issues the metadata request to find the partitions for those
>> >> topics
>> >> >> > and issues a FindCoordinatorRequest to find the mirror partition
>> >> state
>> >> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
>> >> partition
>> >> >> > state coordinator and ReadClusterMirrorLags to the partition
>> leader.
>> >> >> >
>> >> >> > > In the starting mirror flow, the partition leader in the
>> >> destination
>> >> >> > cluster issues a ListMirrorRequest to find the mirror name based
>> on
>> >> its
>> >> >> > cluster id. It then issues a FindCoordinatorRequest to find the
>> >> mirror
>> >> >> > partition state coordinators. Finally, it issues ReadMirrorState
>> to
>> >> the
>> >> >> > mirror partition state coordinator to get the LME.
>> >> >> >
>> >> >> > > We can also improve the ListMirror request to accept a filter
>> >> based on
>> >> >> > mirror name, source cluster ID and topic name.
>> >> >> >
>> >> >> >
>> >> >> > Answers:
>> >> >> >
>> >> >> > > The DescribeClusterMirrorsRequest still returns both the mirror
>> >> >> > partition state and the mirror partition leader state. The
>> problem is
>> >> >> that
>> >> >> > if a caller only needs one of the state, the other state becomes
>> >> >> redundant.
>> >> >> > For example, when starting the mirror, to obtain LME, we only need
>> >> the
>> >> >> > mirror partition state.
>> >> >> >
>> >> >> > Good point!
>> >> >> >
>> >> >> >
>> >> >> > > Here is an alternative design: We expose the mirror
>> >> >> > partition state through ReadMirrorState. The client will use the
>> >> >> > FindCoordinatorRequest
>> >> >> > to find the right mirror partition coordinator to issue that
>> >> request. We
>> >> >> > expose the mirror partition leader state through the
>> >> >> ReadClusterMirrorLags
>> >> >> > request. The client can use the metadata request to find the
>> >> partition
>> >> >> > leader to issue that request.
>> >> >> >
>> >> >> >
>> >> >> > Sounds good. But we try not to expose
>> >> >> > ReadMirrorState/ReadClusterMirrorLags directly to the admin client
>> >> >> because
>> >> >> > users might get confused about these 2 requests, and they are also
>> >> both
>> >> >> > related to describeClusterMirror. I think we can add 2 additional
>> >> >> fields to
>> >> >> > describeClusterMirrorRequest: “includeMirrorStates” and
>> >> >> > “includeMirrorLags”. So the admin client can include either one or
>> >> both
>> >> >> of
>> >> >> > them in the DescrbieClusterMirrorRequest based on the provided
>> >> options.
>> >> >> KIP
>> >> >> > is updated.
>> >> >> >
>> >> >> >
>> >> >> > > In the describing mirror flow, the admin client first issues the
>> >> >> > ListMirrorRequest to find all the mirrors and the topics being
>> >> >> mirrored. It
>> >> >> > then issues the metadata request to find the partitions for those
>> >> topics
>> >> >> > and issues a FindCoordinatorRequest to find the mirror partition
>> >> state
>> >> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
>> >> partition
>> >> >> > state coordinator and ReadClusterMirrorLags to the partition
>> leader.
>> >> >> >
>> >> >> > Makes sense. KIP is updated.
>> >> >> >
>> >> >> >
>> >> >> > > In the starting mirror flow, the partition leader in the
>> >> destination
>> >> >> > cluster issues a ListMirrorRequest to find the mirror name based
>> on
>> >> its
>> >> >> > cluster id. It then issues a FindCoordinatorRequest to find the
>> >> mirror
>> >> >> > partition state coordinators. Finally, it issues ReadMirrorState
>> to
>> >> the
>> >> >> > mirror partition state coordinator to get the LME.
>> >> >> >
>> >> >> > Make sense. So we can send DescribeClusterMirrorRequest with
>> >> >> > “includeMirrorStates” only.
>> >> >> >
>> >> >> > > We can also improve the ListMirror request to accept a filter
>> >> based on
>> >> >> > mirror name, source cluster ID and topic name.
>> >> >> >
>> >> >> > Good suggestion, just like what ListGroupsRequest does.
>> >> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
>> >> >> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP
>> >> >> updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR56. Could you list the ACL needed for each new RPC?
>> >> >> >
>> >> >> >
>> >> >> > Yes, it’s already listed in “Security Control” section.
>> >> >> >
>> >> >> >
>> >> >> > > JR57. In the command workflow, it sometimes says "The user
>> sends"
>> >> and
>> >> >> > some
>> >> >> > other times says "The admin client sends". It's better to use "The
>> >> admin
>> >> >> > client" consistently in all those places.
>> >> >> >
>> >> >> > Agree. KIP updated.
>> >> >> >
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Luke
>> >> >> >
>> >> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <
>> dev@kafka.apache.org
>> >> >
>> >> >> > wrote:
>> >> >> >
>> >> >> >> Hi, Luke,
>> >> >> >>
>> >> >> >> Thanks for the reply.
>> >> >> >>
>> >> >> >> JR36. Sounds good. Should we remove
>> >> MirrorLeaderEpochExceededException?
>> >> >> >>
>> >> >> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in
>> the
>> >> >> mirror
>> >> >> >> fetch request issued to the source cluster, independent of the
>> >> value of
>> >> >> >> mirror.support.unclean.leader.election? It's needed to detect
>> >> >> divergence
>> >> >> >> during failover. However, if it's also set when
>> >> >> >> mirror.support.unclean.leader.election=false,
>> >> >> >> an unclean leader election in the source cluster can halt the
>> >> mirroring
>> >> >> >> since we don’t do anything with the returned divergeEpoch in that
>> >> mode
>> >> >> >> and continue
>> >> >> >> the next fetch.
>> >> >> >>
>> >> >> >> JR55. The DescribeClusterMirrorsRequest still returns both the
>> >> mirror
>> >> >> >> partition state and the mirror partition leader state. The
>> problem
>> >> is
>> >> >> that
>> >> >> >> if a caller only needs one of the state, the other state becomes
>> >> >> >> redundant.
>> >> >> >> For example, when starting the mirror, to obtain LME, we only
>> need
>> >> the
>> >> >> >> mirror partition state. Here is an alternative design: We expose
>> the
>> >> >> >> mirror
>> >> >> >> partition state through ReadMirrorState. The client will use the
>> >> >> >> FindCoordinatorRequest
>> >> >> >> to find the right mirror partition coordinator to issue that
>> >> request.
>> >> >> We
>> >> >> >> expose the mirror partition leader state through the
>> >> >> ReadClusterMirrorLags
>> >> >> >> request. The client can use the metadata request to find the
>> >> partition
>> >> >> >> leader to issue that request.
>> >> >> >>
>> >> >> >> In the describing mirror flow, the admin client first issues the
>> >> >> >> ListMirrorRequest to find all the mirrors and the topics being
>> >> >> mirrored.
>> >> >> >> It
>> >> >> >> then issues the metadata request to find the partitions for those
>> >> >> topics
>> >> >> >> and issues a FindCoordinatorRequest to find the mirror partition
>> >> state
>> >> >> >> coordinators. Finally, it issues ReadMirrorState to the mirror
>> >> >> partition
>> >> >> >> state coordinator and ReadClusterMirrorLags to the partition
>> leader.
>> >> >> >>
>> >> >> >> In the starting mirror flow, the partition leader in the
>> destination
>> >> >> >> cluster issues a ListMirrorRequest to find the mirror name based
>> on
>> >> its
>> >> >> >> cluster id. It then issues a FindCoordinatorRequest to find the
>> >> mirror
>> >> >> >> partition state coordinators. Finally, it issues ReadMirrorState
>> to
>> >> the
>> >> >> >> mirror partition state coordinator to get the LME.
>> >> >> >>
>> >> >> >> We can also improve the ListMirror request to accept a filter
>> based
>> >> on
>> >> >> >> mirror name, source cluster ID and topic name.
>> >> >> >>
>> >> >> >> JR56. Could you list the ACL needed for each new RPC?
>> >> >> >>
>> >> >> >> JR57. In the command workflow, it sometimes says "The user sends"
>> >> and
>> >> >> some
>> >> >> >> other times says "The admin client sends". It's better to use
>> "The
>> >> >> admin
>> >> >> >> client" consistently in all those places.
>> >> >> >>
>> >> >> >> Jun
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com>
>> wrote:
>> >> >> >>
>> >> >> >> > Hi Jun,
>> >> >> >> >
>> >> >> >> > Thanks for your comments.
>> >> >> >> > Answering them below.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> >> >> specifically
>> >> >> >> > designed for the unclean leader election support in Cluster
>> >> Mirroring
>> >> >> >> > because the unclean leader election will break the assumption
>> that
>> >> >> the
>> >> >> >> > committed data won’t be truncated."
>> >> >> >> > If the source cluster has an unclean leader election, the
>> >> destination
>> >> >> >> > cluster will detect the inconsistency and truncate the log at
>> the
>> >> >> >> > destination leader. We can just
>> >> >> >> > extend mirror.support.unclean.leader.election to cover this
>> case,
>> >> >> i.e.,
>> >> >> >> we
>> >> >> >> > wait until all replicas' log end offsets are reduced to the
>> target
>> >> >> >> offset
>> >> >> >> > before resuming mirroring. This avoids the inconsistency.
>> >> >> >> >
>> >> >> >> > If the destination cluster has an unclean leader election, it
>> will
>> >> >> >> > re-fetch the same data from the source. There won't be any
>> >> >> inconsistency
>> >> >> >> > either.
>> >> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Yes, that works, too. I agree this is better because the
>> unclean
>> >> >> leader
>> >> >> >> > election is rare, we don't want to occupy a field just for it.
>> >> >> >> >
>> >> >> >> > So to achieve this, we have to:
>> >> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
>> >> >> >> > 2. When the destination cluster leader completes a log
>> truncation
>> >> >> after
>> >> >> >> a
>> >> >> >> > fetch response due to divergeEpoch, we know it must be caused
>> by
>> >> >> unclean
>> >> >> >> > leader election in the source cluster because we only fetch
>> >> committed
>> >> >> >> > records. If “mirror.support.unclean.leader.election=false”, we
>> >> don’t
>> >> >> do
>> >> >> >> > anything and continue the next fetch. If
>> >> >> >> > “mirror.support.unclean.leader.election=true”, we remove the
>> >> mirror
>> >> >> >> fetcher
>> >> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
>> >> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to
>> be
>> >> >> caught
>> >> >> >> up
>> >> >> >> > into ISR and reduced to the expected LEO.
>> >> >> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
>> >> >> >> >
>> >> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch
>> API.
>> >> >> >> > KIP is updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR51. Make sense. To truncate based on LME, we need to issue
>> >> >> another
>> >> >> >> RPC
>> >> >> >> > to translate the LME to an offset. It would be useful to
>> include
>> >> >> that in
>> >> >> >> > the workflow description.
>> >> >> >> >
>> >> >> >> > About the additional RPC to translate the LME to the offset
>> (i.e.
>> >> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on this
>> to do
>> >> >> the
>> >> >> >> > offset translation. We use the local leader epoch cache in the
>> >> >> >> destination
>> >> >> >> > cluster to translate the epoch to save the RPC call. This works
>> >> >> because
>> >> >> >> of
>> >> >> >> > an invariant the LME truncation protocol maintains: for any
>> leader
>> >> >> >> epoch ≤
>> >> >> >> > LME, that epoch's data belongs to exactly one cluster, either
>> it
>> >> >> >> originated
>> >> >> >> > there, or it was mirrored. This holds even across repeated
>> mirror
>> >> ->
>> >> >> >> > reverse-mirror -> mirror cycles, since each direction's
>> >> >> LOG_TRUNCATION
>> >> >> >> > phase re-establishes the invariant before that cluster starts
>> >> >> mirroring.
>> >> >> >> > With this invariant, the replication protocol can find out the
>> >> >> diverge
>> >> >> >> > point and truncate the log correctly when mirroring. For
>> example:
>> >> >> >> >
>> >> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> > Offset 2: A (LE: 0)
>> >> >> >> > Offset 3: A (LE: 1)
>> >> >> >> >
>> >> >> >> > Cluster A crashes while Cluster B mirrors to offset 1. Failover
>> >> to B
>> >> >> and
>> >> >> >> > store the LME=0 in cluster B. When cluster A recovers and
>> starts
>> >> the
>> >> >> >> > reverse mirroring, it’ll query the LME from B and get back
>> LME: 0.
>> >> >> Then
>> >> >> >> > cluster A gets the end offset for epoch 0 from
>> leaderEpochCache,
>> >> and
>> >> >> >> gets
>> >> >> >> > offset 2. So offset 3 is truncated. The log is like this now:
>> >> >> >> >
>> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
>> >> >> >> > Offset 3: B (LE:
>> 10)
>> >> >> >> >
>> >> >> >> > When mirror starts, cluster A sends the fetch request with {
>> >> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the
>> divergingEpoch
>> >> >> check
>> >> >> >> in
>> >> >> >> > the cluster B queries the endOffset for leader epoch 0, and
>> found
>> >> the
>> >> >> >> end
>> >> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns
>> >> fetch
>> >> >> >> > response with divergingEpoch info to cluster A for log
>> >> truncation. In
>> >> >> >> the
>> >> >> >> > end, offset 2 in cluster A will be truncated and then log
>> >> converges
>> >> >> with
>> >> >> >> > cluster B.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit
>> >> strange
>> >> >> >> > because it mixes querying the mirror partition state with
>> querying
>> >> >> the
>> >> >> >> > mirror partition leader. Another option is to use
>> >> >> >> ReadMirrorStatesRequest
>> >> >> >> > to query the mirror partition state and a new request to query
>> the
>> >> >> >> mirror
>> >> >> >> > partition leader. If we want the adminClient to query the right
>> >> >> >> ClusterMirrorCoordinator
>> >> >> >> > directly, we can extend the FindCoordinatorRequest to support
>> >> that.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Good suggestion. Yes, we need to fan out to all brokers
>> because we
>> >> >> have
>> >> >> >> to
>> >> >> >> > collect info from mirror partition leaders and mirror
>> >> coordinators.
>> >> >> If
>> >> >> >> we
>> >> >> >> > create a new API to contact mirror partition leaders, it will
>> make
>> >> >> the
>> >> >> >> API
>> >> >> >> > cleaner and clearer. So we will do this:
>> >> >> >> >
>> >> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields from
>> >> >> >> > DescribeClusterMirror request/response
>> >> >> >> >
>> >> >> >> > b. Add a new API: ReadClusterMirrorLags API to request for
>> >> >> SourceOffset,
>> >> >> >> > DestinationOffset, and Lag info.
>> >> >> >> >
>> >> >> >> > The workflow for admin client describeClusterMirror will be
>> like
>> >> >> this:
>> >> >> >> > 1. adminClient sends describeClusterMirror to the
>> leastLoadedNode
>> >> >> broker
>> >> >> >> > 2. The received broker collects the current mirrored
>> partitions,
>> >> and
>> >> >> >> > groups them by coordinator node and by partition leader node.
>> >> >> >> > 3. The broker sends ReadMirrorState request to the coordinator
>> >> nodes
>> >> >> to
>> >> >> >> > get the mirror state related info
>> >> >> >> > 4. The broker sends ReadClusterMirrorLags request to the mirror
>> >> >> >> partition
>> >> >> >> > leader nodes to get the mirror lag, source offset, destination
>> >> offset
>> >> >> >> > (which updates on every fetch response).
>> >> >> >> > 5. The broker merge all the response into describeClusterMirror
>> >> >> response
>> >> >> >> > and send to admin client
>> >> >> >> >
>> >> >> >> > KIP is updated. Thanks for the suggestion.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR55. For __mirror_state topic, could we describe whether it
>> is
>> >> >> >> > compacted?
>> >> >> >> >
>> >> >> >> > Yes, it is a compacted topic. I made it much clearer in the
>> >> >> >> > “ClusterMirrorCoordinator” section.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Thank you,
>> >> >> >> > Luke
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
>> >> dev@kafka.apache.org
>> >> >> >
>> >> >> >> > wrote:
>> >> >> >> >
>> >> >> >> >> Hi, Luke,
>> >> >> >> >>
>> >> >> >> >> Thanks for the reply.
>> >> >> >> >>
>> >> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> >> >> specifically
>> >> >> >> >> designed for the unclean leader election support in Cluster
>> >> >> Mirroring
>> >> >> >> >> because the unclean leader election will break the assumption
>> >> that
>> >> >> the
>> >> >> >> >> committed data won’t be truncated."
>> >> >> >> >> If the source cluster has an unclean leader election, the
>> >> >> destination
>> >> >> >> >> cluster will detect the inconsistency and truncate the log at
>> the
>> >> >> >> >> destination leader. We can just
>> >> >> >> >> extend mirror.support.unclean.leader.election to cover this
>> case,
>> >> >> >> i.e., we
>> >> >> >> >> wait until all replicas' log end offsets are reduced to the
>> >> target
>> >> >> >> offset
>> >> >> >> >> before resuming mirroring. This avoids the inconsistency.
>> >> >> >> >>
>> >> >> >> >> If the destination cluster has an unclean leader election, it
>> >> will
>> >> >> >> >> re-fetch
>> >> >> >> >> the same data from the source. There won't be any
>> inconsistency
>> >> >> either.
>> >> >> >> >>
>> >> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >> >> >>
>> >> >> >> >> JR51. Make sense. To truncate based on LME, we need to issue
>> >> another
>> >> >> >> RPC
>> >> >> >> >> to
>> >> >> >> >> translate the LME to an offset. It would be useful to include
>> >> that
>> >> >> in
>> >> >> >> the
>> >> >> >> >> workflow description.
>> >> >> >> >>
>> >> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit
>> strange
>> >> >> >> because
>> >> >> >> >> it mixes querying the mirror partition state with querying the
>> >> >> mirror
>> >> >> >> >> partition leader. Another option is to use
>> >> ReadMirrorStatesRequest
>> >> >> to
>> >> >> >> >> query
>> >> >> >> >> the mirror partition state and a new request to query the
>> mirror
>> >> >> >> partition
>> >> >> >> >> leader. If we want the adminClient to query the right
>> >> >> >> >> ClusterMirrorCoordinator
>> >> >> >> >> directly, we can extend the FindCoordinatorRequest to support
>> >> that.
>> >> >> >> >>
>> >> >> >> >> JR55. For __mirror_state topic, could we describe whether it
>> is
>> >> >> >> compacted?
>> >> >> >> >>
>> >> >> >> >> Jun
>> >> >> >> >>
>> >> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com>
>> >> wrote:
>> >> >> >> >>
>> >> >> >> >> > Hi Jun,
>> >> >> >> >> >
>> >> >> >> >> > Thanks for the comments.
>> >> >> >> >> > Answering them below:
>> >> >> >> >> >
>> >> >> >> >> > > JR31.3 It makes sense for the new partition leader to
>> query
>> >> the
>> >> >> >> >> current
>> >> >> >> >> > mirror state from the local or remote MirrorMetadataManager
>> >> that
>> >> >> owns
>> >> >> >> >> the
>> >> >> >> >> > partition's state. However, the wording in the KIP is
>> >> confusing.
>> >> >> >> >> > MirrorMetadataManager
>> >> >> >> >> > mirror partition state cache should be populated only by one
>> >> way,
>> >> >> >> which
>> >> >> >> >> is
>> >> >> >> >> > replaying __mirror_state partitions this broker leads. By
>> >> saying
>> >> >> >> "and by
>> >> >> >> >> > fetching state from remote coordinators via ReadMirrorStates
>> >> >> RPCs.",
>> >> >> >> >> it's
>> >> >> >> >> > as if the mirror partition state cache on each broker stores
>> >> the
>> >> >> >> state
>> >> >> >> >> for
>> >> >> >> >> > all mirror partitions.
>> >> >> >> >> >
>> >> >> >> >> > You’re right. KIP is updated to make it clear.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR36. I understand the intention now, but I still don't
>> >> think we
>> >> >> >> need
>> >> >> >> >> > MirrorLeaderEpoch. Let's
>> >> >> >> >> > assume the common case: no unclean leader election in the
>> >> source
>> >> >> >> cluster
>> >> >> >> >> > during mirroring. Mirroring only fetches the committed data
>> >> from
>> >> >> the
>> >> >> >> >> > source, which never changes if obtained a second time. In
>> your
>> >> >> >> example
>> >> >> >> >> > image, follower broker 1 in the destination cluster can
>> never
>> >> >> obtain
>> >> >> >> the
>> >> >> >> >> > record B(0) because it's never committed in the source and
>> >> thus is
>> >> >> >> never
>> >> >> >> >> > exposed to the destination leader. It's possible for broker
>> 1
>> >> to
>> >> >> >> first
>> >> >> >> >> get
>> >> >> >> >> > D(1) which is truncated later when broker 0 follows a new
>> >> leader.
>> >> >> But
>> >> >> >> >> the
>> >> >> >> >> > new leader will refetch the same D(1) from the source
>> cluster.
>> >> So,
>> >> >> >> it's
>> >> >> >> >> ok
>> >> >> >> >> > for broker 1 to keep D(1).
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Yes, you are correct. Because mirroring only fetches the
>> >> committed
>> >> >> >> data
>> >> >> >> >> > from the source, the MirrorLeaderEpoch is unnecessary for
>> >> normal
>> >> >> >> >> mirroring.
>> >> >> >> >> > Thanks for the clarification. So I think we can say, the
>> >> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the unclean
>> >> >> leader
>> >> >> >> >> > election support in Cluster Mirroring because the unclean
>> >> leader
>> >> >> >> >> election
>> >> >> >> >> > will break the assumption that the committed data won’t be
>> >> >> truncated.
>> >> >> >> >> >
>> >> >> >> >> > Given that, we could gate the MirrorLeaderEpoch field by
>> only
>> >> >> >> setting it
>> >> >> >> >> > in the Fetch request when
>> >> >> >> "mirror.support.unclean.leader.election=true"
>> >> >> >> >> > (false by default), and skip it entirely for normal
>> mirroring.
>> >> >> What
>> >> >> >> do
>> >> >> >> >> you
>> >> >> >> >> > think?
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR51. Sounds good. Why do we choose to store the last
>> mirror
>> >> >> epoch
>> >> >> >> >> > instead
>> >> >> >> >> > of a last mirror offset?
>> >> >> >> >> >
>> >> >> >> >> > Yes, originally we stored the “last mirror offset” instead
>> of
>> >> >> “last
>> >> >> >> >> mirror
>> >> >> >> >> > epoch”. But later, we found it doesn’t work for unclean
>> leader
>> >> >> >> election
>> >> >> >> >> > cases. (yes, `unclean leader election` again :) ) For
>> example:
>> >> >> >> >> >
>> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> >> >
>> >> >> >> >> > Cluster B is mirroring from A. And A crashes and failover to
>> >> B. B
>> >> >> >> stores
>> >> >> >> >> > the “last mirror offset” as 1.
>> >> >> >> >> > When cluster A recovers, somehow the unclean leader election
>> >> >> happens,
>> >> >> >> >> log
>> >> >> >> >> > is empty in the new leader, and then some more leadership
>> >> changes
>> >> >> to
>> >> >> >> >> make
>> >> >> >> >> > the leader epoch become 10. Then it starts the reverse
>> mirror
>> >> >> from B.
>> >> >> >> >> The
>> >> >> >> >> > log looks like this:
>> >> >> >> >> >
>> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
>> >> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
>> >> >> >> >> > Offset 2:
>> pid_res
>> >> (LE:
>> >> >> >> 10)
>> >> >> >> >> >
>> >> >> >> >> > When A starts mirroring from B, log is truncated to last
>> mirror
>> >> >> >> offset:
>> >> >> >> >> 1
>> >> >> >> >> > (no-op), and then fetch from cluster B. The 1st fetch
>> request
>> >> is
>> >> >> like
>> >> >> >> >> this:
>> >> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset:
>> >> 2, …
>> >> >> }.
>> >> >> >> The
>> >> >> >> >> > divergingEpoch check in the cluster B queries the endOffset
>> for
>> >> >> >> leader
>> >> >> >> >> > epoch 10 (in Partition#readRecords), and found the end
>> offset
>> >> for
>> >> >> LE
>> >> >> >> 10
>> >> >> >> >> is
>> >> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch
>> validation
>> >> >> >> passes
>> >> >> >> >> and
>> >> >> >> >> > returns offset 2 to cluster A. In short, relying on “last
>> >> mirror
>> >> >> >> offset”
>> >> >> >> >> > cannot resolve the log diverge issue in unclean leader
>> election
>> >> >> case.
>> >> >> >> >> >
>> >> >> >> >> > If instead, we truncate until “last mirror epoch”: 0, the
>> >> offset 0
>> >> >> >> and 1
>> >> >> >> >> > in cluster A will be truncated as expected and then log
>> >> converge
>> >> >> with
>> >> >> >> >> > cluster B correctly in the end.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Also, about JR53.2:
>> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
>> >> brokers
>> >> >> and
>> >> >> >> >> > merges
>> >> >> >> >> > the responses client-side."
>> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
>> partition.
>> >> So
>> >> >> >> sending
>> >> >> >> >> > the request to every broker is wasteful. It's probably
>> better
>> >> to
>> >> >> send
>> >> >> >> >> the
>> >> >> >> >> > request to a single broker, which can then contact the right
>> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >
>> >> >> >> >> > While working on this, I found we need to add one more
>> tagged
>> >> >> field
>> >> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema
>> because
>> >> >> >> >> otherwise
>> >> >> >> >> > the receiving broker doesn’t know if this is the request
>> from
>> >> >> client
>> >> >> >> >> and it
>> >> >> >> >> > needs to contact other brokers, or this is just a forwarded
>> >> >> request.
>> >> >> >> The
>> >> >> >> >> > workflow is like this:
>> >> >> >> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest
>> to a
>> >> >> >> >> > leastLoaded broker, which doesn’t contain the "Forwarded"
>> >> field.
>> >> >> >> >> > 2. The broker receives the request and sends
>> >> >> >> >> DescribeClusterMirrorsRequest
>> >> >> >> >> > with "Forwarded=true" to the responsible brokers and waits
>> for
>> >> the
>> >> >> >> >> response
>> >> >> >> >> > async.
>> >> >> >> >> > 3. When all responses are received, merge the results and
>> >> respond
>> >> >> to
>> >> >> >> the
>> >> >> >> >> > client.
>> >> >> >> >> >
>> >> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here because it
>> >> >> doesn’t
>> >> >> >> >> > contain the lag info in the coordinator.
>> >> >> >> >> >
>> >> >> >> >> > The alternative solution is the adminClient sends
>> >> >> >> >> > DescribeClusterMirrorsRequest to the “responsible brokers
>> only”
>> >> >> (via
>> >> >> >> >> > describeTopics), instead of ALL brokers, and merge the
>> results
>> >> in
>> >> >> the
>> >> >> >> >> > client side. This solution doesn’t need to add a new field
>> in
>> >> the
>> >> >> >> >> > DescribeClusterMirrorsRequest and still can avoid
>> unnecessary
>> >> >> >> requests
>> >> >> >> >> > sent.
>> >> >> >> >> >
>> >> >> >> >> > My preference is to keep the merge logic on the admin client
>> >> side,
>> >> >> >> which
>> >> >> >> >> > saves the overhead in the broker side.
>> >> >> >> >> > What do you think?
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Thank you,
>> >> >> >> >> > Luke
>> >> >> >> >> >
>> >> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
>> >> >> dev@kafka.apache.org
>> >> >> >> >
>> >> >> >> >> > wrote:
>> >> >> >> >> >
>> >> >> >> >> >> Hi, Luke,
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks for the reply.
>> >> >> >> >> >>
>> >> >> >> >> >> JR31.3 It makes sense for the new partition leader to query
>> >> the
>> >> >> >> current
>> >> >> >> >> >> mirror state from the local or remote MirrorMetadataManager
>> >> that
>> >> >> >> owns
>> >> >> >> >> the
>> >> >> >> >> >> partition's state. However, the wording in the KIP is
>> >> confusing.
>> >> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> mirror partition state cache should be populated only by
>> one
>> >> way,
>> >> >> >> >> which is
>> >> >> >> >> >> replaying __mirror_state partitions this broker leads. By
>> >> saying
>> >> >> >> "and
>> >> >> >> >> by
>> >> >> >> >> >> fetching state from remote coordinators via
>> ReadMirrorStates
>> >> >> RPCs.",
>> >> >> >> >> it's
>> >> >> >> >> >> as if the mirror partition state cache on each broker
>> stores
>> >> the
>> >> >> >> state
>> >> >> >> >> for
>> >> >> >> >> >> all mirror partitions.
>> >> >> >> >> >>
>> >> >> >> >> >> JR36. I understand the intention now, but I still don't
>> think
>> >> we
>> >> >> >> need
>> >> >> >> >> >> MirrorLeaderEpoch. Let's
>> >> >> >> >> >> assume the common case: no unclean leader election in the
>> >> source
>> >> >> >> >> cluster
>> >> >> >> >> >> during mirroring. Mirroring only fetches the committed data
>> >> from
>> >> >> the
>> >> >> >> >> >> source, which never changes if obtained a second time. In
>> your
>> >> >> >> example
>> >> >> >> >> >> image, follower broker 1 in the destination cluster can
>> never
>> >> >> obtain
>> >> >> >> >> the
>> >> >> >> >> >> record B(0) because it's never committed in the source and
>> >> thus
>> >> >> is
>> >> >> >> >> never
>> >> >> >> >> >> exposed to the destination leader. It's possible for
>> broker 1
>> >> to
>> >> >> >> first
>> >> >> >> >> get
>> >> >> >> >> >> D(1) which is truncated later when broker 0 follows a new
>> >> leader.
>> >> >> >> But
>> >> >> >> >> the
>> >> >> >> >> >> new leader will refetch the same D(1) from the source
>> cluster.
>> >> >> So,
>> >> >> >> >> it's ok
>> >> >> >> >> >> for broker 1 to keep D(1).
>> >> >> >> >> >>
>> >> >> >> >> >> JR51. Sounds good. Why do we choose to store the last
>> mirror
>> >> >> epoch
>> >> >> >> >> instead
>> >> >> >> >> >> of a last mirror offset?
>> >> >> >> >> >>
>> >> >> >> >> >> Jun
>> >> >> >> >> >>
>> >> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <
>> showuon@gmail.com>
>> >> >> wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> > Hi Jun,
>> >> >> >> >> >> >
>> >> >> >> >> >> > Thanks for the great comments again.
>> >> >> >> >> >> > Answering your question below.
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror partition
>> >> cache
>> >> >> >> lives
>> >> >> >> >> in
>> >> >> >> >> >> > MirrorMetadataManager. It is populated in two ways: by
>> >> >> replaying
>> >> >> >> >> >> > __mirror_state partitions this broker leads, and by
>> fetching
>> >> >> state
>> >> >> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> >> >> > Hmm, a given broker only caches the mirror partition
>> state
>> >> for
>> >> >> >> >> >> partitions
>> >> >> >> >> >> > hashed into the __mirror_state partition hosted on this
>> >> broker,
>> >> >> >> >> right?
>> >> >> >> >> >> Why
>> >> >> >> >> >> > does it need to fetch the state from other remote
>> >> coordinators?
>> >> >> >> >> >> >
>> >> >> >> >> >> > > a given broker only caches the mirror partition state
>> for
>> >> >> >> >> partitions
>> >> >> >> >> >> > hashed into the __mirror_state partition hosted on this
>> >> broker,
>> >> >> >> >> right?
>> >> >> >> >> >> >
>> >> >> >> >> >> > Right.
>> >> >> >> >> >> >
>> >> >> >> >> >> > > Why does it need to fetch the state from other remote
>> >> >> >> coordinators?
>> >> >> >> >> >> >
>> >> >> >> >> >> > Because the data partition leader needs to know the
>> current
>> >> >> mirror
>> >> >> >> >> state
>> >> >> >> >> >> > of the partitions it leads. On a leadership change, the
>> >> >> metadata
>> >> >> >> >> delta
>> >> >> >> >> >> is
>> >> >> >> >> >> > published to the MirrorMetadataManager, which queries the
>> >> >> current
>> >> >> >> >> mirror
>> >> >> >> >> >> > state for its led partitions from the coordinator (local
>> or
>> >> >> >> remote),
>> >> >> >> >> and
>> >> >> >> >> >> > processes tasks based on that state. This is the reason
>> why
>> >> it
>> >> >> >> >> fetches
>> >> >> >> >> >> the
>> >> >> >> >> >> > state from other remote coordinator. This part of cache
>> >> will be
>> >> >> >> >> cleared
>> >> >> >> >> >> > after the leadership change.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR36. I am still not quite sure why we need
>> >> >> MirrorLeaderEpoch in
>> >> >> >> >> the
>> >> >> >> >> >> > Fetch
>> >> >> >> >> >> > request. In the current Fetch request, in addition to
>> >> >> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> >> > it has another field LastFetchedEpoch, which captures the
>> >> last
>> >> >> >> leader
>> >> >> >> >> >> epoch
>> >> >> >> >> >> > in the data and is used to resolve the inconsistency
>> between
>> >> >> the
>> >> >> >> >> >> follower
>> >> >> >> >> >> > and the leader. Does it achieve the same purpose as
>> >> >> >> >> MirrorLeaderEpoch?
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to do
>> the
>> >> >> >> >> validation
>> >> >> >> >> >> to
>> >> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any fetched
>> >> >> batches
>> >> >> >> with
>> >> >> >> >> >> epoch
>> >> >> >> >> >> > higher than the LastFetchedEpoch, it means the follower
>> can
>> >> >> never
>> >> >> >> >> accept
>> >> >> >> >> >> > new appended batches with higher epoch in the leader
>> node.
>> >> >> >> >> >> >
>> >> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2
>> purposes in
>> >> >> the
>> >> >> >> >> fetch
>> >> >> >> >> >> > API:
>> >> >> >> >> >> > 1. The receiver (leader node) uses it to validate it
>> matches
>> >> >> the
>> >> >> >> >> leader
>> >> >> >> >> >> > epoch in the leader node
>> >> >> >> >> >> > 2. The sender (follower node) uses it to validate the
>> epoch
>> >> in
>> >> >> the
>> >> >> >> >> >> fetched
>> >> >> >> >> >> > batches are all <= this value.
>> >> >> >> >> >> >
>> >> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still serving
>> for
>> >> >> >> >> followers
>> >> >> >> >> >> in
>> >> >> >> >> >> > the destination cluster. For purpose (2), my
>> understanding
>> >> of
>> >> >> the
>> >> >> >> >> >> > validation for KAFKA-18723 is like: the follower knows
>> the
>> >> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader node
>> >> >> should
>> >> >> >> >> >> contain in
>> >> >> >> >> >> > the log batches when fetch request is sent. Any log batch
>> >> >> beyond
>> >> >> >> this
>> >> >> >> >> >> epoch
>> >> >> >> >> >> > in the fetch response means the follower’s metadata is
>> >> stale.
>> >> >> The
>> >> >> >> >> >> batches
>> >> >> >> >> >> > need to be rejected and follower needs to update metadata
>> >> for
>> >> >> next
>> >> >> >> >> >> fetch.
>> >> >> >> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch
>> >> serves
>> >> >> >> this
>> >> >> >> >> >> purpose
>> >> >> >> >> >> > to let the followers know the highest epoch in the leader
>> >> node
>> >> >> >> log in
>> >> >> >> >> >> each
>> >> >> >> >> >> > fetch response. Once the info is stale in the follower
>> >> side, it
>> >> >> >> >> updates
>> >> >> >> >> >> it
>> >> >> >> >> >> > from fetch response, and send next fetch request with
>> >> correct
>> >> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is
>> >> possible
>> >> >> >> like
>> >> >> >> >> >> > KAFKA-18723.
>> >> >> >> >> >> >
>> >> >> >> >> >> > I tried not to add new field in Fetch API, but this is
>> the
>> >> best
>> >> >> >> >> >> solution I
>> >> >> >> >> >> > can think of.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR48. "The NumPartitions field is needed so the
>> controller
>> >> >> can
>> >> >> >> >> create
>> >> >> >> >> >> the
>> >> >> >> >> >> > destination topic if it does not already exist. When a
>> >> topic is
>> >> >> >> being
>> >> >> >> >> >> > mirrored for the first time, the destination cluster has
>> no
>> >> >> >> knowledge
>> >> >> >> >> >> > of the source topic's partition count, so the request
>> must
>> >> >> carry
>> >> >> >> it."
>> >> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is started,
>> the
>> >> >> source
>> >> >> >> >> topic
>> >> >> >> >> >> > may not exist yet. When the source topic is created
>> later,
>> >> we
>> >> >> can
>> >> >> >> >> start
>> >> >> >> >> >> > mirroring without requiring the user to provide the
>> number
>> >> of
>> >> >> >> >> >> partitions.
>> >> >> >> >> >> >
>> >> >> >> >> >> > > When the source topic is created later, we can start
>> >> >> mirroring
>> >> >> >> >> without
>> >> >> >> >> >> > requiring the user to provide the number of partitions.
>> >> >> >> >> >> >
>> >> >> >> >> >> > This is because this happens in periodical sync up in the
>> >> >> >> destination
>> >> >> >> >> >> > cluster, and the numPartitions and other topic info in
>> the
>> >> >> source
>> >> >> >> >> >> cluster
>> >> >> >> >> >> > is already retrieved during the sync up process. Also, if
>> >> users
>> >> >> >> are
>> >> >> >> >> >> > creating mirror topics via admin API, this info can also
>> be
>> >> >> >> retrieved
>> >> >> >> >> >> > inside the admin client if not provided.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly need the
>> >> last
>> >> >> >> mirror
>> >> >> >> >> >> epoch
>> >> >> >> >> >> > for truncation when starting a mirror. When a mirror is
>> >> >> stopped at
>> >> >> >> >> the
>> >> >> >> >> >> > source, we already bump up the leader epoch and write
>> some
>> >> >> >> additional
>> >> >> >> >> >> > records for cleanup. It seems that we can just use the
>> >> >> >> DivergingEpoch
>> >> >> >> >> >> > returned in the initial mirror Fetch request to detect
>> the
>> >> >> >> diverging
>> >> >> >> >> >> epoch
>> >> >> >> >> >> > and use that for truncation at the mirror destination.
>> >> >> >> >> >> >
>> >> >> >> >> >> > That’s a good question. I thought about this before but
>> >> >> >> >> unfortunately it
>> >> >> >> >> >> > doesn’t work. The reason we need the last mirror epoch
>> >> record
>> >> >> is
>> >> >> >> >> because
>> >> >> >> >> >> > the destination cluster might not completely sync with
>> the
>> >> >> source
>> >> >> >> >> >> cluster
>> >> >> >> >> >> > before STOPPED. And this un-mirrored data might confuse
>> the
>> >> >> >> >> replication
>> >> >> >> >> >> > protocol when reverse mirror. For example,
>> >> >> >> >> >> >
>> >> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
>> >> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> >> >> >> >> > Offset 2: from A (LE: 10)
>> >> >> >> >> >> >
>> >> >> >> >> >> > While cluster B mirrors until offset 1, the cluster A
>> >> crashes
>> >> >> and
>> >> >> >> >> >> failover
>> >> >> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
>> >> >> >> MIRROR_PID_RESET
>> >> >> >> >> >> with
>> >> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from
>> >> cluster B,
>> >> >> >> the
>> >> >> >> >> log
>> >> >> >> >> >> > will be like this:
>> >> >> >> >> >> >
>> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> >> > foo-0 leader log: foo-0 leader log:
>> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE:
>> 10)
>> >> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
>> >> >> >> >> >> >
>> >> >> >> >> >> > As you can see, cluster A (new destination cluster)
>> doesn’t
>> >> >> know
>> >> >> >> the
>> >> >> >> >> >> > offset 2 is not belonging to the cluster B, which causes
>> >> >> >> inconsistent
>> >> >> >> >> >> logs.
>> >> >> >> >> >> > So like what we described in the KIP, the last mirror
>> epoch
>> >> is
>> >> >> the
>> >> >> >> >> >> > synchronization point between source and destination. Any
>> >> data
>> >> >> >> beyond
>> >> >> >> >> >> the
>> >> >> >> >> >> > point is treated as unknown to the source cluster and
>> >> should be
>> >> >> >> >> >> truncated.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR52. Stop Mirror:
>> >> >> >> >> >> > > JR52.1 "3. If patterns are provided, the controller
>> >> removes
>> >> >> >> >> matching
>> >> >> >> >> >> > entries from mirror.topics.include or adds them to
>> >> >> >> >> >> mirror.topics.exclude on
>> >> >> >> >> >> > the CLUSTER_MIRROR resource."
>> >> >> >> >> >> > Why is it 'or' and not 'and'?
>> >> >> >> >> >> >
>> >> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> > discovers new source topics matching the persisted
>> >> >> include/exclude
>> >> >> >> >> >> patterns
>> >> >> >> >> >> > and repeats steps 3 through 7 for each."
>> >> >> >> >> >> > The broker will send a create topic request to the
>> >> controller.
>> >> >> How
>> >> >> >> >> does
>> >> >> >> >> >> the
>> >> >> >> >> >> > controller associate it with the mirror name to create a
>> >> >> >> >> >> > `MirrorTopicStateChangeRecord`?
>> >> >> >> >> >> >
>> >> >> >> >> >> > No, we don’t send a create topic request to the
>> controller
>> >> >> here.
>> >> >> >> >> >> Actually
>> >> >> >> >> >> > we send a StartMirrorTopics request to the controller,
>> which
>> >> >> >> contains
>> >> >> >> >> >> the
>> >> >> >> >> >> > mirror name and topic metadata like NumPartitions,
>> topicId…
>> >> >> etc.
>> >> >> >> So
>> >> >> >> >> the
>> >> >> >> >> >> > controller can create the non-existed topic as explained
>> in
>> >> >> JR48,
>> >> >> >> and
>> >> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR53. Describe Mirrors:
>> >> >> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors
>> request
>> >> >> with
>> >> >> >> >> >> optional
>> >> >> >> >> >> > mirror names (empty means all mirrors)."
>> >> >> >> >> >> > In metadata request, we use a null arrary to represent
>> all
>> >> >> >> topics. It
>> >> >> >> >> >> would
>> >> >> >> >> >> > be useful to be consistent here.
>> >> >> >> >> >> >
>> >> >> >> >> >> > OK, KIP updated.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
>> >> >> brokers
>> >> >> >> and
>> >> >> >> >> >> > merges
>> >> >> >> >> >> > the responses client-side."
>> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
>> >> partition.
>> >> >> So
>> >> >> >> >> sending
>> >> >> >> >> >> > the request to every broker is wasteful. It's probably
>> >> better
>> >> >> to
>> >> >> >> send
>> >> >> >> >> >> the
>> >> >> >> >> >> > request to a single broker, which can then contact the
>> right
>> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >> >
>> >> >> >> >> >> > Agree, KIP updated.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are
>> >> >> MIRRORING,
>> >> >> >> >> >> PAUSED,
>> >> >> >> >> >> > STOPPED the only validate states? The state table has
>> other
>> >> >> states
>> >> >> >> >> too.
>> >> >> >> >> >> > Ditto for MirrorPartitionState.
>> >> >> >> >> >> > The DesiredState field in MirrorTopicStateChangeRecord
>> is to
>> >> >> >> define
>> >> >> >> >> the
>> >> >> >> >> >> > target state the user wants. And the desired state users
>> can
>> >> >> >> assign
>> >> >> >> >> is
>> >> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired
>> >> state
>> >> >> as
>> >> >> >> >> >> STOPPING
>> >> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
>> >> intermediate
>> >> >> >> >> states.
>> >> >> >> >> >> >
>> >> >> >> >> >> > About MirrorPartitionState record, it’ll record any
>> state in
>> >> >> the
>> >> >> >> >> state
>> >> >> >> >> >> > table that the current partition is. This is the real
>> state
>> >> for
>> >> >> >> each
>> >> >> >> >> >> > partition. It’s like in the kubernetes world, when we
>> change
>> >> >> the
>> >> >> >> >> replica
>> >> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired
>> state), it
>> >> >> won’t
>> >> >> >> >> >> increase
>> >> >> >> >> >> > to 5 immediately. The current replica
>> (MirrorPartitionState
>> >> >> >> record)
>> >> >> >> >> is
>> >> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > Thank you,
>> >> >> >> >> >> > Luke
>> >> >> >> >> >> >
>> >> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
>> >> >> >> dev@kafka.apache.org
>> >> >> >> >> >
>> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >
>> >> >> >> >> >> >> Hi, Fede,
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> Thanks for the reply. A few more comments.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror partition
>> cache
>> >> >> >> lives in
>> >> >> >> >> >> >> MirrorMetadataManager. It is populated in two ways: by
>> >> >> replaying
>> >> >> >> >> >> >> __mirror_state partitions this broker leads, and by
>> >> fetching
>> >> >> >> state
>> >> >> >> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> >> >> >> Hmm, a given broker only caches the mirror partition
>> state
>> >> for
>> >> >> >> >> >> partitions
>> >> >> >> >> >> >> hashed into the __mirror_state partition hosted on this
>> >> >> broker,
>> >> >> >> >> right?
>> >> >> >> >> >> Why
>> >> >> >> >> >> >> does it need to fetch the state from other remote
>> >> >> coordinators?
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR36. I am still not quite sure why we need
>> >> MirrorLeaderEpoch
>> >> >> in
>> >> >> >> the
>> >> >> >> >> >> Fetch
>> >> >> >> >> >> >> request. In the current Fetch request, in addition to
>> >> >> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> >> >> it has another field LastFetchedEpoch, which captures
>> the
>> >> last
>> >> >> >> >> leader
>> >> >> >> >> >> >> epoch
>> >> >> >> >> >> >> in the data and is used to resolve the inconsistency
>> >> between
>> >> >> the
>> >> >> >> >> >> follower
>> >> >> >> >> >> >> and the leader. Does it achieve the same purpose as
>> >> >> >> >> MirrorLeaderEpoch?
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR48. "The NumPartitions field is needed so the
>> controller
>> >> can
>> >> >> >> >> create
>> >> >> >> >> >> the
>> >> >> >> >> >> >> destination topic if it does not already exist. When a
>> >> topic
>> >> >> is
>> >> >> >> >> being
>> >> >> >> >> >> >> mirrored for the first time, the destination cluster
>> has no
>> >> >> >> >> knowledge
>> >> >> >> >> >> >> of the source topic's partition count, so the request
>> must
>> >> >> carry
>> >> >> >> >> it."
>> >> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is started,
>> the
>> >> >> >> source
>> >> >> >> >> >> topic
>> >> >> >> >> >> >> may not exist yet. When the source topic is created
>> later,
>> >> we
>> >> >> can
>> >> >> >> >> start
>> >> >> >> >> >> >> mirroring without requiring the user to provide the
>> number
>> >> of
>> >> >> >> >> >> partitions.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need the
>> >> last
>> >> >> >> mirror
>> >> >> >> >> >> epoch
>> >> >> >> >> >> >> for truncation when starting a mirror. When a mirror is
>> >> >> stopped
>> >> >> >> at
>> >> >> >> >> the
>> >> >> >> >> >> >> source, we already bump up the leader epoch and write
>> some
>> >> >> >> >> additional
>> >> >> >> >> >> >> records for cleanup. It seems that we can just use the
>> >> >> >> >> DivergingEpoch
>> >> >> >> >> >> >> returned in the initial mirror Fetch request to detect
>> the
>> >> >> >> diverging
>> >> >> >> >> >> epoch
>> >> >> >> >> >> >> and use that for truncation at the mirror destination.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR52. Stop Mirror:
>> >> >> >> >> >> >> JR52.1 "3. If patterns are provided, the controller
>> removes
>> >> >> >> matching
>> >> >> >> >> >> >> entries from mirror.topics.include or adds them to
>> >> >> >> >> >> mirror.topics.exclude
>> >> >> >> >> >> >> on
>> >> >> >> >> >> >> the CLUSTER_MIRROR resource."
>> >> >> >> >> >> >> Why is it 'or' and not 'and'?
>> >> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> >> discovers new source topics matching the persisted
>> >> >> >> include/exclude
>> >> >> >> >> >> >> patterns
>> >> >> >> >> >> >> and repeats steps 3 through 7 for each."
>> >> >> >> >> >> >> The broker will send a create topic request to the
>> >> controller.
>> >> >> >> How
>> >> >> >> >> does
>> >> >> >> >> >> >> the
>> >> >> >> >> >> >> controller associate it with the mirror name to create a
>> >> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR53. Describe Mirrors:
>> >> >> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors
>> request
>> >> >> with
>> >> >> >> >> >> optional
>> >> >> >> >> >> >> mirror names (empty means all mirrors)."
>> >> >> >> >> >> >> In metadata request, we use a null arrary to represent
>> all
>> >> >> >> topics.
>> >> >> >> >> It
>> >> >> >> >> >> >> would
>> >> >> >> >> >> >> be useful to be consistent here.
>> >> >> >> >> >> >> JR53.2 "2. The admin client fans out the request to all
>> >> >> brokers
>> >> >> >> and
>> >> >> >> >> >> merges
>> >> >> >> >> >> >> the responses client-side."
>> >> >> >> >> >> >> This is not ideal. Not every broker hosts a mirror
>> >> partition.
>> >> >> So
>> >> >> >> >> >> sending
>> >> >> >> >> >> >> the request to every broker is wasteful. It's probably
>> >> better
>> >> >> to
>> >> >> >> >> send
>> >> >> >> >> >> the
>> >> >> >> >> >> >> request to a single broker, which can then contact the
>> >> right
>> >> >> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are
>> >> >> MIRRORING,
>> >> >> >> >> PAUSED,
>> >> >> >> >> >> >> STOPPED the only validate states? The state table has
>> other
>> >> >> >> states
>> >> >> >> >> too.
>> >> >> >> >> >> >> Ditto for MirrorPartitionState.
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> Jun
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
>> >> >> >> >> fedevaleri@gmail.com>
>> >> >> >> >> >> >> wrote:
>> >> >> >> >> >> >>
>> >> >> >> >> >> >> > Hi Andrew, yes, validation is called at the start of
>> >> >> >> createTopic.
>> >> >> >> >> If
>> >> >> >> >> >> >> > the topic ID is invalid, the request fails with
>> >> >> >> INVALID_REQUEST.
>> >> >> >> >> If
>> >> >> >> >> >> >> > the topic ID is already used by a different topic
>> name,
>> >> it
>> >> >> >> fails
>> >> >> >> >> with
>> >> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the
>> individual
>> >> >> topic
>> >> >> >> in
>> >> >> >> >> the
>> >> >> >> >> >> >> > batch fails; other topics in the same CreateTopics
>> >> request
>> >> >> >> >> continue
>> >> >> >> >> >> >> > processing normally. Added this information to the
>> KIP.
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
>> >> >> >> >> >> >> schofielaj@apache.org>
>> >> >> >> >> >> >> > wrote:
>> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> > > Hi Fede,
>> >> >> >> >> >> >> > > One more small question.
>> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> > > AS37: If the controller fails validation for the
>> >> >> >> >> MirrorInfo.TopicId
>> >> >> >> >> >> >> > added to the CreateTopic request, what does it do? I
>> >> suspect
>> >> >> >> that
>> >> >> >> >> it
>> >> >> >> >> >> >> fails
>> >> >> >> >> >> >> > the request with a particular error code.
>> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> > > Thanks,
>> >> >> >> >> >> >> > > Andrew
>> >> >> >> >> >> >> > >
>> >> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> >> >> >> >> >> >> > > > Hi all,
>> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> > > > We updated the KIP to include Coordinator Runtime
>> >> >> >> >> configurations
>> >> >> >> >> >> and
>> >> >> >> >> >> >> > > > the coordinator state transition validation
>> >> paragraph.
>> >> >> The
>> >> >> >> >> latter
>> >> >> >> >> >> >> > > > covers the per-partition epoch fencing mechanism
>> >> (leader
>> >> >> >> epoch
>> >> >> >> >> >> and
>> >> >> >> >> >> >> > > > state epoch) that complements the existing
>> per-topic
>> >> >> >> >> StateOffset
>> >> >> >> >> >> >> > > > fencing for lifecycle operations.
>> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> > > > Thanks,
>> >> >> >> >> >> >> > > > Fede
>> >> >> >> >> >> >> > > >
>> >> >> >> >> >> >> >
>> >> >> >> >> >> >>
>> >> >> >> >> >> >
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for the good comments,
Answering them below.
> JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
supports regex in the ClusterMirrorComamnd, but we don’t need to add the
“patterns”."
So, in this case, we use topics to support both specific topics and regex.
However, startMirrorTopicsOptions uses topics for specific topics and
includePatterns for regex. Could we make them consistent?
The purpose of the “includePatterns” field is for the
“mirror.topics.include” update, so that if there is any newly created topic
in the source cluster that matches the regex, we can create it. And the
“Topics” field in the StartMirrorTopicsRequest is to specify the topicId
and numPartitions in the source cluster. So I think “includePatterns” and
“Topics” both are necessary in the StartMirrorTopicsRequest. On the other
hand, in PauseMirrorTopics, because it won’t change the
“mirror.topics.include”/”mirror.topics.exclude” configs, we don’t need to
provide the regex in the request.
> JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also -1 if
it's not included in the request?
If “IncludeMirrorOffset” is disabled, we won’t query the mirror partition
leader, so we can’t get the sourceOffset.
I assume you want to ask `if it’s still -1 if not actively mirroring, while
“IncludeMirrorOffset” is enabled`. That’s a good question. I think our
purpose of this command is to show the lag to users. So if the state is not
MIRRORING (ex: STOPPED/PAUSED), we think it’s not helpful to users to show
only the sourceOffset. Of course users can use `kafka-get-offsets.sh` to
get the offset if they really want to know.
> JR83. Start Mirror Topics:
> JR83.1 Should the controller verify the expected state for the topic?
As replied in JR84, the desired state validation should happen in the
broker. And the controller validates the StateOffset. I’ve updated the
workflow to make it clear.
> JR83.2 "d. truncates the log accordingly,"
Could you add a description of the truncation offset?
OK, added one more sentence:
d. based on the last mirror epoch response in (b), the broker translates
the epoch into the offset by querying the leaderEpochCache
(endOffsetForEpoch), and truncates the log accordingly, and waits for all
ISR members…
> JR84. Stop Mirror Topics: "The controller verifies that the desired state
of all topics are in MIRRORING or PAUSED state."
Should the broker verfy the topic level state? I thought the controller
will use StateOffset in the request to make sure the validation on the
broker is stil valid. If the controller performs the verification, is
StateOffset still useful?
Oh, you’re right. We’re doing duplicated work here. Yes, the broker already
verifies the “desired state” of all topics are in MIRRORING or PAUSED
state, as well as the “actual state” is in MIRRORING or PAUSED state. If
all validation passes, we’ll forward it to the controller. So the
controller only needs to validate the StateOffset to make sure the
validation on the broker is still valid. So, yes, we don’t need the desired
state validation on the controller side. I’ve updated the KIP to make it
clear.
One more thing to add is that, I’ve thought about not adding StateOffset,
and relying on the desired state validation to make sure we’re safe to do
the deletion or other operations. The problem is that, even if the broker
sees the desired state for topic A is STOPPED, and then the controller also
sees the desired state is STOPPED, it still cannot make sure the
controller’s STOPPED state equals the broker’s STOPPED state. For example:
1. Broker sees STOPPED, then forward to controller
2. User changes the desired state to MIRRORING
3. User changes the desired state to STOPPED
4. Controller sees the desired state is STOPPED
So I think StateOffset is required.
Thanks for catching it BTW.
> JR85. There are still a few references to includeMirrorLags. They should
be
IncludeMirrorOffset.
Sorry, just updated.
> JR86. Metrics:
> JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
KIP is updated. I also group the metrics into new added metrics and other
inherited metrics.
> JR86.2 Could you describe a bit more what MinFetchRate represents?
It means the lowest fetch request rate among all of this mirror fetcher
manager's fetcher threads. It inherits from the AbstractFetcherManager’s
metric.
> JR87. Version Compatibility: "For migration or DR use cases where failback
is not required, this proposal supports source brokers up to version 2.1
included"
This KIP needs topicId and KIP-516 was introduced later than 2.1.
Yes, when mirroring from a cluster without topic id support, we will assign
a random topic ID in the destination cluster and then fetch (without topic
id in the fetch request using the old version of Fetch API) as usual. Of
course this is for the one-way mirroring use case. It can’t start the
reverse mirror(failback) under this setup. You can check the “Compatibility
Matrix” section for more detail.
Thank you,
Luke
On Fri, Aug 14, 2026 at 7:09 AM Jun Rao via dev <de...@kafka.apache.org>
wrote:
> Hi, Luke,
>
> Thanks for the reply.
>
> JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
> supports regex in the ClusterMirrorComamnd, but we don’t need to add the
> “patterns”."
> So, in this case, we use topics to support both specific topics and regex.
> However, startMirrorTopicsOptions uses topics for specific topics and
> includePatterns for regex. Could we make them consistent?
>
> JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also -1 if
> it's not included in the request?
>
> JR83. Start Mirror Topics:
> JR83.1 Should the controller verify the expected state for the topic?
> JR83.2 "d. truncates the log accordingly,"
> Could you add a description of the truncation offset?
>
> JR84. Stop Mirror Topics: "The controller verifies that the desired state
> of all topics are in MIRRORING or PAUSED state."
> Should the broker verfy the topic level state? I thought the controller
> will use StateOffset in the request to make sure the validation on the
> broker is stil valid. If the controller performs the verification, is
> StateOffset still useful?
>
> JR85. There are still a few references to includeMirrorLags. They should be
> IncludeMirrorOffset.
>
> JR86. Metrics:
> JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
> JR86.2 Could you describe a bit more what MinFetchRate represents?
>
> JR87. Version Compatibility: "For migration or DR use cases where failback
> is not required, this proposal supports source brokers up to version 2.1
> included"
> This KIP needs topicId and KIP-516 was introduced later than 2.1.
>
> Jun
>
>
> On Thu, Aug 13, 2026 at 5:14 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the good comments.
> > Answering them below.
> >
> > > JR59.1 Could you add the RPC to the workflow? The broker first needs to
> > issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
> > mirror name and then issue DescribeClusterMirrors to get the LME.
> >
> > Yes, you’re right. The workflow is updated to make them clear.
> >
> > > JR71. mirror.support.unclean.leader.election : The name is confusing.
> > It's
> > also used in LOG_TRUNCATION, which is not related to unclean leader
> > election. Could we use a better name?
> >
> > I personally think the name should be fine because even though we use in
> > “LOG_TRUNCATION” state, the config is to allow us to know if users want
> to
> > support unclean leader election, if so, we wait for all replicas to catch
> > up. If we want to explicitly show the intention of the config, how about
> > “mirror.full.replica.convergence.required”? I’m open to other
> suggestions.
> >
> >
> > > JR72. DescribeClusterMirrors
> > > JR72.1 A user may specify just the topic name when describing a mirror.
> > Should we include the topic name in the request?
> >
> > Hmm.. we didn’t think about the use case. But yes, if users only care
> > about the state of a topic, they don’t need the info for other topics.
> I’ve
> > combined with JR72.2, added “TopicPartitions” filed into
> > DescribeClusterMirrorsRequest, and by default is null, which means all
> > topic partitions under the requested mirrors will be returned. If set,
> > we’ll query the mirrors and return the topic partitions that users
> > expected. KIP is updated.
> >
> >
> > > JR72.2 LastMirrorEpochLookups is a bit weird since the response returns
> > either mirror state or mirror lag and LastMirrorEpochLookups doesn't
> match
> > either.
> >
> > Yes, after the recent refactor, I think the LastMirrorEpochLookups can be
> > merged into one level up, combining with the topic name filter in above
> > comment, which will allow users to query down to the partition level. The
> > LastMirrorEpochLookups will query along with normal mirror state by the
> > readMirrorState from the coordinator. The difference is that after
> > receiving the response, it will filter out the mirrors that don’t match
> the
> > requested sourceClusterId, and then add the lastMirrorEpoch into
> > DescribeClusterMirrorResponse. KIP is updated.
> >
> >
> > > JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >
> > Right. Added.
> >
> >
> > > JR72.4 "includeMirrorLags: source offset, destination offset, lag"
> > DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
> > should be omitted.
> >
> > Sorry, I removed the “SourceOffset” and “DestinationOffset” fields from
> > DescribeClusterMirrorsResponse because I added them into
> ReadMirrorOffsets
> > (ex-ReadMirrorLags) API. But I forgot we still need them in the
> > DescribeClusterMirrorsResponse so that the adminClient can receive the
> > results. KIP is updated.
> >
> >
> > > JR73. MirrorPartitionState record, LastMirrorEpochs record and
> > MirrorPartitionStateRequest: The values for type and name are incorrect.
> >
> > Ah, that’s my bad. Fixed.
> >
> > > JR74. ListClusterMirrorsRequest
> > > JR74.1 When is DesiredMirrorStateFilter used? There is no option to
> > specify
> > the desired state in the mirror-topic command.
> >
> > Yes, referred to kafka-consumer-groups.sh, I added the `--state` option
> to
> > allow users to set the state when listMirror.
> >
> > > JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
> >
> > Yes, KIP updated.
> >
> > > JR75. ReadMirrorLags
> > > JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
> > include lag?
> >
> > Indeed. Updated.
> >
> > > JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >
> > Yes, added
> >
> > > JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to
> be
> > consistent.
> >
> > Thanks for pointing it out. KIP is updated.
> >
> > > JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
> >
> > Yes, we should validate the mirror name before writing the data. KIP is
> > updated.
> >
> > > JR77. Delete Mirror: "3. The controller verifies that no topics are in
> > MIRRORING or PAUSED state."
> > Is this necessary since step 2 has verified all partitions are in STOPPED
> > state, which can only happen if the topics are in STOPPED state.
> >
> > It’s true, but there could be some delay between “actual state” and
> > “desired state”. In step (2), we verify all partitions’ “actual state”
> are
> > in STOPPED state. But the current desired state could be MIRRORING, it’s
> > just that the mirror partition leaders haven’t got the metadata update
> yet.
> > So, the controller still needs to verify the desired state is not in
> > MIRRORING or PAUSED. I found we didn’t specify that the controller is
> > actually verifying the “desired state”, not “actual state”. And this
> > controller’s desired state validation also applies in
> > startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
> > I’ve updated the KIP to make it clear.
> >
> > > JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
> > bumped,
> > ABORT markers are appended for all ongoing transactions, last mirror
> epochs
> > are persisted, and a MIRROR_PID_RESET record appended."
> > This describes the state for STOPPED, not STOPPING.
> >
> > No, these are the tasks during STOPPING state, like what we described in
> > “Stop Mirror Topics” workflow. If we do these in STOPPED state, we can’t
> > safely move the readonly topic into writable because not all required
> tasks
> > are done. I think I know why it confuses you, it’s because all the tasks
> > are in the completed state. I’ve rephrased it to make it clear.
> >
> >
> > > JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
> > timeout field?
> >
> > Make sense. Added into `DescribeClusterMirrors` request, as well as
> >
> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
> > because we will query the partition states from coordinator for state
> > validation in these APIs.
> >
> >
> > > JR80. PauseMirrorTopicsOptions: Should it support patterns? The command
> > line seems to suppor it.
> >
> > Sorry, it’s a little confusing here. The “pattern” in
> > startMirrorTopicsOptions or stopMirrorTopicsOptions is to update the
> > “mirror.topics.include” and “mirror.topics.exclude” configuration for
> topic
> > pattern discovery periodically. For the PauseMirrorTopics and
> > ResumeMirrorTopics in the clusterMirrorCommand, users can provide regex
> in
> > the “--topics” option, and the clusterMirrorCommand will listTopic in the
> > destination cluster and match with the patterns users provided, and then
> > feeds the matched topics into the PauseMirrorTopics/ResumeMirrorTopics
> > requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics supports
> > regex in the ClusterMirrorComamnd, but we don’t need to add the
> “patterns”.
> >
> >
> > > JR81. AdminClient
> > > JR81.1 ListClusterMirrors: We need to add the option to specify
> different
> > filters supported in the request.
> >
> > We already have the `--mirror` option. Added `--state` and
> > `--sourceClusterId` options to support the other filters.
> >
> > > JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
> > request doesn't support this option. Could we add other options supported
> > in the request?
> >
> > Currently, we have “includeAuthorizedOperations” in DescribeCluster,
> > DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
> > DescribeStreamsGroups requests. But none of them include the option in
> the
> > tool to allow users to enable it. I think the reason is that we don’t
> want
> > to show this to users on the tool output. But having this field, we allow
> > the advanced users to invoke admin API with includeAuthorizedOperations
> > enabled to get the results they want.
> >
> >
> > > JR81.3 LeaderStateDescription: Should it be named
> > MirrorPartitionStateDescription? Also, could we make the fields
> consistent
> > with those in the response?
> >
> > OK. KIP is updated.
> >
> >
> > Thank you,
> > Luke
> >
> > On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Luke,
> >>
> >> Thanks for the reply.
> >>
> >> JR59.1 Could you add the RPC to the workflow? The broker first needs to
> >> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
> >> mirror name and then issue DescribeClusterMirrors to get the LME.
> >>
> >> JR71. mirror.support.unclean.leader.election : The name is confusing.
> It's
> >> also used in LOG_TRUNCATION, which is not related to unclean leader
> >> election. Could we use a better name?
> >>
> >> JR72. DescribeClusterMirrors
> >> JR72.1 A user may specify just the topic name when describing a mirror.
> >> Should we include the topic name in the request?
> >> JR72.2 LastMirrorEpochLookups is a bit weird since the response returns
> >> either mirror state or mirror lag and LastMirrorEpochLookups doesn't
> match
> >> either.
> >> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> JR72.4 "includeMirrorLags: source offset, destination offset, lag"
> >> DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
> >> should be omitted.
> >>
> >> JR73. MirrorPartitionState record, LastMirrorEpochs record and
> >> MirrorPartitionStateRequest: The values for type and name are incorrect.
> >>
> >> JR74. ListClusterMirrorsRequest
> >> JR74.1 When is DesiredMirrorStateFilter used? There is no option to
> >> specify
> >> the desired state in the mirror-topic command.
> >> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
> >>
> >> JR75. ReadMirrorLags
> >> JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
> >> include lag?
> >> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> >> JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to be
> >> consistent.
> >>
> >> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
> >>
> >> JR77. Delete Mirror: "3. The controller verifies that no topics are in
> >> MIRRORING or PAUSED state."
> >> Is this necessary since step 2 has verified all partitions are in
> STOPPED
> >> state, which can only happen if the topics are in STOPPED state.
> >>
> >> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
> >> bumped,
> >> ABORT markers are appended for all ongoing transactions, last mirror
> >> epochs
> >> are persisted, and a MIRROR_PID_RESET record appended."
> >> This describes the state for STOPPED, not STOPPING.
> >>
> >> JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
> >> timeout field?
> >>
> >> JR80. PauseMirrorTopicsOptions: Should it support patterns? The command
> >> line seems to suppor it.
> >>
> >> JR81. AdminClient
> >> JR81.1 ListClusterMirrors: We need to add the option to specify
> different
> >> filters supported in the request.
> >> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
> >> request doesn't support this option. Could we add other options
> supported
> >> in the request?
> >> JR81.3 LeaderStateDescription: Should it be named
> >> MirrorPartitionStateDescription? Also, could we make the fields
> consistent
> >> with those in the response?
> >>
> >> Jun
> >>
> >> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thanks for the good comments again.
> >> > Answering them below.
> >> >
> >> > > JR51. "ULE_LOG_TRUNCATION When
> mirror.support.unclean.leader.election
> >> is
> >> > enabled and the destination leader has log truncation, the state will
> >> move
> >> > from MIRRORING to this state."
> >> > This still suggests that divergeEpoch is only handled if
> >> > mirror.support.unclean.leader.election
> >> > is enabled.
> >> >
> >> >
> >> > OK, KIP is updated to make it clear.
> >> >
> >> > > JR55. Should the mirror state and the mirror lags always be exposed
> >> > together? If so, we can remove IncludeMirrorState and
> IncludeMirrorLags
> >> > from DescribeClusterMirrorsRequest. If not, it's better to use two
> >> separate
> >> > RPCs.
> >> >
> >> >
> >> > No, we sometimes only need MirrorState, and sometimes need both. So
> I’ll
> >> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
> >> > DescribeClusterMirrorsRequest. And we will use ReadMirrorStates and
> >> > ReadMirrorLags to get each info.
> >> >
> >> >
> >> > > DescribeClusterMirrorsRequest should be used in the Describe Mirror
> >> flow,
> >> > right? Currently it's missing. Also, I assume the admin client refers
> to
> >> > the one in the kafka-cluster-mirrors tool. But it seems that sometimes
> >> it
> >> > also refers to the admin client on the broker? Perhaps we can use 'the
> >> > tool' instead to make it clear.
> >> >
> >> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
> >> > And sorry for the confusion. I’d like to keep the admin client as the
> >> one
> >> > invoking from the tool (or user’s direct admin client API invocation).
> >> On
> >> > the broker side, we actually use InterBrokerSendThread to invoke these
> >> RPCs
> >> > or get directly from MetadataImage. I’ve updated the KIP to make it
> >> clear.
> >> >
> >> >
> >> > > JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
> >> > instead of just to the controller?
> >> >
> >> > Yes, we only need to issue to the controller. Updated to the
> >> > BumpLeaderEpochs API.
> >> >
> >> >
> >> > > JR59. Start Mirror Topics:
> >> > > JR59.1 "During this state, the broker queries the source cluster for
> >> last
> >> > mirror epochs".
> >> > Could you describe which RPC is issued to find the LME?
> >> >
> >> > We’re using describeClusterMirror API via admin client here.
> >> >
> >> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
> >> > doesn't
> >> > contain a replication factor.
> >> >
> >> > We use “default.replication.factor” when topic creation, so the
> >> > INVALID_REPLICATION_FACTOR will be thrown like normal topic creation.
> >> Also,
> >> > because we require the replicas for the partition assignment to be all
> >> > active and not in fenced or controlled shutdown(JR35), we’ll throw
> >> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
> >> >
> >> >
> >> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
> >> > different topic name"
> >> > Should we also return this error when the topic name is used as a
> >> regular
> >> > topic?
> >> >
> >> > There are 2 errors we will throw when topic id <-> topic name
> mismatch.
> >> > Suppose provided topic id: ID_A, topic name: NAME_A
> >> > 1. Destination cluster exists topic id: ID_A, but topic name is:
> NAME_B
> >> > TOPIC_ALREADY_EXISTS will be thrown
> >> > 2. Destination cluster exists topic id: ID_B, but topic name is:
> NAME_A
> >> > INCONSISTENT_TOPIC_ID will be thrown
> >> >
> >> > Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS error
> >> for
> >> > these 2 cases makes more sense. KIP updated.
> >> >
> >> >
> >> > > JR60. CreateClusterMirrorRequest: When will it receive an
> >> > INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> > CLUSTER_MIRROR_ALREADY_EXISTS?
> >> >
> >> >
> >> > Sorry, createClusterMirrorRequest doesn’t throw
> >> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate the
> >> mirror
> >> > states when creating a cluster mirror. We only care if the “mirror
> name”
> >> > exists or not. KIP updated.
> >> >
> >> >
> >> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
> >> remove
> >> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
> >> > specific state causing the error.
> >> >
> >> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following errors:
> >> > a. MIRROR_TOPIC_ALREADY_PAUSED
> >> > b. MIRROR_TOPIC_NOT_PAUSED,
> >> > c. MIRROR_TOPIC_ALREADY_STOPPED,
> >> > d. CLUSTER_MIRROR_NOT_EMPTY
> >> > e. MIRROR_TOPIC_NOT_STOPPED
> >> >
> >> > KIP is updated.
> >> >
> >> >
> >> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
> >> requests
> >> > (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
> Could
> >> we
> >> > make them consistent?
> >> >
> >> >
> >> > Good catch. The updated errors for the new added RPC are this:
> >> >
> >> > CreateClusterMirror: NOT_CONTROLLER
> >> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >> >
> >> >
> >> > > JR63. StopMirrorTopics:
> >> > > JR63.1 "The broker validates that all target topic partitions are in
> >> > either
> >> > LOG_TRUNCATION or MIRRORING or FAILED state."
> >> > Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
> >> >
> >> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is
> updated.
> >> >
> >> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
> >> PAUSED
> >> > -> STOPPED is valid?
> >> >
> >> > Yes, it is not correct. KIP updated to remove this error.
> >> >
> >> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
> >> > state"
> >> > This is weird. The reason to issue StopMirrorTopics is that it's not
> in
> >> > stopped state.
> >> >
> >> > Indeed. KIP is updated to only include INVALID_CLUSTER_MIRROR_STATES
> >> > error, and will be thrown when states are not in MIRRORING or PAUSED
> >> state.
> >> >
> >> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
> >> derived
> >> > from SourceOffset and DestinationOffset.
> >> >
> >> > True. Removed.
> >> >
> >> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that
> we
> >> > could just set the partition level state to STOPPED.
> >> >
> >> > Nice catch. We don’t use it at all. That is some old leftover field we
> >> > forgot to remove. KIP updated.
> >> >
> >> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be
> sent
> >> to
> >> > the controller?
> >> >
> >> > No, these 2 only send to the broker. KIP updated.
> >> >
> >> >
> >> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch?
> >> It
> >> > seems both are used for fencing and fencing based on the latter is
> >> > stronger.
> >> >
> >> > Yes, basically the StateEpoch can fence for most cases. The
> LeaderEpoch
> >> > fencing is used for cases like:
> >> >
> >> > 1. b1 is the current leader for foo-0, preparing write state from
> >> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
> >> > 2. leadership change to b2 for foo-0, LE changes to 2.
> >> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1 }.
> >> > 4. The coordinator receives the read from b2 in step (3), the highest
> >> > lastSeen LE for foo-0 updates to 2 in coordinator.
> >> > 5. The writes state from b1 in step (1) arrives the coordinator, but
> it
> >> > rejects b1 writes because the highest lastSeen LE is 2, which is > 1
> in
> >> the
> >> > request.
> >> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2, stateEpoch: 1
> and
> >> > succeeds.
> >> >
> >> > The leaderEpoch fence is to avoid any stale broker writes after the
> new
> >> > leader epoch read. This cannot be protected by the stateEpoch. The
> >> > stateEpoch can prevent the write with older stateEpoch. So in the
> >> situation
> >> > above, the stateEpoch cannot determine if we should accept b1 writes
> or
> >> > not, but leaderEpoch can. By the way, we borrowed this concept from
> >> > KIP-932:
> >> >
> >>
> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
> >> <
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!u8EkPGXm8_VKdltXy8lrIW14mR2kgYIoaBPD7LO2uAN6rjhrfCX37Pfa6r85QPpSwSPh4V6eW_bKlns$
> >
> >> > <
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
> >> >
> >> >
> >> >
> >> > > JR68. Could we make the usage of topicId and topicName in the RPCs
> >> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> > BumpLeaderEpochsReponse uses TopicName.
> >> >
> >> > Yes, we have made:
> >> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use TopicName.
> >> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse use
> >> > TopicName.
> >> > 3. MirrorTopicStateChangeRecord uses TopicName because we will do
> >> > validation before writing the metadata log.
> >> >
> >> >
> >> > > JR69. MIRROR_PID_RESET:
> >> > > JR69.1 Could you specify the value of the control record's type?
> >> >
> >> > It’s MIRROR_PID_RESET(7). KIP updated.
> >> >
> >> > > JR69.2 How is the SourceClusterId field used?
> >> >
> >> > I’m also surprised to see this field in the MirrorPidResetRecord. I
> >> traced
> >> > back to the internal PR we added for this field, and confirmed it is
> not
> >> > used since it was added. Sorry for the confusion and nice catch! KIP
> >> > updated.
> >> >
> >> > > JR70. MirrorPartitionState: type should be "data" not
> >> "coordinator-key",
> >> > right? Ditto for LastMirrorEpochs.
> >> > We store both MirrorPartitionState and LastMirrorEpochs records into
> >> > internal __mirror_state topic, and rely on Coordinator Runtime to
> >> > write/read the records, like how `OffsetCommit` and `GroupMetadata`
> >> work.
> >> > So I think we should keep them as
> "coordinator-key"/"coordinator-value"
> >> > type.
> >> >
> >> >
> >> > Thank you,
> >> > Luke
> >> >
> >> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <dev@kafka.apache.org
> >
> >> > wrote:
> >> >
> >> >> Hi, Luke,
> >> >>
> >> >> Thanks for the reply.
> >> >>
> >> >> JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election
> >> is
> >> >> enabled and the destination leader has log truncation, the state will
> >> move
> >> >> from MIRRORING to this state."
> >> >> This still suggests that divergeEpoch is only handled if
> >> >> mirror.support.unclean.leader.election
> >> >> is enabled.
> >> >>
> >> >> JR55. Should the mirror state and the mirror lags always be exposed
> >> >> together? If so, we can remove IncludeMirrorState and
> IncludeMirrorLags
> >> >> from DescribeClusterMirrorsRequest. If not, it's better to use two
> >> >> separate
> >> >> RPCs.
> >> >>
> >> >> DescribeClusterMirrorsRequest should be used in the Describe Mirror
> >> flow,
> >> >> right? Currently it's missing. Also, I assume the admin client refers
> >> to
> >> >> the one in the kafka-cluster-mirrors tool. But it seems that
> sometimes
> >> it
> >> >> also refers to the admin client on the broker? Perhaps we can use
> 'the
> >> >> tool' instead to make it clear.
> >> >>
> >> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
> >> >> instead of just to the controller?
> >> >>
> >> >> JR59. Start Mirror Topics:
> >> >> JR59.1 "During this state, the broker queries the source cluster for
> >> last
> >> >> mirror epochs".
> >> >> Could you describe which RPC is issued to find the LME?
> >> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
> >> >> doesn't
> >> >> contain a replication factor.
> >> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
> >> >> different topic name"
> >> >> Should we also return this error when the topic name is used as a
> >> regular
> >> >> topic?
> >> >>
> >> >> JR60. CreateClusterMirrorRequest: When will it receive an
> >> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> >> CLUSTER_MIRROR_ALREADY_EXISTS?
> >> >>
> >> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
> >> remove
> >> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe
> the
> >> >> specific state causing the error.
> >> >>
> >> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
> >> requests
> >> >> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
> >> Could we
> >> >> make them consistent?
> >> >>
> >> >> JR63. StopMirrorTopics:
> >> >> JR63.1 "The broker validates that all target topic partitions are in
> >> >> either
> >> >> LOG_TRUNCATION or MIRRORING or FAILED state."
> >> >> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED
> can.
> >> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
> >> PAUSED
> >> >> -> STOPPED is valid?
> >> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
> >> >> state"
> >> >> This is weird. The reason to issue StopMirrorTopics is that it's not
> in
> >> >> stopped state.
> >> >>
> >> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
> derived
> >> >> from SourceOffset and DestinationOffset.
> >> >>
> >> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that
> we
> >> >> could just set the partition level state to STOPPED.
> >> >>
> >> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent
> >> to
> >> >> the controller?
> >> >>
> >> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch?
> It
> >> >> seems both are used for fencing and fencing based on the latter is
> >> >> stronger.
> >> >>
> >> >> JR68. Could we make the usage of topicId and topicName in the RPCs
> >> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> >> BumpLeaderEpochsReponse uses TopicName.
> >> >>
> >> >> JR69. MIRROR_PID_RESET:
> >> >> JR69.1 Could you specify the value of the control record's type?
> >> >> JR69.2 How is the SourceClusterId field used?
> >> >>
> >> >> JR70. MirrorPartitionState: type should be "data" not
> >> "coordinator-key",
> >> >> right? Ditto for LastMirrorEpochs.
> >> >>
> >> >> Jun
> >> >>
> >> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com> wrote:
> >> >>
> >> >> > Hi Jun,
> >> >> >
> >> >> > Thanks for the comments.
> >> >> > Answering them below.
> >> >> >
> >> >> > > JR36. Sounds good. Should we remove
> >> >> MirrorLeaderEpochExceededException?
> >> >> >
> >> >> > Sorry, it’s a confusing name. The
> MirrorLeaderEpochExceededException
> >> is
> >> >> > used and thrown when the fetched mirror batch leader epoch > local
> >> >> leader
> >> >> > epoch in destination cluster. And then we’ll stop mirroring and
> move
> >> the
> >> >> > partition to EPOCH_FENSING state to wait for local leader epoch
> >> bump. I
> >> >> > think since we don’t have the concept of `MirrorLeaderEpoch`
> anymore,
> >> >> the
> >> >> > `MirrorLeaderEpochExceededException` should not cause confusion
> now.
> >> >> I’ll
> >> >> > keep the name for the exception.
> >> >> >
> >> >> >
> >> >> > > JR51. Sounds good. Do we plan to always set lastFetchedEpoch in
> the
> >> >> > mirror
> >> >> > fetch request issued to the source cluster, independent of the
> value
> >> of
> >> >> > mirror.support.unclean.leader.election? It's needed to detect
> >> divergence
> >> >> > during failover. However, if it's also set when
> >> >> > mirror.support.unclean.leader.election=false,
> >> >> > an unclean leader election in the source cluster can halt the
> >> mirroring
> >> >> > since we don’t do anything with the returned divergeEpoch in that
> >> mode
> >> >> > and continue
> >> >> > the next fetch.
> >> >> >
> >> >> > > Do we plan to always set lastFetchedEpoch in the mirror
> >> >> > fetch request issued to the source cluster, independent of the
> value
> >> of
> >> >> > Mirror.support.unclean.leader.election?
> >> >> >
> >> >> > Yes, we always set lastFetchedEpoch in the mirror fetch request.
> >> >> >
> >> >> > > However, if it's also set when
> >> >> > mirror.support.unclean.leader.election=false,
> >> >> > an unclean leader election in the source cluster can halt the
> >> mirroring
> >> >> > since we don’t do anything with the returned divergeEpoch in that
> >> mode
> >> >> > and continue the next fetch.
> >> >> >
> >> >> > Sorry I didn’t make it clear. When
> >> >> > “mirror.support.unclean.leader.election=false” and a “divergeEpoch”
> >> is
> >> >> set
> >> >> > in fetch response, the leader node will handle the divergeEpoch and
> >> then
> >> >> > continue the next fetch as usual. The difference with
> >> >> > “mirror.support.unclean.leader.election=true” is that “should we
> wait
> >> >> for
> >> >> > ALL replicas caught up (complete truncation)”. Without the waiting,
> >> it’s
> >> >> > possible the inconsistent log issue in KAFKA-18723 could happen
> >> between
> >> >> > leader and follower nodes in the destination cluster because we
> don’t
> >> >> have
> >> >> > “MirrorLeaderEpoch” concept now to reject higher leader epoch
> >> batches.
> >> >> That
> >> >> > is, we won't halt the mirroring if "divergeEpoch" is set but
> >> >> > “mirror.support.unclean.leader.election=false”.
> >> >> >
> >> >> >
> >> >> > > JR55. The DescribeClusterMirrorsRequest still returns both the
> >> mirror
> >> >> > partition state and the mirror partition leader state. The problem
> is
> >> >> that
> >> >> > if a caller only needs one of the state, the other state becomes
> >> >> redundant.
> >> >> > For example, when starting the mirror, to obtain LME, we only need
> >> the
> >> >> > mirror partition state. Here is an alternative design: We expose
> the
> >> >> mirror
> >> >> > partition state through ReadMirrorState. The client will use the
> >> >> > FindCoordinatorRequest
> >> >> > to find the right mirror partition coordinator to issue that
> >> request. We
> >> >> > expose the mirror partition leader state through the
> >> >> ReadClusterMirrorLags
> >> >> > request. The client can use the metadata request to find the
> >> partition
> >> >> > leader to issue that request.
> >> >> >
> >> >> > > In the describing mirror flow, the admin client first issues the
> >> >> > ListMirrorRequest to find all the mirrors and the topics being
> >> >> mirrored. It
> >> >> > then issues the metadata request to find the partitions for those
> >> topics
> >> >> > and issues a FindCoordinatorRequest to find the mirror partition
> >> state
> >> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
> >> partition
> >> >> > state coordinator and ReadClusterMirrorLags to the partition
> leader.
> >> >> >
> >> >> > > In the starting mirror flow, the partition leader in the
> >> destination
> >> >> > cluster issues a ListMirrorRequest to find the mirror name based on
> >> its
> >> >> > cluster id. It then issues a FindCoordinatorRequest to find the
> >> mirror
> >> >> > partition state coordinators. Finally, it issues ReadMirrorState to
> >> the
> >> >> > mirror partition state coordinator to get the LME.
> >> >> >
> >> >> > > We can also improve the ListMirror request to accept a filter
> >> based on
> >> >> > mirror name, source cluster ID and topic name.
> >> >> >
> >> >> >
> >> >> > Answers:
> >> >> >
> >> >> > > The DescribeClusterMirrorsRequest still returns both the mirror
> >> >> > partition state and the mirror partition leader state. The problem
> is
> >> >> that
> >> >> > if a caller only needs one of the state, the other state becomes
> >> >> redundant.
> >> >> > For example, when starting the mirror, to obtain LME, we only need
> >> the
> >> >> > mirror partition state.
> >> >> >
> >> >> > Good point!
> >> >> >
> >> >> >
> >> >> > > Here is an alternative design: We expose the mirror
> >> >> > partition state through ReadMirrorState. The client will use the
> >> >> > FindCoordinatorRequest
> >> >> > to find the right mirror partition coordinator to issue that
> >> request. We
> >> >> > expose the mirror partition leader state through the
> >> >> ReadClusterMirrorLags
> >> >> > request. The client can use the metadata request to find the
> >> partition
> >> >> > leader to issue that request.
> >> >> >
> >> >> >
> >> >> > Sounds good. But we try not to expose
> >> >> > ReadMirrorState/ReadClusterMirrorLags directly to the admin client
> >> >> because
> >> >> > users might get confused about these 2 requests, and they are also
> >> both
> >> >> > related to describeClusterMirror. I think we can add 2 additional
> >> >> fields to
> >> >> > describeClusterMirrorRequest: “includeMirrorStates” and
> >> >> > “includeMirrorLags”. So the admin client can include either one or
> >> both
> >> >> of
> >> >> > them in the DescrbieClusterMirrorRequest based on the provided
> >> options.
> >> >> KIP
> >> >> > is updated.
> >> >> >
> >> >> >
> >> >> > > In the describing mirror flow, the admin client first issues the
> >> >> > ListMirrorRequest to find all the mirrors and the topics being
> >> >> mirrored. It
> >> >> > then issues the metadata request to find the partitions for those
> >> topics
> >> >> > and issues a FindCoordinatorRequest to find the mirror partition
> >> state
> >> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
> >> partition
> >> >> > state coordinator and ReadClusterMirrorLags to the partition
> leader.
> >> >> >
> >> >> > Makes sense. KIP is updated.
> >> >> >
> >> >> >
> >> >> > > In the starting mirror flow, the partition leader in the
> >> destination
> >> >> > cluster issues a ListMirrorRequest to find the mirror name based on
> >> its
> >> >> > cluster id. It then issues a FindCoordinatorRequest to find the
> >> mirror
> >> >> > partition state coordinators. Finally, it issues ReadMirrorState to
> >> the
> >> >> > mirror partition state coordinator to get the LME.
> >> >> >
> >> >> > Make sense. So we can send DescribeClusterMirrorRequest with
> >> >> > “includeMirrorStates” only.
> >> >> >
> >> >> > > We can also improve the ListMirror request to accept a filter
> >> based on
> >> >> > mirror name, source cluster ID and topic name.
> >> >> >
> >> >> > Good suggestion, just like what ListGroupsRequest does.
> >> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
> >> >> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP
> >> >> updated.
> >> >> >
> >> >> >
> >> >> > > JR56. Could you list the ACL needed for each new RPC?
> >> >> >
> >> >> >
> >> >> > Yes, it’s already listed in “Security Control” section.
> >> >> >
> >> >> >
> >> >> > > JR57. In the command workflow, it sometimes says "The user sends"
> >> and
> >> >> > some
> >> >> > other times says "The admin client sends". It's better to use "The
> >> admin
> >> >> > client" consistently in all those places.
> >> >> >
> >> >> > Agree. KIP updated.
> >> >> >
> >> >> >
> >> >> > Thank you,
> >> >> > Luke
> >> >> >
> >> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <
> dev@kafka.apache.org
> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> Hi, Luke,
> >> >> >>
> >> >> >> Thanks for the reply.
> >> >> >>
> >> >> >> JR36. Sounds good. Should we remove
> >> MirrorLeaderEpochExceededException?
> >> >> >>
> >> >> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in
> the
> >> >> mirror
> >> >> >> fetch request issued to the source cluster, independent of the
> >> value of
> >> >> >> mirror.support.unclean.leader.election? It's needed to detect
> >> >> divergence
> >> >> >> during failover. However, if it's also set when
> >> >> >> mirror.support.unclean.leader.election=false,
> >> >> >> an unclean leader election in the source cluster can halt the
> >> mirroring
> >> >> >> since we don’t do anything with the returned divergeEpoch in that
> >> mode
> >> >> >> and continue
> >> >> >> the next fetch.
> >> >> >>
> >> >> >> JR55. The DescribeClusterMirrorsRequest still returns both the
> >> mirror
> >> >> >> partition state and the mirror partition leader state. The problem
> >> is
> >> >> that
> >> >> >> if a caller only needs one of the state, the other state becomes
> >> >> >> redundant.
> >> >> >> For example, when starting the mirror, to obtain LME, we only need
> >> the
> >> >> >> mirror partition state. Here is an alternative design: We expose
> the
> >> >> >> mirror
> >> >> >> partition state through ReadMirrorState. The client will use the
> >> >> >> FindCoordinatorRequest
> >> >> >> to find the right mirror partition coordinator to issue that
> >> request.
> >> >> We
> >> >> >> expose the mirror partition leader state through the
> >> >> ReadClusterMirrorLags
> >> >> >> request. The client can use the metadata request to find the
> >> partition
> >> >> >> leader to issue that request.
> >> >> >>
> >> >> >> In the describing mirror flow, the admin client first issues the
> >> >> >> ListMirrorRequest to find all the mirrors and the topics being
> >> >> mirrored.
> >> >> >> It
> >> >> >> then issues the metadata request to find the partitions for those
> >> >> topics
> >> >> >> and issues a FindCoordinatorRequest to find the mirror partition
> >> state
> >> >> >> coordinators. Finally, it issues ReadMirrorState to the mirror
> >> >> partition
> >> >> >> state coordinator and ReadClusterMirrorLags to the partition
> leader.
> >> >> >>
> >> >> >> In the starting mirror flow, the partition leader in the
> destination
> >> >> >> cluster issues a ListMirrorRequest to find the mirror name based
> on
> >> its
> >> >> >> cluster id. It then issues a FindCoordinatorRequest to find the
> >> mirror
> >> >> >> partition state coordinators. Finally, it issues ReadMirrorState
> to
> >> the
> >> >> >> mirror partition state coordinator to get the LME.
> >> >> >>
> >> >> >> We can also improve the ListMirror request to accept a filter
> based
> >> on
> >> >> >> mirror name, source cluster ID and topic name.
> >> >> >>
> >> >> >> JR56. Could you list the ACL needed for each new RPC?
> >> >> >>
> >> >> >> JR57. In the command workflow, it sometimes says "The user sends"
> >> and
> >> >> some
> >> >> >> other times says "The admin client sends". It's better to use "The
> >> >> admin
> >> >> >> client" consistently in all those places.
> >> >> >>
> >> >> >> Jun
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com>
> wrote:
> >> >> >>
> >> >> >> > Hi Jun,
> >> >> >> >
> >> >> >> > Thanks for your comments.
> >> >> >> > Answering them below.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> >> >> specifically
> >> >> >> > designed for the unclean leader election support in Cluster
> >> Mirroring
> >> >> >> > because the unclean leader election will break the assumption
> that
> >> >> the
> >> >> >> > committed data won’t be truncated."
> >> >> >> > If the source cluster has an unclean leader election, the
> >> destination
> >> >> >> > cluster will detect the inconsistency and truncate the log at
> the
> >> >> >> > destination leader. We can just
> >> >> >> > extend mirror.support.unclean.leader.election to cover this
> case,
> >> >> i.e.,
> >> >> >> we
> >> >> >> > wait until all replicas' log end offsets are reduced to the
> target
> >> >> >> offset
> >> >> >> > before resuming mirroring. This avoids the inconsistency.
> >> >> >> >
> >> >> >> > If the destination cluster has an unclean leader election, it
> will
> >> >> >> > re-fetch the same data from the source. There won't be any
> >> >> inconsistency
> >> >> >> > either.
> >> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >> >
> >> >> >> >
> >> >> >> > Yes, that works, too. I agree this is better because the unclean
> >> >> leader
> >> >> >> > election is rare, we don't want to occupy a field just for it.
> >> >> >> >
> >> >> >> > So to achieve this, we have to:
> >> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
> >> >> >> > 2. When the destination cluster leader completes a log
> truncation
> >> >> after
> >> >> >> a
> >> >> >> > fetch response due to divergeEpoch, we know it must be caused by
> >> >> unclean
> >> >> >> > leader election in the source cluster because we only fetch
> >> committed
> >> >> >> > records. If “mirror.support.unclean.leader.election=false”, we
> >> don’t
> >> >> do
> >> >> >> > anything and continue the next fetch. If
> >> >> >> > “mirror.support.unclean.leader.election=true”, we remove the
> >> mirror
> >> >> >> fetcher
> >> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
> >> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be
> >> >> caught
> >> >> >> up
> >> >> >> > into ISR and reduced to the expected LEO.
> >> >> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
> >> >> >> >
> >> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch
> API.
> >> >> >> > KIP is updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR51. Make sense. To truncate based on LME, we need to issue
> >> >> another
> >> >> >> RPC
> >> >> >> > to translate the LME to an offset. It would be useful to include
> >> >> that in
> >> >> >> > the workflow description.
> >> >> >> >
> >> >> >> > About the additional RPC to translate the LME to the offset
> (i.e.
> >> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on this to
> do
> >> >> the
> >> >> >> > offset translation. We use the local leader epoch cache in the
> >> >> >> destination
> >> >> >> > cluster to translate the epoch to save the RPC call. This works
> >> >> because
> >> >> >> of
> >> >> >> > an invariant the LME truncation protocol maintains: for any
> leader
> >> >> >> epoch ≤
> >> >> >> > LME, that epoch's data belongs to exactly one cluster, either it
> >> >> >> originated
> >> >> >> > there, or it was mirrored. This holds even across repeated
> mirror
> >> ->
> >> >> >> > reverse-mirror -> mirror cycles, since each direction's
> >> >> LOG_TRUNCATION
> >> >> >> > phase re-establishes the invariant before that cluster starts
> >> >> mirroring.
> >> >> >> > With this invariant, the replication protocol can find out the
> >> >> diverge
> >> >> >> > point and truncate the log correctly when mirroring. For
> example:
> >> >> >> >
> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> > Offset 2: A (LE: 0)
> >> >> >> > Offset 3: A (LE: 1)
> >> >> >> >
> >> >> >> > Cluster A crashes while Cluster B mirrors to offset 1. Failover
> >> to B
> >> >> and
> >> >> >> > store the LME=0 in cluster B. When cluster A recovers and starts
> >> the
> >> >> >> > reverse mirroring, it’ll query the LME from B and get back LME:
> 0.
> >> >> Then
> >> >> >> > cluster A gets the end offset for epoch 0 from leaderEpochCache,
> >> and
> >> >> >> gets
> >> >> >> > offset 2. So offset 3 is truncated. The log is like this now:
> >> >> >> >
> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
> >> >> >> > Offset 3: B (LE: 10)
> >> >> >> >
> >> >> >> > When mirror starts, cluster A sends the fetch request with {
> >> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the
> divergingEpoch
> >> >> check
> >> >> >> in
> >> >> >> > the cluster B queries the endOffset for leader epoch 0, and
> found
> >> the
> >> >> >> end
> >> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns
> >> fetch
> >> >> >> > response with divergingEpoch info to cluster A for log
> >> truncation. In
> >> >> >> the
> >> >> >> > end, offset 2 in cluster A will be truncated and then log
> >> converges
> >> >> with
> >> >> >> > cluster B.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit
> >> strange
> >> >> >> > because it mixes querying the mirror partition state with
> querying
> >> >> the
> >> >> >> > mirror partition leader. Another option is to use
> >> >> >> ReadMirrorStatesRequest
> >> >> >> > to query the mirror partition state and a new request to query
> the
> >> >> >> mirror
> >> >> >> > partition leader. If we want the adminClient to query the right
> >> >> >> ClusterMirrorCoordinator
> >> >> >> > directly, we can extend the FindCoordinatorRequest to support
> >> that.
> >> >> >> >
> >> >> >> >
> >> >> >> > Good suggestion. Yes, we need to fan out to all brokers because
> we
> >> >> have
> >> >> >> to
> >> >> >> > collect info from mirror partition leaders and mirror
> >> coordinators.
> >> >> If
> >> >> >> we
> >> >> >> > create a new API to contact mirror partition leaders, it will
> make
> >> >> the
> >> >> >> API
> >> >> >> > cleaner and clearer. So we will do this:
> >> >> >> >
> >> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields from
> >> >> >> > DescribeClusterMirror request/response
> >> >> >> >
> >> >> >> > b. Add a new API: ReadClusterMirrorLags API to request for
> >> >> SourceOffset,
> >> >> >> > DestinationOffset, and Lag info.
> >> >> >> >
> >> >> >> > The workflow for admin client describeClusterMirror will be like
> >> >> this:
> >> >> >> > 1. adminClient sends describeClusterMirror to the
> leastLoadedNode
> >> >> broker
> >> >> >> > 2. The received broker collects the current mirrored partitions,
> >> and
> >> >> >> > groups them by coordinator node and by partition leader node.
> >> >> >> > 3. The broker sends ReadMirrorState request to the coordinator
> >> nodes
> >> >> to
> >> >> >> > get the mirror state related info
> >> >> >> > 4. The broker sends ReadClusterMirrorLags request to the mirror
> >> >> >> partition
> >> >> >> > leader nodes to get the mirror lag, source offset, destination
> >> offset
> >> >> >> > (which updates on every fetch response).
> >> >> >> > 5. The broker merge all the response into describeClusterMirror
> >> >> response
> >> >> >> > and send to admin client
> >> >> >> >
> >> >> >> > KIP is updated. Thanks for the suggestion.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR55. For __mirror_state topic, could we describe whether it
> is
> >> >> >> > compacted?
> >> >> >> >
> >> >> >> > Yes, it is a compacted topic. I made it much clearer in the
> >> >> >> > “ClusterMirrorCoordinator” section.
> >> >> >> >
> >> >> >> >
> >> >> >> > Thank you,
> >> >> >> > Luke
> >> >> >> >
> >> >> >> >
> >> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
> >> dev@kafka.apache.org
> >> >> >
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Hi, Luke,
> >> >> >> >>
> >> >> >> >> Thanks for the reply.
> >> >> >> >>
> >> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> >> >> specifically
> >> >> >> >> designed for the unclean leader election support in Cluster
> >> >> Mirroring
> >> >> >> >> because the unclean leader election will break the assumption
> >> that
> >> >> the
> >> >> >> >> committed data won’t be truncated."
> >> >> >> >> If the source cluster has an unclean leader election, the
> >> >> destination
> >> >> >> >> cluster will detect the inconsistency and truncate the log at
> the
> >> >> >> >> destination leader. We can just
> >> >> >> >> extend mirror.support.unclean.leader.election to cover this
> case,
> >> >> >> i.e., we
> >> >> >> >> wait until all replicas' log end offsets are reduced to the
> >> target
> >> >> >> offset
> >> >> >> >> before resuming mirroring. This avoids the inconsistency.
> >> >> >> >>
> >> >> >> >> If the destination cluster has an unclean leader election, it
> >> will
> >> >> >> >> re-fetch
> >> >> >> >> the same data from the source. There won't be any inconsistency
> >> >> either.
> >> >> >> >>
> >> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >> >>
> >> >> >> >> JR51. Make sense. To truncate based on LME, we need to issue
> >> another
> >> >> >> RPC
> >> >> >> >> to
> >> >> >> >> translate the LME to an offset. It would be useful to include
> >> that
> >> >> in
> >> >> >> the
> >> >> >> >> workflow description.
> >> >> >> >>
> >> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit
> strange
> >> >> >> because
> >> >> >> >> it mixes querying the mirror partition state with querying the
> >> >> mirror
> >> >> >> >> partition leader. Another option is to use
> >> ReadMirrorStatesRequest
> >> >> to
> >> >> >> >> query
> >> >> >> >> the mirror partition state and a new request to query the
> mirror
> >> >> >> partition
> >> >> >> >> leader. If we want the adminClient to query the right
> >> >> >> >> ClusterMirrorCoordinator
> >> >> >> >> directly, we can extend the FindCoordinatorRequest to support
> >> that.
> >> >> >> >>
> >> >> >> >> JR55. For __mirror_state topic, could we describe whether it is
> >> >> >> compacted?
> >> >> >> >>
> >> >> >> >> Jun
> >> >> >> >>
> >> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com>
> >> wrote:
> >> >> >> >>
> >> >> >> >> > Hi Jun,
> >> >> >> >> >
> >> >> >> >> > Thanks for the comments.
> >> >> >> >> > Answering them below:
> >> >> >> >> >
> >> >> >> >> > > JR31.3 It makes sense for the new partition leader to query
> >> the
> >> >> >> >> current
> >> >> >> >> > mirror state from the local or remote MirrorMetadataManager
> >> that
> >> >> owns
> >> >> >> >> the
> >> >> >> >> > partition's state. However, the wording in the KIP is
> >> confusing.
> >> >> >> >> > MirrorMetadataManager
> >> >> >> >> > mirror partition state cache should be populated only by one
> >> way,
> >> >> >> which
> >> >> >> >> is
> >> >> >> >> > replaying __mirror_state partitions this broker leads. By
> >> saying
> >> >> >> "and by
> >> >> >> >> > fetching state from remote coordinators via ReadMirrorStates
> >> >> RPCs.",
> >> >> >> >> it's
> >> >> >> >> > as if the mirror partition state cache on each broker stores
> >> the
> >> >> >> state
> >> >> >> >> for
> >> >> >> >> > all mirror partitions.
> >> >> >> >> >
> >> >> >> >> > You’re right. KIP is updated to make it clear.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR36. I understand the intention now, but I still don't
> >> think we
> >> >> >> need
> >> >> >> >> > MirrorLeaderEpoch. Let's
> >> >> >> >> > assume the common case: no unclean leader election in the
> >> source
> >> >> >> cluster
> >> >> >> >> > during mirroring. Mirroring only fetches the committed data
> >> from
> >> >> the
> >> >> >> >> > source, which never changes if obtained a second time. In
> your
> >> >> >> example
> >> >> >> >> > image, follower broker 1 in the destination cluster can never
> >> >> obtain
> >> >> >> the
> >> >> >> >> > record B(0) because it's never committed in the source and
> >> thus is
> >> >> >> never
> >> >> >> >> > exposed to the destination leader. It's possible for broker 1
> >> to
> >> >> >> first
> >> >> >> >> get
> >> >> >> >> > D(1) which is truncated later when broker 0 follows a new
> >> leader.
> >> >> But
> >> >> >> >> the
> >> >> >> >> > new leader will refetch the same D(1) from the source
> cluster.
> >> So,
> >> >> >> it's
> >> >> >> >> ok
> >> >> >> >> > for broker 1 to keep D(1).
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Yes, you are correct. Because mirroring only fetches the
> >> committed
> >> >> >> data
> >> >> >> >> > from the source, the MirrorLeaderEpoch is unnecessary for
> >> normal
> >> >> >> >> mirroring.
> >> >> >> >> > Thanks for the clarification. So I think we can say, the
> >> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the unclean
> >> >> leader
> >> >> >> >> > election support in Cluster Mirroring because the unclean
> >> leader
> >> >> >> >> election
> >> >> >> >> > will break the assumption that the committed data won’t be
> >> >> truncated.
> >> >> >> >> >
> >> >> >> >> > Given that, we could gate the MirrorLeaderEpoch field by only
> >> >> >> setting it
> >> >> >> >> > in the Fetch request when
> >> >> >> "mirror.support.unclean.leader.election=true"
> >> >> >> >> > (false by default), and skip it entirely for normal
> mirroring.
> >> >> What
> >> >> >> do
> >> >> >> >> you
> >> >> >> >> > think?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR51. Sounds good. Why do we choose to store the last
> mirror
> >> >> epoch
> >> >> >> >> > instead
> >> >> >> >> > of a last mirror offset?
> >> >> >> >> >
> >> >> >> >> > Yes, originally we stored the “last mirror offset” instead of
> >> >> “last
> >> >> >> >> mirror
> >> >> >> >> > epoch”. But later, we found it doesn’t work for unclean
> leader
> >> >> >> election
> >> >> >> >> > cases. (yes, `unclean leader election` again :) ) For
> example:
> >> >> >> >> >
> >> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >> >
> >> >> >> >> > Cluster B is mirroring from A. And A crashes and failover to
> >> B. B
> >> >> >> stores
> >> >> >> >> > the “last mirror offset” as 1.
> >> >> >> >> > When cluster A recovers, somehow the unclean leader election
> >> >> happens,
> >> >> >> >> log
> >> >> >> >> > is empty in the new leader, and then some more leadership
> >> changes
> >> >> to
> >> >> >> >> make
> >> >> >> >> > the leader epoch become 10. Then it starts the reverse mirror
> >> >> from B.
> >> >> >> >> The
> >> >> >> >> > log looks like this:
> >> >> >> >> >
> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> >> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> >> >> >> >> > Offset 2: pid_res
> >> (LE:
> >> >> >> 10)
> >> >> >> >> >
> >> >> >> >> > When A starts mirroring from B, log is truncated to last
> mirror
> >> >> >> offset:
> >> >> >> >> 1
> >> >> >> >> > (no-op), and then fetch from cluster B. The 1st fetch request
> >> is
> >> >> like
> >> >> >> >> this:
> >> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset:
> >> 2, …
> >> >> }.
> >> >> >> The
> >> >> >> >> > divergingEpoch check in the cluster B queries the endOffset
> for
> >> >> >> leader
> >> >> >> >> > epoch 10 (in Partition#readRecords), and found the end offset
> >> for
> >> >> LE
> >> >> >> 10
> >> >> >> >> is
> >> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch
> validation
> >> >> >> passes
> >> >> >> >> and
> >> >> >> >> > returns offset 2 to cluster A. In short, relying on “last
> >> mirror
> >> >> >> offset”
> >> >> >> >> > cannot resolve the log diverge issue in unclean leader
> election
> >> >> case.
> >> >> >> >> >
> >> >> >> >> > If instead, we truncate until “last mirror epoch”: 0, the
> >> offset 0
> >> >> >> and 1
> >> >> >> >> > in cluster A will be truncated as expected and then log
> >> converge
> >> >> with
> >> >> >> >> > cluster B correctly in the end.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Also, about JR53.2:
> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
> >> brokers
> >> >> and
> >> >> >> >> > merges
> >> >> >> >> > the responses client-side."
> >> >> >> >> > This is not ideal. Not every broker hosts a mirror partition.
> >> So
> >> >> >> sending
> >> >> >> >> > the request to every broker is wasteful. It's probably better
> >> to
> >> >> send
> >> >> >> >> the
> >> >> >> >> > request to a single broker, which can then contact the right
> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >
> >> >> >> >> > While working on this, I found we need to add one more tagged
> >> >> field
> >> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema
> because
> >> >> >> >> otherwise
> >> >> >> >> > the receiving broker doesn’t know if this is the request from
> >> >> client
> >> >> >> >> and it
> >> >> >> >> > needs to contact other brokers, or this is just a forwarded
> >> >> request.
> >> >> >> The
> >> >> >> >> > workflow is like this:
> >> >> >> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest
> to a
> >> >> >> >> > leastLoaded broker, which doesn’t contain the "Forwarded"
> >> field.
> >> >> >> >> > 2. The broker receives the request and sends
> >> >> >> >> DescribeClusterMirrorsRequest
> >> >> >> >> > with "Forwarded=true" to the responsible brokers and waits
> for
> >> the
> >> >> >> >> response
> >> >> >> >> > async.
> >> >> >> >> > 3. When all responses are received, merge the results and
> >> respond
> >> >> to
> >> >> >> the
> >> >> >> >> > client.
> >> >> >> >> >
> >> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here because it
> >> >> doesn’t
> >> >> >> >> > contain the lag info in the coordinator.
> >> >> >> >> >
> >> >> >> >> > The alternative solution is the adminClient sends
> >> >> >> >> > DescribeClusterMirrorsRequest to the “responsible brokers
> only”
> >> >> (via
> >> >> >> >> > describeTopics), instead of ALL brokers, and merge the
> results
> >> in
> >> >> the
> >> >> >> >> > client side. This solution doesn’t need to add a new field in
> >> the
> >> >> >> >> > DescribeClusterMirrorsRequest and still can avoid unnecessary
> >> >> >> requests
> >> >> >> >> > sent.
> >> >> >> >> >
> >> >> >> >> > My preference is to keep the merge logic on the admin client
> >> side,
> >> >> >> which
> >> >> >> >> > saves the overhead in the broker side.
> >> >> >> >> > What do you think?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Thank you,
> >> >> >> >> > Luke
> >> >> >> >> >
> >> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
> >> >> dev@kafka.apache.org
> >> >> >> >
> >> >> >> >> > wrote:
> >> >> >> >> >
> >> >> >> >> >> Hi, Luke,
> >> >> >> >> >>
> >> >> >> >> >> Thanks for the reply.
> >> >> >> >> >>
> >> >> >> >> >> JR31.3 It makes sense for the new partition leader to query
> >> the
> >> >> >> current
> >> >> >> >> >> mirror state from the local or remote MirrorMetadataManager
> >> that
> >> >> >> owns
> >> >> >> >> the
> >> >> >> >> >> partition's state. However, the wording in the KIP is
> >> confusing.
> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> mirror partition state cache should be populated only by one
> >> way,
> >> >> >> >> which is
> >> >> >> >> >> replaying __mirror_state partitions this broker leads. By
> >> saying
> >> >> >> "and
> >> >> >> >> by
> >> >> >> >> >> fetching state from remote coordinators via ReadMirrorStates
> >> >> RPCs.",
> >> >> >> >> it's
> >> >> >> >> >> as if the mirror partition state cache on each broker stores
> >> the
> >> >> >> state
> >> >> >> >> for
> >> >> >> >> >> all mirror partitions.
> >> >> >> >> >>
> >> >> >> >> >> JR36. I understand the intention now, but I still don't
> think
> >> we
> >> >> >> need
> >> >> >> >> >> MirrorLeaderEpoch. Let's
> >> >> >> >> >> assume the common case: no unclean leader election in the
> >> source
> >> >> >> >> cluster
> >> >> >> >> >> during mirroring. Mirroring only fetches the committed data
> >> from
> >> >> the
> >> >> >> >> >> source, which never changes if obtained a second time. In
> your
> >> >> >> example
> >> >> >> >> >> image, follower broker 1 in the destination cluster can
> never
> >> >> obtain
> >> >> >> >> the
> >> >> >> >> >> record B(0) because it's never committed in the source and
> >> thus
> >> >> is
> >> >> >> >> never
> >> >> >> >> >> exposed to the destination leader. It's possible for broker
> 1
> >> to
> >> >> >> first
> >> >> >> >> get
> >> >> >> >> >> D(1) which is truncated later when broker 0 follows a new
> >> leader.
> >> >> >> But
> >> >> >> >> the
> >> >> >> >> >> new leader will refetch the same D(1) from the source
> cluster.
> >> >> So,
> >> >> >> >> it's ok
> >> >> >> >> >> for broker 1 to keep D(1).
> >> >> >> >> >>
> >> >> >> >> >> JR51. Sounds good. Why do we choose to store the last mirror
> >> >> epoch
> >> >> >> >> instead
> >> >> >> >> >> of a last mirror offset?
> >> >> >> >> >>
> >> >> >> >> >> Jun
> >> >> >> >> >>
> >> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <showuon@gmail.com
> >
> >> >> wrote:
> >> >> >> >> >>
> >> >> >> >> >> > Hi Jun,
> >> >> >> >> >> >
> >> >> >> >> >> > Thanks for the great comments again.
> >> >> >> >> >> > Answering your question below.
> >> >> >> >> >> >
> >> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror partition
> >> cache
> >> >> >> lives
> >> >> >> >> in
> >> >> >> >> >> > MirrorMetadataManager. It is populated in two ways: by
> >> >> replaying
> >> >> >> >> >> > __mirror_state partitions this broker leads, and by
> fetching
> >> >> state
> >> >> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> >> >> > Hmm, a given broker only caches the mirror partition state
> >> for
> >> >> >> >> >> partitions
> >> >> >> >> >> > hashed into the __mirror_state partition hosted on this
> >> broker,
> >> >> >> >> right?
> >> >> >> >> >> Why
> >> >> >> >> >> > does it need to fetch the state from other remote
> >> coordinators?
> >> >> >> >> >> >
> >> >> >> >> >> > > a given broker only caches the mirror partition state
> for
> >> >> >> >> partitions
> >> >> >> >> >> > hashed into the __mirror_state partition hosted on this
> >> broker,
> >> >> >> >> right?
> >> >> >> >> >> >
> >> >> >> >> >> > Right.
> >> >> >> >> >> >
> >> >> >> >> >> > > Why does it need to fetch the state from other remote
> >> >> >> coordinators?
> >> >> >> >> >> >
> >> >> >> >> >> > Because the data partition leader needs to know the
> current
> >> >> mirror
> >> >> >> >> state
> >> >> >> >> >> > of the partitions it leads. On a leadership change, the
> >> >> metadata
> >> >> >> >> delta
> >> >> >> >> >> is
> >> >> >> >> >> > published to the MirrorMetadataManager, which queries the
> >> >> current
> >> >> >> >> mirror
> >> >> >> >> >> > state for its led partitions from the coordinator (local
> or
> >> >> >> remote),
> >> >> >> >> and
> >> >> >> >> >> > processes tasks based on that state. This is the reason
> why
> >> it
> >> >> >> >> fetches
> >> >> >> >> >> the
> >> >> >> >> >> > state from other remote coordinator. This part of cache
> >> will be
> >> >> >> >> cleared
> >> >> >> >> >> > after the leadership change.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR36. I am still not quite sure why we need
> >> >> MirrorLeaderEpoch in
> >> >> >> >> the
> >> >> >> >> >> > Fetch
> >> >> >> >> >> > request. In the current Fetch request, in addition to
> >> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> > it has another field LastFetchedEpoch, which captures the
> >> last
> >> >> >> leader
> >> >> >> >> >> epoch
> >> >> >> >> >> > in the data and is used to resolve the inconsistency
> between
> >> >> the
> >> >> >> >> >> follower
> >> >> >> >> >> > and the leader. Does it achieve the same purpose as
> >> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to do
> the
> >> >> >> >> validation
> >> >> >> >> >> to
> >> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any fetched
> >> >> batches
> >> >> >> with
> >> >> >> >> >> epoch
> >> >> >> >> >> > higher than the LastFetchedEpoch, it means the follower
> can
> >> >> never
> >> >> >> >> accept
> >> >> >> >> >> > new appended batches with higher epoch in the leader node.
> >> >> >> >> >> >
> >> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes
> in
> >> >> the
> >> >> >> >> fetch
> >> >> >> >> >> > API:
> >> >> >> >> >> > 1. The receiver (leader node) uses it to validate it
> matches
> >> >> the
> >> >> >> >> leader
> >> >> >> >> >> > epoch in the leader node
> >> >> >> >> >> > 2. The sender (follower node) uses it to validate the
> epoch
> >> in
> >> >> the
> >> >> >> >> >> fetched
> >> >> >> >> >> > batches are all <= this value.
> >> >> >> >> >> >
> >> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still serving
> for
> >> >> >> >> followers
> >> >> >> >> >> in
> >> >> >> >> >> > the destination cluster. For purpose (2), my understanding
> >> of
> >> >> the
> >> >> >> >> >> > validation for KAFKA-18723 is like: the follower knows the
> >> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader node
> >> >> should
> >> >> >> >> >> contain in
> >> >> >> >> >> > the log batches when fetch request is sent. Any log batch
> >> >> beyond
> >> >> >> this
> >> >> >> >> >> epoch
> >> >> >> >> >> > in the fetch response means the follower’s metadata is
> >> stale.
> >> >> The
> >> >> >> >> >> batches
> >> >> >> >> >> > need to be rejected and follower needs to update metadata
> >> for
> >> >> next
> >> >> >> >> >> fetch.
> >> >> >> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch
> >> serves
> >> >> >> this
> >> >> >> >> >> purpose
> >> >> >> >> >> > to let the followers know the highest epoch in the leader
> >> node
> >> >> >> log in
> >> >> >> >> >> each
> >> >> >> >> >> > fetch response. Once the info is stale in the follower
> >> side, it
> >> >> >> >> updates
> >> >> >> >> >> it
> >> >> >> >> >> > from fetch response, and send next fetch request with
> >> correct
> >> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is
> >> possible
> >> >> >> like
> >> >> >> >> >> > KAFKA-18723.
> >> >> >> >> >> >
> >> >> >> >> >> > I tried not to add new field in Fetch API, but this is the
> >> best
> >> >> >> >> >> solution I
> >> >> >> >> >> > can think of.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR48. "The NumPartitions field is needed so the
> controller
> >> >> can
> >> >> >> >> create
> >> >> >> >> >> the
> >> >> >> >> >> > destination topic if it does not already exist. When a
> >> topic is
> >> >> >> being
> >> >> >> >> >> > mirrored for the first time, the destination cluster has
> no
> >> >> >> knowledge
> >> >> >> >> >> > of the source topic's partition count, so the request must
> >> >> carry
> >> >> >> it."
> >> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is started,
> the
> >> >> source
> >> >> >> >> topic
> >> >> >> >> >> > may not exist yet. When the source topic is created later,
> >> we
> >> >> can
> >> >> >> >> start
> >> >> >> >> >> > mirroring without requiring the user to provide the number
> >> of
> >> >> >> >> >> partitions.
> >> >> >> >> >> >
> >> >> >> >> >> > > When the source topic is created later, we can start
> >> >> mirroring
> >> >> >> >> without
> >> >> >> >> >> > requiring the user to provide the number of partitions.
> >> >> >> >> >> >
> >> >> >> >> >> > This is because this happens in periodical sync up in the
> >> >> >> destination
> >> >> >> >> >> > cluster, and the numPartitions and other topic info in the
> >> >> source
> >> >> >> >> >> cluster
> >> >> >> >> >> > is already retrieved during the sync up process. Also, if
> >> users
> >> >> >> are
> >> >> >> >> >> > creating mirror topics via admin API, this info can also
> be
> >> >> >> retrieved
> >> >> >> >> >> > inside the admin client if not provided.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly need the
> >> last
> >> >> >> mirror
> >> >> >> >> >> epoch
> >> >> >> >> >> > for truncation when starting a mirror. When a mirror is
> >> >> stopped at
> >> >> >> >> the
> >> >> >> >> >> > source, we already bump up the leader epoch and write some
> >> >> >> additional
> >> >> >> >> >> > records for cleanup. It seems that we can just use the
> >> >> >> DivergingEpoch
> >> >> >> >> >> > returned in the initial mirror Fetch request to detect the
> >> >> >> diverging
> >> >> >> >> >> epoch
> >> >> >> >> >> > and use that for truncation at the mirror destination.
> >> >> >> >> >> >
> >> >> >> >> >> > That’s a good question. I thought about this before but
> >> >> >> >> unfortunately it
> >> >> >> >> >> > doesn’t work. The reason we need the last mirror epoch
> >> record
> >> >> is
> >> >> >> >> because
> >> >> >> >> >> > the destination cluster might not completely sync with the
> >> >> source
> >> >> >> >> >> cluster
> >> >> >> >> >> > before STOPPED. And this un-mirrored data might confuse
> the
> >> >> >> >> replication
> >> >> >> >> >> > protocol when reverse mirror. For example,
> >> >> >> >> >> >
> >> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
> >> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> >> >> >> > Offset 2: from A (LE: 10)
> >> >> >> >> >> >
> >> >> >> >> >> > While cluster B mirrors until offset 1, the cluster A
> >> crashes
> >> >> and
> >> >> >> >> >> failover
> >> >> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
> >> >> >> MIRROR_PID_RESET
> >> >> >> >> >> with
> >> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from
> >> cluster B,
> >> >> >> the
> >> >> >> >> log
> >> >> >> >> >> > will be like this:
> >> >> >> >> >> >
> >> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> >> > foo-0 leader log: foo-0 leader log:
> >> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
> >> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
> >> >> >> >> >> >
> >> >> >> >> >> > As you can see, cluster A (new destination cluster)
> doesn’t
> >> >> know
> >> >> >> the
> >> >> >> >> >> > offset 2 is not belonging to the cluster B, which causes
> >> >> >> inconsistent
> >> >> >> >> >> logs.
> >> >> >> >> >> > So like what we described in the KIP, the last mirror
> epoch
> >> is
> >> >> the
> >> >> >> >> >> > synchronization point between source and destination. Any
> >> data
> >> >> >> beyond
> >> >> >> >> >> the
> >> >> >> >> >> > point is treated as unknown to the source cluster and
> >> should be
> >> >> >> >> >> truncated.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR52. Stop Mirror:
> >> >> >> >> >> > > JR52.1 "3. If patterns are provided, the controller
> >> removes
> >> >> >> >> matching
> >> >> >> >> >> > entries from mirror.topics.include or adds them to
> >> >> >> >> >> mirror.topics.exclude on
> >> >> >> >> >> > the CLUSTER_MIRROR resource."
> >> >> >> >> >> > Why is it 'or' and not 'and'?
> >> >> >> >> >> >
> >> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> > discovers new source topics matching the persisted
> >> >> include/exclude
> >> >> >> >> >> patterns
> >> >> >> >> >> > and repeats steps 3 through 7 for each."
> >> >> >> >> >> > The broker will send a create topic request to the
> >> controller.
> >> >> How
> >> >> >> >> does
> >> >> >> >> >> the
> >> >> >> >> >> > controller associate it with the mirror name to create a
> >> >> >> >> >> > `MirrorTopicStateChangeRecord`?
> >> >> >> >> >> >
> >> >> >> >> >> > No, we don’t send a create topic request to the controller
> >> >> here.
> >> >> >> >> >> Actually
> >> >> >> >> >> > we send a StartMirrorTopics request to the controller,
> which
> >> >> >> contains
> >> >> >> >> >> the
> >> >> >> >> >> > mirror name and topic metadata like NumPartitions,
> topicId…
> >> >> etc.
> >> >> >> So
> >> >> >> >> the
> >> >> >> >> >> > controller can create the non-existed topic as explained
> in
> >> >> JR48,
> >> >> >> and
> >> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR53. Describe Mirrors:
> >> >> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors
> request
> >> >> with
> >> >> >> >> >> optional
> >> >> >> >> >> > mirror names (empty means all mirrors)."
> >> >> >> >> >> > In metadata request, we use a null arrary to represent all
> >> >> >> topics. It
> >> >> >> >> >> would
> >> >> >> >> >> > be useful to be consistent here.
> >> >> >> >> >> >
> >> >> >> >> >> > OK, KIP updated.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
> >> >> brokers
> >> >> >> and
> >> >> >> >> >> > merges
> >> >> >> >> >> > the responses client-side."
> >> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
> >> partition.
> >> >> So
> >> >> >> >> sending
> >> >> >> >> >> > the request to every broker is wasteful. It's probably
> >> better
> >> >> to
> >> >> >> send
> >> >> >> >> >> the
> >> >> >> >> >> > request to a single broker, which can then contact the
> right
> >> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >
> >> >> >> >> >> > Agree, KIP updated.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are
> >> >> MIRRORING,
> >> >> >> >> >> PAUSED,
> >> >> >> >> >> > STOPPED the only validate states? The state table has
> other
> >> >> states
> >> >> >> >> too.
> >> >> >> >> >> > Ditto for MirrorPartitionState.
> >> >> >> >> >> > The DesiredState field in MirrorTopicStateChangeRecord is
> to
> >> >> >> define
> >> >> >> >> the
> >> >> >> >> >> > target state the user wants. And the desired state users
> can
> >> >> >> assign
> >> >> >> >> is
> >> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired
> >> state
> >> >> as
> >> >> >> >> >> STOPPING
> >> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
> >> intermediate
> >> >> >> >> states.
> >> >> >> >> >> >
> >> >> >> >> >> > About MirrorPartitionState record, it’ll record any state
> in
> >> >> the
> >> >> >> >> state
> >> >> >> >> >> > table that the current partition is. This is the real
> state
> >> for
> >> >> >> each
> >> >> >> >> >> > partition. It’s like in the kubernetes world, when we
> change
> >> >> the
> >> >> >> >> replica
> >> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired state),
> it
> >> >> won’t
> >> >> >> >> >> increase
> >> >> >> >> >> > to 5 immediately. The current replica
> (MirrorPartitionState
> >> >> >> record)
> >> >> >> >> is
> >> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > Thank you,
> >> >> >> >> >> > Luke
> >> >> >> >> >> >
> >> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
> >> >> >> dev@kafka.apache.org
> >> >> >> >> >
> >> >> >> >> >> > wrote:
> >> >> >> >> >> >
> >> >> >> >> >> >> Hi, Fede,
> >> >> >> >> >> >>
> >> >> >> >> >> >> Thanks for the reply. A few more comments.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror partition
> cache
> >> >> >> lives in
> >> >> >> >> >> >> MirrorMetadataManager. It is populated in two ways: by
> >> >> replaying
> >> >> >> >> >> >> __mirror_state partitions this broker leads, and by
> >> fetching
> >> >> >> state
> >> >> >> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> >> >> >> Hmm, a given broker only caches the mirror partition
> state
> >> for
> >> >> >> >> >> partitions
> >> >> >> >> >> >> hashed into the __mirror_state partition hosted on this
> >> >> broker,
> >> >> >> >> right?
> >> >> >> >> >> Why
> >> >> >> >> >> >> does it need to fetch the state from other remote
> >> >> coordinators?
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR36. I am still not quite sure why we need
> >> MirrorLeaderEpoch
> >> >> in
> >> >> >> the
> >> >> >> >> >> Fetch
> >> >> >> >> >> >> request. In the current Fetch request, in addition to
> >> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> >> it has another field LastFetchedEpoch, which captures the
> >> last
> >> >> >> >> leader
> >> >> >> >> >> >> epoch
> >> >> >> >> >> >> in the data and is used to resolve the inconsistency
> >> between
> >> >> the
> >> >> >> >> >> follower
> >> >> >> >> >> >> and the leader. Does it achieve the same purpose as
> >> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR48. "The NumPartitions field is needed so the
> controller
> >> can
> >> >> >> >> create
> >> >> >> >> >> the
> >> >> >> >> >> >> destination topic if it does not already exist. When a
> >> topic
> >> >> is
> >> >> >> >> being
> >> >> >> >> >> >> mirrored for the first time, the destination cluster has
> no
> >> >> >> >> knowledge
> >> >> >> >> >> >> of the source topic's partition count, so the request
> must
> >> >> carry
> >> >> >> >> it."
> >> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is started,
> the
> >> >> >> source
> >> >> >> >> >> topic
> >> >> >> >> >> >> may not exist yet. When the source topic is created
> later,
> >> we
> >> >> can
> >> >> >> >> start
> >> >> >> >> >> >> mirroring without requiring the user to provide the
> number
> >> of
> >> >> >> >> >> partitions.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need the
> >> last
> >> >> >> mirror
> >> >> >> >> >> epoch
> >> >> >> >> >> >> for truncation when starting a mirror. When a mirror is
> >> >> stopped
> >> >> >> at
> >> >> >> >> the
> >> >> >> >> >> >> source, we already bump up the leader epoch and write
> some
> >> >> >> >> additional
> >> >> >> >> >> >> records for cleanup. It seems that we can just use the
> >> >> >> >> DivergingEpoch
> >> >> >> >> >> >> returned in the initial mirror Fetch request to detect
> the
> >> >> >> diverging
> >> >> >> >> >> epoch
> >> >> >> >> >> >> and use that for truncation at the mirror destination.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR52. Stop Mirror:
> >> >> >> >> >> >> JR52.1 "3. If patterns are provided, the controller
> removes
> >> >> >> matching
> >> >> >> >> >> >> entries from mirror.topics.include or adds them to
> >> >> >> >> >> mirror.topics.exclude
> >> >> >> >> >> >> on
> >> >> >> >> >> >> the CLUSTER_MIRROR resource."
> >> >> >> >> >> >> Why is it 'or' and not 'and'?
> >> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> >> MirrorMetadataManager
> >> >> >> >> >> >> discovers new source topics matching the persisted
> >> >> >> include/exclude
> >> >> >> >> >> >> patterns
> >> >> >> >> >> >> and repeats steps 3 through 7 for each."
> >> >> >> >> >> >> The broker will send a create topic request to the
> >> controller.
> >> >> >> How
> >> >> >> >> does
> >> >> >> >> >> >> the
> >> >> >> >> >> >> controller associate it with the mirror name to create a
> >> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR53. Describe Mirrors:
> >> >> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors
> request
> >> >> with
> >> >> >> >> >> optional
> >> >> >> >> >> >> mirror names (empty means all mirrors)."
> >> >> >> >> >> >> In metadata request, we use a null arrary to represent
> all
> >> >> >> topics.
> >> >> >> >> It
> >> >> >> >> >> >> would
> >> >> >> >> >> >> be useful to be consistent here.
> >> >> >> >> >> >> JR53.2 "2. The admin client fans out the request to all
> >> >> brokers
> >> >> >> and
> >> >> >> >> >> merges
> >> >> >> >> >> >> the responses client-side."
> >> >> >> >> >> >> This is not ideal. Not every broker hosts a mirror
> >> partition.
> >> >> So
> >> >> >> >> >> sending
> >> >> >> >> >> >> the request to every broker is wasteful. It's probably
> >> better
> >> >> to
> >> >> >> >> send
> >> >> >> >> >> the
> >> >> >> >> >> >> request to a single broker, which can then contact the
> >> right
> >> >> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >> >>
> >> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are
> >> >> MIRRORING,
> >> >> >> >> PAUSED,
> >> >> >> >> >> >> STOPPED the only validate states? The state table has
> other
> >> >> >> states
> >> >> >> >> too.
> >> >> >> >> >> >> Ditto for MirrorPartitionState.
> >> >> >> >> >> >>
> >> >> >> >> >> >> Jun
> >> >> >> >> >> >>
> >> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
> >> >> >> >> fedevaleri@gmail.com>
> >> >> >> >> >> >> wrote:
> >> >> >> >> >> >>
> >> >> >> >> >> >> > Hi Andrew, yes, validation is called at the start of
> >> >> >> createTopic.
> >> >> >> >> If
> >> >> >> >> >> >> > the topic ID is invalid, the request fails with
> >> >> >> INVALID_REQUEST.
> >> >> >> >> If
> >> >> >> >> >> >> > the topic ID is already used by a different topic name,
> >> it
> >> >> >> fails
> >> >> >> >> with
> >> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the
> individual
> >> >> topic
> >> >> >> in
> >> >> >> >> the
> >> >> >> >> >> >> > batch fails; other topics in the same CreateTopics
> >> request
> >> >> >> >> continue
> >> >> >> >> >> >> > processing normally. Added this information to the KIP.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
> >> >> >> >> >> >> schofielaj@apache.org>
> >> >> >> >> >> >> > wrote:
> >> >> >> >> >> >> > >
> >> >> >> >> >> >> > > Hi Fede,
> >> >> >> >> >> >> > > One more small question.
> >> >> >> >> >> >> > >
> >> >> >> >> >> >> > > AS37: If the controller fails validation for the
> >> >> >> >> MirrorInfo.TopicId
> >> >> >> >> >> >> > added to the CreateTopic request, what does it do? I
> >> suspect
> >> >> >> that
> >> >> >> >> it
> >> >> >> >> >> >> fails
> >> >> >> >> >> >> > the request with a particular error code.
> >> >> >> >> >> >> > >
> >> >> >> >> >> >> > > Thanks,
> >> >> >> >> >> >> > > Andrew
> >> >> >> >> >> >> > >
> >> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> >> >> >> >> >> >> > > > Hi all,
> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> > > > We updated the KIP to include Coordinator Runtime
> >> >> >> >> configurations
> >> >> >> >> >> and
> >> >> >> >> >> >> > > > the coordinator state transition validation
> >> paragraph.
> >> >> The
> >> >> >> >> latter
> >> >> >> >> >> >> > > > covers the per-partition epoch fencing mechanism
> >> (leader
> >> >> >> epoch
> >> >> >> >> >> and
> >> >> >> >> >> >> > > > state epoch) that complements the existing
> per-topic
> >> >> >> >> StateOffset
> >> >> >> >> >> >> > > > fencing for lifecycle operations.
> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> > > > Thanks,
> >> >> >> >> >> >> > > > Fede
> >> >> >> >> >> >> > > >
> >> >> >> >> >> >> >
> >> >> >> >> >> >>
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply.
JR80. PauseMirrorTopicsOptions: "PauseMirrorTopics/ResumeMirrorTopics
supports regex in the ClusterMirrorComamnd, but we don’t need to add the
“patterns”."
So, in this case, we use topics to support both specific topics and regex.
However, startMirrorTopicsOptions uses topics for specific topics and
includePatterns for regex. Could we make them consistent?
JR82. DescribeClusterMirrorsResponse.SourceOffset: Is the value also -1 if
it's not included in the request?
JR83. Start Mirror Topics:
JR83.1 Should the controller verify the expected state for the topic?
JR83.2 "d. truncates the log accordingly,"
Could you add a description of the truncation offset?
JR84. Stop Mirror Topics: "The controller verifies that the desired state
of all topics are in MIRRORING or PAUSED state."
Should the broker verfy the topic level state? I thought the controller
will use StateOffset in the request to make sure the validation on the
broker is stil valid. If the controller performs the verification, is
StateOffset still useful?
JR85. There are still a few references to includeMirrorLags. They should be
IncludeMirrorOffset.
JR86. Metrics:
JR86.1 Could you describe the type (gauge, meter, etc) of each metric?
JR86.2 Could you describe a bit more what MinFetchRate represents?
JR87. Version Compatibility: "For migration or DR use cases where failback
is not required, this proposal supports source brokers up to version 2.1
included"
This KIP needs topicId and KIP-516 was introduced later than 2.1.
Jun
On Thu, Aug 13, 2026 at 5:14 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the good comments.
> Answering them below.
>
> > JR59.1 Could you add the RPC to the workflow? The broker first needs to
> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
> mirror name and then issue DescribeClusterMirrors to get the LME.
>
> Yes, you’re right. The workflow is updated to make them clear.
>
> > JR71. mirror.support.unclean.leader.election : The name is confusing.
> It's
> also used in LOG_TRUNCATION, which is not related to unclean leader
> election. Could we use a better name?
>
> I personally think the name should be fine because even though we use in
> “LOG_TRUNCATION” state, the config is to allow us to know if users want to
> support unclean leader election, if so, we wait for all replicas to catch
> up. If we want to explicitly show the intention of the config, how about
> “mirror.full.replica.convergence.required”? I’m open to other suggestions.
>
>
> > JR72. DescribeClusterMirrors
> > JR72.1 A user may specify just the topic name when describing a mirror.
> Should we include the topic name in the request?
>
> Hmm.. we didn’t think about the use case. But yes, if users only care
> about the state of a topic, they don’t need the info for other topics. I’ve
> combined with JR72.2, added “TopicPartitions” filed into
> DescribeClusterMirrorsRequest, and by default is null, which means all
> topic partitions under the requested mirrors will be returned. If set,
> we’ll query the mirrors and return the topic partitions that users
> expected. KIP is updated.
>
>
> > JR72.2 LastMirrorEpochLookups is a bit weird since the response returns
> either mirror state or mirror lag and LastMirrorEpochLookups doesn't match
> either.
>
> Yes, after the recent refactor, I think the LastMirrorEpochLookups can be
> merged into one level up, combining with the topic name filter in above
> comment, which will allow users to query down to the partition level. The
> LastMirrorEpochLookups will query along with normal mirror state by the
> readMirrorState from the coordinator. The difference is that after
> receiving the response, it will filter out the mirrors that don’t match the
> requested sourceClusterId, and then add the lastMirrorEpoch into
> DescribeClusterMirrorResponse. KIP is updated.
>
>
> > JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
>
> Right. Added.
>
>
> > JR72.4 "includeMirrorLags: source offset, destination offset, lag"
> DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
> should be omitted.
>
> Sorry, I removed the “SourceOffset” and “DestinationOffset” fields from
> DescribeClusterMirrorsResponse because I added them into ReadMirrorOffsets
> (ex-ReadMirrorLags) API. But I forgot we still need them in the
> DescribeClusterMirrorsResponse so that the adminClient can receive the
> results. KIP is updated.
>
>
> > JR73. MirrorPartitionState record, LastMirrorEpochs record and
> MirrorPartitionStateRequest: The values for type and name are incorrect.
>
> Ah, that’s my bad. Fixed.
>
> > JR74. ListClusterMirrorsRequest
> > JR74.1 When is DesiredMirrorStateFilter used? There is no option to
> specify
> the desired state in the mirror-topic command.
>
> Yes, referred to kafka-consumer-groups.sh, I added the `--state` option to
> allow users to set the state when listMirror.
>
> > JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>
> Yes, KIP updated.
>
> > JR75. ReadMirrorLags
> > JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
> include lag?
>
> Indeed. Updated.
>
> > JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
>
> Yes, added
>
> > JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to be
> consistent.
>
> Thanks for pointing it out. KIP is updated.
>
> > JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
>
> Yes, we should validate the mirror name before writing the data. KIP is
> updated.
>
> > JR77. Delete Mirror: "3. The controller verifies that no topics are in
> MIRRORING or PAUSED state."
> Is this necessary since step 2 has verified all partitions are in STOPPED
> state, which can only happen if the topics are in STOPPED state.
>
> It’s true, but there could be some delay between “actual state” and
> “desired state”. In step (2), we verify all partitions’ “actual state” are
> in STOPPED state. But the current desired state could be MIRRORING, it’s
> just that the mirror partition leaders haven’t got the metadata update yet.
> So, the controller still needs to verify the desired state is not in
> MIRRORING or PAUSED. I found we didn’t specify that the controller is
> actually verifying the “desired state”, not “actual state”. And this
> controller’s desired state validation also applies in
> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
> I’ve updated the KIP to make it clear.
>
> > JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
> bumped,
> ABORT markers are appended for all ongoing transactions, last mirror epochs
> are persisted, and a MIRROR_PID_RESET record appended."
> This describes the state for STOPPED, not STOPPING.
>
> No, these are the tasks during STOPPING state, like what we described in
> “Stop Mirror Topics” workflow. If we do these in STOPPED state, we can’t
> safely move the readonly topic into writable because not all required tasks
> are done. I think I know why it confuses you, it’s because all the tasks
> are in the completed state. I’ve rephrased it to make it clear.
>
>
> > JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
> timeout field?
>
> Make sense. Added into `DescribeClusterMirrors` request, as well as
> startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
> because we will query the partition states from coordinator for state
> validation in these APIs.
>
>
> > JR80. PauseMirrorTopicsOptions: Should it support patterns? The command
> line seems to suppor it.
>
> Sorry, it’s a little confusing here. The “pattern” in
> startMirrorTopicsOptions or stopMirrorTopicsOptions is to update the
> “mirror.topics.include” and “mirror.topics.exclude” configuration for topic
> pattern discovery periodically. For the PauseMirrorTopics and
> ResumeMirrorTopics in the clusterMirrorCommand, users can provide regex in
> the “--topics” option, and the clusterMirrorCommand will listTopic in the
> destination cluster and match with the patterns users provided, and then
> feeds the matched topics into the PauseMirrorTopics/ResumeMirrorTopics
> requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics supports
> regex in the ClusterMirrorComamnd, but we don’t need to add the “patterns”.
>
>
> > JR81. AdminClient
> > JR81.1 ListClusterMirrors: We need to add the option to specify different
> filters supported in the request.
>
> We already have the `--mirror` option. Added `--state` and
> `--sourceClusterId` options to support the other filters.
>
> > JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
> request doesn't support this option. Could we add other options supported
> in the request?
>
> Currently, we have “includeAuthorizedOperations” in DescribeCluster,
> DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
> DescribeStreamsGroups requests. But none of them include the option in the
> tool to allow users to enable it. I think the reason is that we don’t want
> to show this to users on the tool output. But having this field, we allow
> the advanced users to invoke admin API with includeAuthorizedOperations
> enabled to get the results they want.
>
>
> > JR81.3 LeaderStateDescription: Should it be named
> MirrorPartitionStateDescription? Also, could we make the fields consistent
> with those in the response?
>
> OK. KIP is updated.
>
>
> Thank you,
> Luke
>
> On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Luke,
>>
>> Thanks for the reply.
>>
>> JR59.1 Could you add the RPC to the workflow? The broker first needs to
>> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
>> mirror name and then issue DescribeClusterMirrors to get the LME.
>>
>> JR71. mirror.support.unclean.leader.election : The name is confusing. It's
>> also used in LOG_TRUNCATION, which is not related to unclean leader
>> election. Could we use a better name?
>>
>> JR72. DescribeClusterMirrors
>> JR72.1 A user may specify just the topic name when describing a mirror.
>> Should we include the topic name in the request?
>> JR72.2 LastMirrorEpochLookups is a bit weird since the response returns
>> either mirror state or mirror lag and LastMirrorEpochLookups doesn't match
>> either.
>> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> JR72.4 "includeMirrorLags: source offset, destination offset, lag"
>> DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
>> should be omitted.
>>
>> JR73. MirrorPartitionState record, LastMirrorEpochs record and
>> MirrorPartitionStateRequest: The values for type and name are incorrect.
>>
>> JR74. ListClusterMirrorsRequest
>> JR74.1 When is DesiredMirrorStateFilter used? There is no option to
>> specify
>> the desired state in the mirror-topic command.
>> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>>
>> JR75. ReadMirrorLags
>> JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
>> include lag?
>> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
>> JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to be
>> consistent.
>>
>> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
>>
>> JR77. Delete Mirror: "3. The controller verifies that no topics are in
>> MIRRORING or PAUSED state."
>> Is this necessary since step 2 has verified all partitions are in STOPPED
>> state, which can only happen if the topics are in STOPPED state.
>>
>> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
>> bumped,
>> ABORT markers are appended for all ongoing transactions, last mirror
>> epochs
>> are persisted, and a MIRROR_PID_RESET record appended."
>> This describes the state for STOPPED, not STOPPING.
>>
>> JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
>> timeout field?
>>
>> JR80. PauseMirrorTopicsOptions: Should it support patterns? The command
>> line seems to suppor it.
>>
>> JR81. AdminClient
>> JR81.1 ListClusterMirrors: We need to add the option to specify different
>> filters supported in the request.
>> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
>> request doesn't support this option. Could we add other options supported
>> in the request?
>> JR81.3 LeaderStateDescription: Should it be named
>> MirrorPartitionStateDescription? Also, could we make the fields consistent
>> with those in the response?
>>
>> Jun
>>
>> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for the good comments again.
>> > Answering them below.
>> >
>> > > JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election
>> is
>> > enabled and the destination leader has log truncation, the state will
>> move
>> > from MIRRORING to this state."
>> > This still suggests that divergeEpoch is only handled if
>> > mirror.support.unclean.leader.election
>> > is enabled.
>> >
>> >
>> > OK, KIP is updated to make it clear.
>> >
>> > > JR55. Should the mirror state and the mirror lags always be exposed
>> > together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
>> > from DescribeClusterMirrorsRequest. If not, it's better to use two
>> separate
>> > RPCs.
>> >
>> >
>> > No, we sometimes only need MirrorState, and sometimes need both. So I’ll
>> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
>> > DescribeClusterMirrorsRequest. And we will use ReadMirrorStates and
>> > ReadMirrorLags to get each info.
>> >
>> >
>> > > DescribeClusterMirrorsRequest should be used in the Describe Mirror
>> flow,
>> > right? Currently it's missing. Also, I assume the admin client refers to
>> > the one in the kafka-cluster-mirrors tool. But it seems that sometimes
>> it
>> > also refers to the admin client on the broker? Perhaps we can use 'the
>> > tool' instead to make it clear.
>> >
>> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
>> > And sorry for the confusion. I’d like to keep the admin client as the
>> one
>> > invoking from the tool (or user’s direct admin client API invocation).
>> On
>> > the broker side, we actually use InterBrokerSendThread to invoke these
>> RPCs
>> > or get directly from MetadataImage. I’ve updated the KIP to make it
>> clear.
>> >
>> >
>> > > JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
>> > instead of just to the controller?
>> >
>> > Yes, we only need to issue to the controller. Updated to the
>> > BumpLeaderEpochs API.
>> >
>> >
>> > > JR59. Start Mirror Topics:
>> > > JR59.1 "During this state, the broker queries the source cluster for
>> last
>> > mirror epochs".
>> > Could you describe which RPC is issued to find the LME?
>> >
>> > We’re using describeClusterMirror API via admin client here.
>> >
>> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
>> > doesn't
>> > contain a replication factor.
>> >
>> > We use “default.replication.factor” when topic creation, so the
>> > INVALID_REPLICATION_FACTOR will be thrown like normal topic creation.
>> Also,
>> > because we require the replicas for the partition assignment to be all
>> > active and not in fenced or controlled shutdown(JR35), we’ll throw
>> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
>> >
>> >
>> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
>> > different topic name"
>> > Should we also return this error when the topic name is used as a
>> regular
>> > topic?
>> >
>> > There are 2 errors we will throw when topic id <-> topic name mismatch.
>> > Suppose provided topic id: ID_A, topic name: NAME_A
>> > 1. Destination cluster exists topic id: ID_A, but topic name is: NAME_B
>> > TOPIC_ALREADY_EXISTS will be thrown
>> > 2. Destination cluster exists topic id: ID_B, but topic name is: NAME_A
>> > INCONSISTENT_TOPIC_ID will be thrown
>> >
>> > Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS error
>> for
>> > these 2 cases makes more sense. KIP updated.
>> >
>> >
>> > > JR60. CreateClusterMirrorRequest: When will it receive an
>> > INVALID_CLUSTER_MIRROR_STATES error not covered by
>> > CLUSTER_MIRROR_ALREADY_EXISTS?
>> >
>> >
>> > Sorry, createClusterMirrorRequest doesn’t throw
>> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate the
>> mirror
>> > states when creating a cluster mirror. We only care if the “mirror name”
>> > exists or not. KIP updated.
>> >
>> >
>> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
>> remove
>> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
>> > specific state causing the error.
>> >
>> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following errors:
>> > a. MIRROR_TOPIC_ALREADY_PAUSED
>> > b. MIRROR_TOPIC_NOT_PAUSED,
>> > c. MIRROR_TOPIC_ALREADY_STOPPED,
>> > d. CLUSTER_MIRROR_NOT_EMPTY
>> > e. MIRROR_TOPIC_NOT_STOPPED
>> >
>> > KIP is updated.
>> >
>> >
>> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
>> requests
>> > (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics). Could
>> we
>> > make them consistent?
>> >
>> >
>> > Good catch. The updated errors for the new added RPC are this:
>> >
>> > CreateClusterMirror: NOT_CONTROLLER
>> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>> >
>> >
>> > > JR63. StopMirrorTopics:
>> > > JR63.1 "The broker validates that all target topic partitions are in
>> > either
>> > LOG_TRUNCATION or MIRRORING or FAILED state."
>> > Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
>> >
>> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is updated.
>> >
>> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
>> PAUSED
>> > -> STOPPED is valid?
>> >
>> > Yes, it is not correct. KIP updated to remove this error.
>> >
>> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
>> > state"
>> > This is weird. The reason to issue StopMirrorTopics is that it's not in
>> > stopped state.
>> >
>> > Indeed. KIP is updated to only include INVALID_CLUSTER_MIRROR_STATES
>> > error, and will be thrown when states are not in MIRRORING or PAUSED
>> state.
>> >
>> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's
>> derived
>> > from SourceOffset and DestinationOffset.
>> >
>> > True. Removed.
>> >
>> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
>> > could just set the partition level state to STOPPED.
>> >
>> > Nice catch. We don’t use it at all. That is some old leftover field we
>> > forgot to remove. KIP updated.
>> >
>> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent
>> to
>> > the controller?
>> >
>> > No, these 2 only send to the broker. KIP updated.
>> >
>> >
>> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch?
>> It
>> > seems both are used for fencing and fencing based on the latter is
>> > stronger.
>> >
>> > Yes, basically the StateEpoch can fence for most cases. The LeaderEpoch
>> > fencing is used for cases like:
>> >
>> > 1. b1 is the current leader for foo-0, preparing write state from
>> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
>> > 2. leadership change to b2 for foo-0, LE changes to 2.
>> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1 }.
>> > 4. The coordinator receives the read from b2 in step (3), the highest
>> > lastSeen LE for foo-0 updates to 2 in coordinator.
>> > 5. The writes state from b1 in step (1) arrives the coordinator, but it
>> > rejects b1 writes because the highest lastSeen LE is 2, which is > 1 in
>> the
>> > request.
>> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2, stateEpoch: 1 and
>> > succeeds.
>> >
>> > The leaderEpoch fence is to avoid any stale broker writes after the new
>> > leader epoch read. This cannot be protected by the stateEpoch. The
>> > stateEpoch can prevent the write with older stateEpoch. So in the
>> situation
>> > above, the stateEpoch cannot determine if we should accept b1 writes or
>> > not, but leaderEpoch can. By the way, we borrowed this concept from
>> > KIP-932:
>> >
>> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
>> <https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!u8EkPGXm8_VKdltXy8lrIW14mR2kgYIoaBPD7LO2uAN6rjhrfCX37Pfa6r85QPpSwSPh4V6eW_bKlns$>
>> > <
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
>> >
>> >
>> >
>> > > JR68. Could we make the usage of topicId and topicName in the RPCs
>> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> > BumpLeaderEpochsReponse uses TopicName.
>> >
>> > Yes, we have made:
>> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use TopicName.
>> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse use
>> > TopicName.
>> > 3. MirrorTopicStateChangeRecord uses TopicName because we will do
>> > validation before writing the metadata log.
>> >
>> >
>> > > JR69. MIRROR_PID_RESET:
>> > > JR69.1 Could you specify the value of the control record's type?
>> >
>> > It’s MIRROR_PID_RESET(7). KIP updated.
>> >
>> > > JR69.2 How is the SourceClusterId field used?
>> >
>> > I’m also surprised to see this field in the MirrorPidResetRecord. I
>> traced
>> > back to the internal PR we added for this field, and confirmed it is not
>> > used since it was added. Sorry for the confusion and nice catch! KIP
>> > updated.
>> >
>> > > JR70. MirrorPartitionState: type should be "data" not
>> "coordinator-key",
>> > right? Ditto for LastMirrorEpochs.
>> > We store both MirrorPartitionState and LastMirrorEpochs records into
>> > internal __mirror_state topic, and rely on Coordinator Runtime to
>> > write/read the records, like how `OffsetCommit` and `GroupMetadata`
>> work.
>> > So I think we should keep them as "coordinator-key"/"coordinator-value"
>> > type.
>> >
>> >
>> > Thank you,
>> > Luke
>> >
>> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <de...@kafka.apache.org>
>> > wrote:
>> >
>> >> Hi, Luke,
>> >>
>> >> Thanks for the reply.
>> >>
>> >> JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election
>> is
>> >> enabled and the destination leader has log truncation, the state will
>> move
>> >> from MIRRORING to this state."
>> >> This still suggests that divergeEpoch is only handled if
>> >> mirror.support.unclean.leader.election
>> >> is enabled.
>> >>
>> >> JR55. Should the mirror state and the mirror lags always be exposed
>> >> together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
>> >> from DescribeClusterMirrorsRequest. If not, it's better to use two
>> >> separate
>> >> RPCs.
>> >>
>> >> DescribeClusterMirrorsRequest should be used in the Describe Mirror
>> flow,
>> >> right? Currently it's missing. Also, I assume the admin client refers
>> to
>> >> the one in the kafka-cluster-mirrors tool. But it seems that sometimes
>> it
>> >> also refers to the admin client on the broker? Perhaps we can use 'the
>> >> tool' instead to make it clear.
>> >>
>> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
>> >> instead of just to the controller?
>> >>
>> >> JR59. Start Mirror Topics:
>> >> JR59.1 "During this state, the broker queries the source cluster for
>> last
>> >> mirror epochs".
>> >> Could you describe which RPC is issued to find the LME?
>> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
>> >> doesn't
>> >> contain a replication factor.
>> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
>> >> different topic name"
>> >> Should we also return this error when the topic name is used as a
>> regular
>> >> topic?
>> >>
>> >> JR60. CreateClusterMirrorRequest: When will it receive an
>> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
>> >> CLUSTER_MIRROR_ALREADY_EXISTS?
>> >>
>> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
>> remove
>> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
>> >> specific state causing the error.
>> >>
>> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
>> requests
>> >> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics).
>> Could we
>> >> make them consistent?
>> >>
>> >> JR63. StopMirrorTopics:
>> >> JR63.1 "The broker validates that all target topic partitions are in
>> >> either
>> >> LOG_TRUNCATION or MIRRORING or FAILED state."
>> >> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
>> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
>> PAUSED
>> >> -> STOPPED is valid?
>> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
>> >> state"
>> >> This is weird. The reason to issue StopMirrorTopics is that it's not in
>> >> stopped state.
>> >>
>> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's derived
>> >> from SourceOffset and DestinationOffset.
>> >>
>> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
>> >> could just set the partition level state to STOPPED.
>> >>
>> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent
>> to
>> >> the controller?
>> >>
>> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch? It
>> >> seems both are used for fencing and fencing based on the latter is
>> >> stronger.
>> >>
>> >> JR68. Could we make the usage of topicId and topicName in the RPCs
>> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> >> BumpLeaderEpochsReponse uses TopicName.
>> >>
>> >> JR69. MIRROR_PID_RESET:
>> >> JR69.1 Could you specify the value of the control record's type?
>> >> JR69.2 How is the SourceClusterId field used?
>> >>
>> >> JR70. MirrorPartitionState: type should be "data" not
>> "coordinator-key",
>> >> right? Ditto for LastMirrorEpochs.
>> >>
>> >> Jun
>> >>
>> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com> wrote:
>> >>
>> >> > Hi Jun,
>> >> >
>> >> > Thanks for the comments.
>> >> > Answering them below.
>> >> >
>> >> > > JR36. Sounds good. Should we remove
>> >> MirrorLeaderEpochExceededException?
>> >> >
>> >> > Sorry, it’s a confusing name. The MirrorLeaderEpochExceededException
>> is
>> >> > used and thrown when the fetched mirror batch leader epoch > local
>> >> leader
>> >> > epoch in destination cluster. And then we’ll stop mirroring and move
>> the
>> >> > partition to EPOCH_FENSING state to wait for local leader epoch
>> bump. I
>> >> > think since we don’t have the concept of `MirrorLeaderEpoch` anymore,
>> >> the
>> >> > `MirrorLeaderEpochExceededException` should not cause confusion now.
>> >> I’ll
>> >> > keep the name for the exception.
>> >> >
>> >> >
>> >> > > JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
>> >> > mirror
>> >> > fetch request issued to the source cluster, independent of the value
>> of
>> >> > mirror.support.unclean.leader.election? It's needed to detect
>> divergence
>> >> > during failover. However, if it's also set when
>> >> > mirror.support.unclean.leader.election=false,
>> >> > an unclean leader election in the source cluster can halt the
>> mirroring
>> >> > since we don’t do anything with the returned divergeEpoch in that
>> mode
>> >> > and continue
>> >> > the next fetch.
>> >> >
>> >> > > Do we plan to always set lastFetchedEpoch in the mirror
>> >> > fetch request issued to the source cluster, independent of the value
>> of
>> >> > Mirror.support.unclean.leader.election?
>> >> >
>> >> > Yes, we always set lastFetchedEpoch in the mirror fetch request.
>> >> >
>> >> > > However, if it's also set when
>> >> > mirror.support.unclean.leader.election=false,
>> >> > an unclean leader election in the source cluster can halt the
>> mirroring
>> >> > since we don’t do anything with the returned divergeEpoch in that
>> mode
>> >> > and continue the next fetch.
>> >> >
>> >> > Sorry I didn’t make it clear. When
>> >> > “mirror.support.unclean.leader.election=false” and a “divergeEpoch”
>> is
>> >> set
>> >> > in fetch response, the leader node will handle the divergeEpoch and
>> then
>> >> > continue the next fetch as usual. The difference with
>> >> > “mirror.support.unclean.leader.election=true” is that “should we wait
>> >> for
>> >> > ALL replicas caught up (complete truncation)”. Without the waiting,
>> it’s
>> >> > possible the inconsistent log issue in KAFKA-18723 could happen
>> between
>> >> > leader and follower nodes in the destination cluster because we don’t
>> >> have
>> >> > “MirrorLeaderEpoch” concept now to reject higher leader epoch
>> batches.
>> >> That
>> >> > is, we won't halt the mirroring if "divergeEpoch" is set but
>> >> > “mirror.support.unclean.leader.election=false”.
>> >> >
>> >> >
>> >> > > JR55. The DescribeClusterMirrorsRequest still returns both the
>> mirror
>> >> > partition state and the mirror partition leader state. The problem is
>> >> that
>> >> > if a caller only needs one of the state, the other state becomes
>> >> redundant.
>> >> > For example, when starting the mirror, to obtain LME, we only need
>> the
>> >> > mirror partition state. Here is an alternative design: We expose the
>> >> mirror
>> >> > partition state through ReadMirrorState. The client will use the
>> >> > FindCoordinatorRequest
>> >> > to find the right mirror partition coordinator to issue that
>> request. We
>> >> > expose the mirror partition leader state through the
>> >> ReadClusterMirrorLags
>> >> > request. The client can use the metadata request to find the
>> partition
>> >> > leader to issue that request.
>> >> >
>> >> > > In the describing mirror flow, the admin client first issues the
>> >> > ListMirrorRequest to find all the mirrors and the topics being
>> >> mirrored. It
>> >> > then issues the metadata request to find the partitions for those
>> topics
>> >> > and issues a FindCoordinatorRequest to find the mirror partition
>> state
>> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
>> partition
>> >> > state coordinator and ReadClusterMirrorLags to the partition leader.
>> >> >
>> >> > > In the starting mirror flow, the partition leader in the
>> destination
>> >> > cluster issues a ListMirrorRequest to find the mirror name based on
>> its
>> >> > cluster id. It then issues a FindCoordinatorRequest to find the
>> mirror
>> >> > partition state coordinators. Finally, it issues ReadMirrorState to
>> the
>> >> > mirror partition state coordinator to get the LME.
>> >> >
>> >> > > We can also improve the ListMirror request to accept a filter
>> based on
>> >> > mirror name, source cluster ID and topic name.
>> >> >
>> >> >
>> >> > Answers:
>> >> >
>> >> > > The DescribeClusterMirrorsRequest still returns both the mirror
>> >> > partition state and the mirror partition leader state. The problem is
>> >> that
>> >> > if a caller only needs one of the state, the other state becomes
>> >> redundant.
>> >> > For example, when starting the mirror, to obtain LME, we only need
>> the
>> >> > mirror partition state.
>> >> >
>> >> > Good point!
>> >> >
>> >> >
>> >> > > Here is an alternative design: We expose the mirror
>> >> > partition state through ReadMirrorState. The client will use the
>> >> > FindCoordinatorRequest
>> >> > to find the right mirror partition coordinator to issue that
>> request. We
>> >> > expose the mirror partition leader state through the
>> >> ReadClusterMirrorLags
>> >> > request. The client can use the metadata request to find the
>> partition
>> >> > leader to issue that request.
>> >> >
>> >> >
>> >> > Sounds good. But we try not to expose
>> >> > ReadMirrorState/ReadClusterMirrorLags directly to the admin client
>> >> because
>> >> > users might get confused about these 2 requests, and they are also
>> both
>> >> > related to describeClusterMirror. I think we can add 2 additional
>> >> fields to
>> >> > describeClusterMirrorRequest: “includeMirrorStates” and
>> >> > “includeMirrorLags”. So the admin client can include either one or
>> both
>> >> of
>> >> > them in the DescrbieClusterMirrorRequest based on the provided
>> options.
>> >> KIP
>> >> > is updated.
>> >> >
>> >> >
>> >> > > In the describing mirror flow, the admin client first issues the
>> >> > ListMirrorRequest to find all the mirrors and the topics being
>> >> mirrored. It
>> >> > then issues the metadata request to find the partitions for those
>> topics
>> >> > and issues a FindCoordinatorRequest to find the mirror partition
>> state
>> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
>> partition
>> >> > state coordinator and ReadClusterMirrorLags to the partition leader.
>> >> >
>> >> > Makes sense. KIP is updated.
>> >> >
>> >> >
>> >> > > In the starting mirror flow, the partition leader in the
>> destination
>> >> > cluster issues a ListMirrorRequest to find the mirror name based on
>> its
>> >> > cluster id. It then issues a FindCoordinatorRequest to find the
>> mirror
>> >> > partition state coordinators. Finally, it issues ReadMirrorState to
>> the
>> >> > mirror partition state coordinator to get the LME.
>> >> >
>> >> > Make sense. So we can send DescribeClusterMirrorRequest with
>> >> > “includeMirrorStates” only.
>> >> >
>> >> > > We can also improve the ListMirror request to accept a filter
>> based on
>> >> > mirror name, source cluster ID and topic name.
>> >> >
>> >> > Good suggestion, just like what ListGroupsRequest does.
>> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
>> >> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP
>> >> updated.
>> >> >
>> >> >
>> >> > > JR56. Could you list the ACL needed for each new RPC?
>> >> >
>> >> >
>> >> > Yes, it’s already listed in “Security Control” section.
>> >> >
>> >> >
>> >> > > JR57. In the command workflow, it sometimes says "The user sends"
>> and
>> >> > some
>> >> > other times says "The admin client sends". It's better to use "The
>> admin
>> >> > client" consistently in all those places.
>> >> >
>> >> > Agree. KIP updated.
>> >> >
>> >> >
>> >> > Thank you,
>> >> > Luke
>> >> >
>> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <dev@kafka.apache.org
>> >
>> >> > wrote:
>> >> >
>> >> >> Hi, Luke,
>> >> >>
>> >> >> Thanks for the reply.
>> >> >>
>> >> >> JR36. Sounds good. Should we remove
>> MirrorLeaderEpochExceededException?
>> >> >>
>> >> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
>> >> mirror
>> >> >> fetch request issued to the source cluster, independent of the
>> value of
>> >> >> mirror.support.unclean.leader.election? It's needed to detect
>> >> divergence
>> >> >> during failover. However, if it's also set when
>> >> >> mirror.support.unclean.leader.election=false,
>> >> >> an unclean leader election in the source cluster can halt the
>> mirroring
>> >> >> since we don’t do anything with the returned divergeEpoch in that
>> mode
>> >> >> and continue
>> >> >> the next fetch.
>> >> >>
>> >> >> JR55. The DescribeClusterMirrorsRequest still returns both the
>> mirror
>> >> >> partition state and the mirror partition leader state. The problem
>> is
>> >> that
>> >> >> if a caller only needs one of the state, the other state becomes
>> >> >> redundant.
>> >> >> For example, when starting the mirror, to obtain LME, we only need
>> the
>> >> >> mirror partition state. Here is an alternative design: We expose the
>> >> >> mirror
>> >> >> partition state through ReadMirrorState. The client will use the
>> >> >> FindCoordinatorRequest
>> >> >> to find the right mirror partition coordinator to issue that
>> request.
>> >> We
>> >> >> expose the mirror partition leader state through the
>> >> ReadClusterMirrorLags
>> >> >> request. The client can use the metadata request to find the
>> partition
>> >> >> leader to issue that request.
>> >> >>
>> >> >> In the describing mirror flow, the admin client first issues the
>> >> >> ListMirrorRequest to find all the mirrors and the topics being
>> >> mirrored.
>> >> >> It
>> >> >> then issues the metadata request to find the partitions for those
>> >> topics
>> >> >> and issues a FindCoordinatorRequest to find the mirror partition
>> state
>> >> >> coordinators. Finally, it issues ReadMirrorState to the mirror
>> >> partition
>> >> >> state coordinator and ReadClusterMirrorLags to the partition leader.
>> >> >>
>> >> >> In the starting mirror flow, the partition leader in the destination
>> >> >> cluster issues a ListMirrorRequest to find the mirror name based on
>> its
>> >> >> cluster id. It then issues a FindCoordinatorRequest to find the
>> mirror
>> >> >> partition state coordinators. Finally, it issues ReadMirrorState to
>> the
>> >> >> mirror partition state coordinator to get the LME.
>> >> >>
>> >> >> We can also improve the ListMirror request to accept a filter based
>> on
>> >> >> mirror name, source cluster ID and topic name.
>> >> >>
>> >> >> JR56. Could you list the ACL needed for each new RPC?
>> >> >>
>> >> >> JR57. In the command workflow, it sometimes says "The user sends"
>> and
>> >> some
>> >> >> other times says "The admin client sends". It's better to use "The
>> >> admin
>> >> >> client" consistently in all those places.
>> >> >>
>> >> >> Jun
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com> wrote:
>> >> >>
>> >> >> > Hi Jun,
>> >> >> >
>> >> >> > Thanks for your comments.
>> >> >> > Answering them below.
>> >> >> >
>> >> >> >
>> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> >> specifically
>> >> >> > designed for the unclean leader election support in Cluster
>> Mirroring
>> >> >> > because the unclean leader election will break the assumption that
>> >> the
>> >> >> > committed data won’t be truncated."
>> >> >> > If the source cluster has an unclean leader election, the
>> destination
>> >> >> > cluster will detect the inconsistency and truncate the log at the
>> >> >> > destination leader. We can just
>> >> >> > extend mirror.support.unclean.leader.election to cover this case,
>> >> i.e.,
>> >> >> we
>> >> >> > wait until all replicas' log end offsets are reduced to the target
>> >> >> offset
>> >> >> > before resuming mirroring. This avoids the inconsistency.
>> >> >> >
>> >> >> > If the destination cluster has an unclean leader election, it will
>> >> >> > re-fetch the same data from the source. There won't be any
>> >> inconsistency
>> >> >> > either.
>> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >> >
>> >> >> >
>> >> >> > Yes, that works, too. I agree this is better because the unclean
>> >> leader
>> >> >> > election is rare, we don't want to occupy a field just for it.
>> >> >> >
>> >> >> > So to achieve this, we have to:
>> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
>> >> >> > 2. When the destination cluster leader completes a log truncation
>> >> after
>> >> >> a
>> >> >> > fetch response due to divergeEpoch, we know it must be caused by
>> >> unclean
>> >> >> > leader election in the source cluster because we only fetch
>> committed
>> >> >> > records. If “mirror.support.unclean.leader.election=false”, we
>> don’t
>> >> do
>> >> >> > anything and continue the next fetch. If
>> >> >> > “mirror.support.unclean.leader.election=true”, we remove the
>> mirror
>> >> >> fetcher
>> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
>> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be
>> >> caught
>> >> >> up
>> >> >> > into ISR and reduced to the expected LEO.
>> >> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
>> >> >> >
>> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch API.
>> >> >> > KIP is updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR51. Make sense. To truncate based on LME, we need to issue
>> >> another
>> >> >> RPC
>> >> >> > to translate the LME to an offset. It would be useful to include
>> >> that in
>> >> >> > the workflow description.
>> >> >> >
>> >> >> > About the additional RPC to translate the LME to the offset (i.e.
>> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on this to do
>> >> the
>> >> >> > offset translation. We use the local leader epoch cache in the
>> >> >> destination
>> >> >> > cluster to translate the epoch to save the RPC call. This works
>> >> because
>> >> >> of
>> >> >> > an invariant the LME truncation protocol maintains: for any leader
>> >> >> epoch ≤
>> >> >> > LME, that epoch's data belongs to exactly one cluster, either it
>> >> >> originated
>> >> >> > there, or it was mirrored. This holds even across repeated mirror
>> ->
>> >> >> > reverse-mirror -> mirror cycles, since each direction's
>> >> LOG_TRUNCATION
>> >> >> > phase re-establishes the invariant before that cluster starts
>> >> mirroring.
>> >> >> > With this invariant, the replication protocol can find out the
>> >> diverge
>> >> >> > point and truncate the log correctly when mirroring. For example:
>> >> >> >
>> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> > foo-0 leader log foo-0 leader log
>> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> > Offset 2: A (LE: 0)
>> >> >> > Offset 3: A (LE: 1)
>> >> >> >
>> >> >> > Cluster A crashes while Cluster B mirrors to offset 1. Failover
>> to B
>> >> and
>> >> >> > store the LME=0 in cluster B. When cluster A recovers and starts
>> the
>> >> >> > reverse mirroring, it’ll query the LME from B and get back LME: 0.
>> >> Then
>> >> >> > cluster A gets the end offset for epoch 0 from leaderEpochCache,
>> and
>> >> >> gets
>> >> >> > offset 2. So offset 3 is truncated. The log is like this now:
>> >> >> >
>> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> > foo-0 leader log foo-0 leader log
>> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
>> >> >> > Offset 3: B (LE: 10)
>> >> >> >
>> >> >> > When mirror starts, cluster A sends the fetch request with {
>> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the divergingEpoch
>> >> check
>> >> >> in
>> >> >> > the cluster B queries the endOffset for leader epoch 0, and found
>> the
>> >> >> end
>> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns
>> fetch
>> >> >> > response with divergingEpoch info to cluster A for log
>> truncation. In
>> >> >> the
>> >> >> > end, offset 2 in cluster A will be truncated and then log
>> converges
>> >> with
>> >> >> > cluster B.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit
>> strange
>> >> >> > because it mixes querying the mirror partition state with querying
>> >> the
>> >> >> > mirror partition leader. Another option is to use
>> >> >> ReadMirrorStatesRequest
>> >> >> > to query the mirror partition state and a new request to query the
>> >> >> mirror
>> >> >> > partition leader. If we want the adminClient to query the right
>> >> >> ClusterMirrorCoordinator
>> >> >> > directly, we can extend the FindCoordinatorRequest to support
>> that.
>> >> >> >
>> >> >> >
>> >> >> > Good suggestion. Yes, we need to fan out to all brokers because we
>> >> have
>> >> >> to
>> >> >> > collect info from mirror partition leaders and mirror
>> coordinators.
>> >> If
>> >> >> we
>> >> >> > create a new API to contact mirror partition leaders, it will make
>> >> the
>> >> >> API
>> >> >> > cleaner and clearer. So we will do this:
>> >> >> >
>> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields from
>> >> >> > DescribeClusterMirror request/response
>> >> >> >
>> >> >> > b. Add a new API: ReadClusterMirrorLags API to request for
>> >> SourceOffset,
>> >> >> > DestinationOffset, and Lag info.
>> >> >> >
>> >> >> > The workflow for admin client describeClusterMirror will be like
>> >> this:
>> >> >> > 1. adminClient sends describeClusterMirror to the leastLoadedNode
>> >> broker
>> >> >> > 2. The received broker collects the current mirrored partitions,
>> and
>> >> >> > groups them by coordinator node and by partition leader node.
>> >> >> > 3. The broker sends ReadMirrorState request to the coordinator
>> nodes
>> >> to
>> >> >> > get the mirror state related info
>> >> >> > 4. The broker sends ReadClusterMirrorLags request to the mirror
>> >> >> partition
>> >> >> > leader nodes to get the mirror lag, source offset, destination
>> offset
>> >> >> > (which updates on every fetch response).
>> >> >> > 5. The broker merge all the response into describeClusterMirror
>> >> response
>> >> >> > and send to admin client
>> >> >> >
>> >> >> > KIP is updated. Thanks for the suggestion.
>> >> >> >
>> >> >> >
>> >> >> > > JR55. For __mirror_state topic, could we describe whether it is
>> >> >> > compacted?
>> >> >> >
>> >> >> > Yes, it is a compacted topic. I made it much clearer in the
>> >> >> > “ClusterMirrorCoordinator” section.
>> >> >> >
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Luke
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
>> dev@kafka.apache.org
>> >> >
>> >> >> > wrote:
>> >> >> >
>> >> >> >> Hi, Luke,
>> >> >> >>
>> >> >> >> Thanks for the reply.
>> >> >> >>
>> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> >> specifically
>> >> >> >> designed for the unclean leader election support in Cluster
>> >> Mirroring
>> >> >> >> because the unclean leader election will break the assumption
>> that
>> >> the
>> >> >> >> committed data won’t be truncated."
>> >> >> >> If the source cluster has an unclean leader election, the
>> >> destination
>> >> >> >> cluster will detect the inconsistency and truncate the log at the
>> >> >> >> destination leader. We can just
>> >> >> >> extend mirror.support.unclean.leader.election to cover this case,
>> >> >> i.e., we
>> >> >> >> wait until all replicas' log end offsets are reduced to the
>> target
>> >> >> offset
>> >> >> >> before resuming mirroring. This avoids the inconsistency.
>> >> >> >>
>> >> >> >> If the destination cluster has an unclean leader election, it
>> will
>> >> >> >> re-fetch
>> >> >> >> the same data from the source. There won't be any inconsistency
>> >> either.
>> >> >> >>
>> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >> >>
>> >> >> >> JR51. Make sense. To truncate based on LME, we need to issue
>> another
>> >> >> RPC
>> >> >> >> to
>> >> >> >> translate the LME to an offset. It would be useful to include
>> that
>> >> in
>> >> >> the
>> >> >> >> workflow description.
>> >> >> >>
>> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
>> >> >> because
>> >> >> >> it mixes querying the mirror partition state with querying the
>> >> mirror
>> >> >> >> partition leader. Another option is to use
>> ReadMirrorStatesRequest
>> >> to
>> >> >> >> query
>> >> >> >> the mirror partition state and a new request to query the mirror
>> >> >> partition
>> >> >> >> leader. If we want the adminClient to query the right
>> >> >> >> ClusterMirrorCoordinator
>> >> >> >> directly, we can extend the FindCoordinatorRequest to support
>> that.
>> >> >> >>
>> >> >> >> JR55. For __mirror_state topic, could we describe whether it is
>> >> >> compacted?
>> >> >> >>
>> >> >> >> Jun
>> >> >> >>
>> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com>
>> wrote:
>> >> >> >>
>> >> >> >> > Hi Jun,
>> >> >> >> >
>> >> >> >> > Thanks for the comments.
>> >> >> >> > Answering them below:
>> >> >> >> >
>> >> >> >> > > JR31.3 It makes sense for the new partition leader to query
>> the
>> >> >> >> current
>> >> >> >> > mirror state from the local or remote MirrorMetadataManager
>> that
>> >> owns
>> >> >> >> the
>> >> >> >> > partition's state. However, the wording in the KIP is
>> confusing.
>> >> >> >> > MirrorMetadataManager
>> >> >> >> > mirror partition state cache should be populated only by one
>> way,
>> >> >> which
>> >> >> >> is
>> >> >> >> > replaying __mirror_state partitions this broker leads. By
>> saying
>> >> >> "and by
>> >> >> >> > fetching state from remote coordinators via ReadMirrorStates
>> >> RPCs.",
>> >> >> >> it's
>> >> >> >> > as if the mirror partition state cache on each broker stores
>> the
>> >> >> state
>> >> >> >> for
>> >> >> >> > all mirror partitions.
>> >> >> >> >
>> >> >> >> > You’re right. KIP is updated to make it clear.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR36. I understand the intention now, but I still don't
>> think we
>> >> >> need
>> >> >> >> > MirrorLeaderEpoch. Let's
>> >> >> >> > assume the common case: no unclean leader election in the
>> source
>> >> >> cluster
>> >> >> >> > during mirroring. Mirroring only fetches the committed data
>> from
>> >> the
>> >> >> >> > source, which never changes if obtained a second time. In your
>> >> >> example
>> >> >> >> > image, follower broker 1 in the destination cluster can never
>> >> obtain
>> >> >> the
>> >> >> >> > record B(0) because it's never committed in the source and
>> thus is
>> >> >> never
>> >> >> >> > exposed to the destination leader. It's possible for broker 1
>> to
>> >> >> first
>> >> >> >> get
>> >> >> >> > D(1) which is truncated later when broker 0 follows a new
>> leader.
>> >> But
>> >> >> >> the
>> >> >> >> > new leader will refetch the same D(1) from the source cluster.
>> So,
>> >> >> it's
>> >> >> >> ok
>> >> >> >> > for broker 1 to keep D(1).
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Yes, you are correct. Because mirroring only fetches the
>> committed
>> >> >> data
>> >> >> >> > from the source, the MirrorLeaderEpoch is unnecessary for
>> normal
>> >> >> >> mirroring.
>> >> >> >> > Thanks for the clarification. So I think we can say, the
>> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the unclean
>> >> leader
>> >> >> >> > election support in Cluster Mirroring because the unclean
>> leader
>> >> >> >> election
>> >> >> >> > will break the assumption that the committed data won’t be
>> >> truncated.
>> >> >> >> >
>> >> >> >> > Given that, we could gate the MirrorLeaderEpoch field by only
>> >> >> setting it
>> >> >> >> > in the Fetch request when
>> >> >> "mirror.support.unclean.leader.election=true"
>> >> >> >> > (false by default), and skip it entirely for normal mirroring.
>> >> What
>> >> >> do
>> >> >> >> you
>> >> >> >> > think?
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR51. Sounds good. Why do we choose to store the last mirror
>> >> epoch
>> >> >> >> > instead
>> >> >> >> > of a last mirror offset?
>> >> >> >> >
>> >> >> >> > Yes, originally we stored the “last mirror offset” instead of
>> >> “last
>> >> >> >> mirror
>> >> >> >> > epoch”. But later, we found it doesn’t work for unclean leader
>> >> >> election
>> >> >> >> > cases. (yes, `unclean leader election` again :) ) For example:
>> >> >> >> >
>> >> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >> >
>> >> >> >> > Cluster B is mirroring from A. And A crashes and failover to
>> B. B
>> >> >> stores
>> >> >> >> > the “last mirror offset” as 1.
>> >> >> >> > When cluster A recovers, somehow the unclean leader election
>> >> happens,
>> >> >> >> log
>> >> >> >> > is empty in the new leader, and then some more leadership
>> changes
>> >> to
>> >> >> >> make
>> >> >> >> > the leader epoch become 10. Then it starts the reverse mirror
>> >> from B.
>> >> >> >> The
>> >> >> >> > log looks like this:
>> >> >> >> >
>> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
>> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
>> >> >> >> > Offset 2: pid_res
>> (LE:
>> >> >> 10)
>> >> >> >> >
>> >> >> >> > When A starts mirroring from B, log is truncated to last mirror
>> >> >> offset:
>> >> >> >> 1
>> >> >> >> > (no-op), and then fetch from cluster B. The 1st fetch request
>> is
>> >> like
>> >> >> >> this:
>> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset:
>> 2, …
>> >> }.
>> >> >> The
>> >> >> >> > divergingEpoch check in the cluster B queries the endOffset for
>> >> >> leader
>> >> >> >> > epoch 10 (in Partition#readRecords), and found the end offset
>> for
>> >> LE
>> >> >> 10
>> >> >> >> is
>> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch validation
>> >> >> passes
>> >> >> >> and
>> >> >> >> > returns offset 2 to cluster A. In short, relying on “last
>> mirror
>> >> >> offset”
>> >> >> >> > cannot resolve the log diverge issue in unclean leader election
>> >> case.
>> >> >> >> >
>> >> >> >> > If instead, we truncate until “last mirror epoch”: 0, the
>> offset 0
>> >> >> and 1
>> >> >> >> > in cluster A will be truncated as expected and then log
>> converge
>> >> with
>> >> >> >> > cluster B correctly in the end.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Also, about JR53.2:
>> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
>> brokers
>> >> and
>> >> >> >> > merges
>> >> >> >> > the responses client-side."
>> >> >> >> > This is not ideal. Not every broker hosts a mirror partition.
>> So
>> >> >> sending
>> >> >> >> > the request to every broker is wasteful. It's probably better
>> to
>> >> send
>> >> >> >> the
>> >> >> >> > request to a single broker, which can then contact the right
>> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >
>> >> >> >> > While working on this, I found we need to add one more tagged
>> >> field
>> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema because
>> >> >> >> otherwise
>> >> >> >> > the receiving broker doesn’t know if this is the request from
>> >> client
>> >> >> >> and it
>> >> >> >> > needs to contact other brokers, or this is just a forwarded
>> >> request.
>> >> >> The
>> >> >> >> > workflow is like this:
>> >> >> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest to a
>> >> >> >> > leastLoaded broker, which doesn’t contain the "Forwarded"
>> field.
>> >> >> >> > 2. The broker receives the request and sends
>> >> >> >> DescribeClusterMirrorsRequest
>> >> >> >> > with "Forwarded=true" to the responsible brokers and waits for
>> the
>> >> >> >> response
>> >> >> >> > async.
>> >> >> >> > 3. When all responses are received, merge the results and
>> respond
>> >> to
>> >> >> the
>> >> >> >> > client.
>> >> >> >> >
>> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here because it
>> >> doesn’t
>> >> >> >> > contain the lag info in the coordinator.
>> >> >> >> >
>> >> >> >> > The alternative solution is the adminClient sends
>> >> >> >> > DescribeClusterMirrorsRequest to the “responsible brokers only”
>> >> (via
>> >> >> >> > describeTopics), instead of ALL brokers, and merge the results
>> in
>> >> the
>> >> >> >> > client side. This solution doesn’t need to add a new field in
>> the
>> >> >> >> > DescribeClusterMirrorsRequest and still can avoid unnecessary
>> >> >> requests
>> >> >> >> > sent.
>> >> >> >> >
>> >> >> >> > My preference is to keep the merge logic on the admin client
>> side,
>> >> >> which
>> >> >> >> > saves the overhead in the broker side.
>> >> >> >> > What do you think?
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Thank you,
>> >> >> >> > Luke
>> >> >> >> >
>> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
>> >> dev@kafka.apache.org
>> >> >> >
>> >> >> >> > wrote:
>> >> >> >> >
>> >> >> >> >> Hi, Luke,
>> >> >> >> >>
>> >> >> >> >> Thanks for the reply.
>> >> >> >> >>
>> >> >> >> >> JR31.3 It makes sense for the new partition leader to query
>> the
>> >> >> current
>> >> >> >> >> mirror state from the local or remote MirrorMetadataManager
>> that
>> >> >> owns
>> >> >> >> the
>> >> >> >> >> partition's state. However, the wording in the KIP is
>> confusing.
>> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> mirror partition state cache should be populated only by one
>> way,
>> >> >> >> which is
>> >> >> >> >> replaying __mirror_state partitions this broker leads. By
>> saying
>> >> >> "and
>> >> >> >> by
>> >> >> >> >> fetching state from remote coordinators via ReadMirrorStates
>> >> RPCs.",
>> >> >> >> it's
>> >> >> >> >> as if the mirror partition state cache on each broker stores
>> the
>> >> >> state
>> >> >> >> for
>> >> >> >> >> all mirror partitions.
>> >> >> >> >>
>> >> >> >> >> JR36. I understand the intention now, but I still don't think
>> we
>> >> >> need
>> >> >> >> >> MirrorLeaderEpoch. Let's
>> >> >> >> >> assume the common case: no unclean leader election in the
>> source
>> >> >> >> cluster
>> >> >> >> >> during mirroring. Mirroring only fetches the committed data
>> from
>> >> the
>> >> >> >> >> source, which never changes if obtained a second time. In your
>> >> >> example
>> >> >> >> >> image, follower broker 1 in the destination cluster can never
>> >> obtain
>> >> >> >> the
>> >> >> >> >> record B(0) because it's never committed in the source and
>> thus
>> >> is
>> >> >> >> never
>> >> >> >> >> exposed to the destination leader. It's possible for broker 1
>> to
>> >> >> first
>> >> >> >> get
>> >> >> >> >> D(1) which is truncated later when broker 0 follows a new
>> leader.
>> >> >> But
>> >> >> >> the
>> >> >> >> >> new leader will refetch the same D(1) from the source cluster.
>> >> So,
>> >> >> >> it's ok
>> >> >> >> >> for broker 1 to keep D(1).
>> >> >> >> >>
>> >> >> >> >> JR51. Sounds good. Why do we choose to store the last mirror
>> >> epoch
>> >> >> >> instead
>> >> >> >> >> of a last mirror offset?
>> >> >> >> >>
>> >> >> >> >> Jun
>> >> >> >> >>
>> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com>
>> >> wrote:
>> >> >> >> >>
>> >> >> >> >> > Hi Jun,
>> >> >> >> >> >
>> >> >> >> >> > Thanks for the great comments again.
>> >> >> >> >> > Answering your question below.
>> >> >> >> >> >
>> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror partition
>> cache
>> >> >> lives
>> >> >> >> in
>> >> >> >> >> > MirrorMetadataManager. It is populated in two ways: by
>> >> replaying
>> >> >> >> >> > __mirror_state partitions this broker leads, and by fetching
>> >> state
>> >> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> >> > Hmm, a given broker only caches the mirror partition state
>> for
>> >> >> >> >> partitions
>> >> >> >> >> > hashed into the __mirror_state partition hosted on this
>> broker,
>> >> >> >> right?
>> >> >> >> >> Why
>> >> >> >> >> > does it need to fetch the state from other remote
>> coordinators?
>> >> >> >> >> >
>> >> >> >> >> > > a given broker only caches the mirror partition state for
>> >> >> >> partitions
>> >> >> >> >> > hashed into the __mirror_state partition hosted on this
>> broker,
>> >> >> >> right?
>> >> >> >> >> >
>> >> >> >> >> > Right.
>> >> >> >> >> >
>> >> >> >> >> > > Why does it need to fetch the state from other remote
>> >> >> coordinators?
>> >> >> >> >> >
>> >> >> >> >> > Because the data partition leader needs to know the current
>> >> mirror
>> >> >> >> state
>> >> >> >> >> > of the partitions it leads. On a leadership change, the
>> >> metadata
>> >> >> >> delta
>> >> >> >> >> is
>> >> >> >> >> > published to the MirrorMetadataManager, which queries the
>> >> current
>> >> >> >> mirror
>> >> >> >> >> > state for its led partitions from the coordinator (local or
>> >> >> remote),
>> >> >> >> and
>> >> >> >> >> > processes tasks based on that state. This is the reason why
>> it
>> >> >> >> fetches
>> >> >> >> >> the
>> >> >> >> >> > state from other remote coordinator. This part of cache
>> will be
>> >> >> >> cleared
>> >> >> >> >> > after the leadership change.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR36. I am still not quite sure why we need
>> >> MirrorLeaderEpoch in
>> >> >> >> the
>> >> >> >> >> > Fetch
>> >> >> >> >> > request. In the current Fetch request, in addition to
>> >> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> > it has another field LastFetchedEpoch, which captures the
>> last
>> >> >> leader
>> >> >> >> >> epoch
>> >> >> >> >> > in the data and is used to resolve the inconsistency between
>> >> the
>> >> >> >> >> follower
>> >> >> >> >> > and the leader. Does it achieve the same purpose as
>> >> >> >> MirrorLeaderEpoch?
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to do the
>> >> >> >> validation
>> >> >> >> >> to
>> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any fetched
>> >> batches
>> >> >> with
>> >> >> >> >> epoch
>> >> >> >> >> > higher than the LastFetchedEpoch, it means the follower can
>> >> never
>> >> >> >> accept
>> >> >> >> >> > new appended batches with higher epoch in the leader node.
>> >> >> >> >> >
>> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes in
>> >> the
>> >> >> >> fetch
>> >> >> >> >> > API:
>> >> >> >> >> > 1. The receiver (leader node) uses it to validate it matches
>> >> the
>> >> >> >> leader
>> >> >> >> >> > epoch in the leader node
>> >> >> >> >> > 2. The sender (follower node) uses it to validate the epoch
>> in
>> >> the
>> >> >> >> >> fetched
>> >> >> >> >> > batches are all <= this value.
>> >> >> >> >> >
>> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still serving for
>> >> >> >> followers
>> >> >> >> >> in
>> >> >> >> >> > the destination cluster. For purpose (2), my understanding
>> of
>> >> the
>> >> >> >> >> > validation for KAFKA-18723 is like: the follower knows the
>> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader node
>> >> should
>> >> >> >> >> contain in
>> >> >> >> >> > the log batches when fetch request is sent. Any log batch
>> >> beyond
>> >> >> this
>> >> >> >> >> epoch
>> >> >> >> >> > in the fetch response means the follower’s metadata is
>> stale.
>> >> The
>> >> >> >> >> batches
>> >> >> >> >> > need to be rejected and follower needs to update metadata
>> for
>> >> next
>> >> >> >> >> fetch.
>> >> >> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch
>> serves
>> >> >> this
>> >> >> >> >> purpose
>> >> >> >> >> > to let the followers know the highest epoch in the leader
>> node
>> >> >> log in
>> >> >> >> >> each
>> >> >> >> >> > fetch response. Once the info is stale in the follower
>> side, it
>> >> >> >> updates
>> >> >> >> >> it
>> >> >> >> >> > from fetch response, and send next fetch request with
>> correct
>> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is
>> possible
>> >> >> like
>> >> >> >> >> > KAFKA-18723.
>> >> >> >> >> >
>> >> >> >> >> > I tried not to add new field in Fetch API, but this is the
>> best
>> >> >> >> >> solution I
>> >> >> >> >> > can think of.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR48. "The NumPartitions field is needed so the controller
>> >> can
>> >> >> >> create
>> >> >> >> >> the
>> >> >> >> >> > destination topic if it does not already exist. When a
>> topic is
>> >> >> being
>> >> >> >> >> > mirrored for the first time, the destination cluster has no
>> >> >> knowledge
>> >> >> >> >> > of the source topic's partition count, so the request must
>> >> carry
>> >> >> it."
>> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is started, the
>> >> source
>> >> >> >> topic
>> >> >> >> >> > may not exist yet. When the source topic is created later,
>> we
>> >> can
>> >> >> >> start
>> >> >> >> >> > mirroring without requiring the user to provide the number
>> of
>> >> >> >> >> partitions.
>> >> >> >> >> >
>> >> >> >> >> > > When the source topic is created later, we can start
>> >> mirroring
>> >> >> >> without
>> >> >> >> >> > requiring the user to provide the number of partitions.
>> >> >> >> >> >
>> >> >> >> >> > This is because this happens in periodical sync up in the
>> >> >> destination
>> >> >> >> >> > cluster, and the numPartitions and other topic info in the
>> >> source
>> >> >> >> >> cluster
>> >> >> >> >> > is already retrieved during the sync up process. Also, if
>> users
>> >> >> are
>> >> >> >> >> > creating mirror topics via admin API, this info can also be
>> >> >> retrieved
>> >> >> >> >> > inside the admin client if not provided.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly need the
>> last
>> >> >> mirror
>> >> >> >> >> epoch
>> >> >> >> >> > for truncation when starting a mirror. When a mirror is
>> >> stopped at
>> >> >> >> the
>> >> >> >> >> > source, we already bump up the leader epoch and write some
>> >> >> additional
>> >> >> >> >> > records for cleanup. It seems that we can just use the
>> >> >> DivergingEpoch
>> >> >> >> >> > returned in the initial mirror Fetch request to detect the
>> >> >> diverging
>> >> >> >> >> epoch
>> >> >> >> >> > and use that for truncation at the mirror destination.
>> >> >> >> >> >
>> >> >> >> >> > That’s a good question. I thought about this before but
>> >> >> >> unfortunately it
>> >> >> >> >> > doesn’t work. The reason we need the last mirror epoch
>> record
>> >> is
>> >> >> >> because
>> >> >> >> >> > the destination cluster might not completely sync with the
>> >> source
>> >> >> >> >> cluster
>> >> >> >> >> > before STOPPED. And this un-mirrored data might confuse the
>> >> >> >> replication
>> >> >> >> >> > protocol when reverse mirror. For example,
>> >> >> >> >> >
>> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
>> >> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> >> >> >> > Offset 2: from A (LE: 10)
>> >> >> >> >> >
>> >> >> >> >> > While cluster B mirrors until offset 1, the cluster A
>> crashes
>> >> and
>> >> >> >> >> failover
>> >> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
>> >> >> MIRROR_PID_RESET
>> >> >> >> >> with
>> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from
>> cluster B,
>> >> >> the
>> >> >> >> log
>> >> >> >> >> > will be like this:
>> >> >> >> >> >
>> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> >> > foo-0 leader log: foo-0 leader log:
>> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
>> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
>> >> >> >> >> >
>> >> >> >> >> > As you can see, cluster A (new destination cluster) doesn’t
>> >> know
>> >> >> the
>> >> >> >> >> > offset 2 is not belonging to the cluster B, which causes
>> >> >> inconsistent
>> >> >> >> >> logs.
>> >> >> >> >> > So like what we described in the KIP, the last mirror epoch
>> is
>> >> the
>> >> >> >> >> > synchronization point between source and destination. Any
>> data
>> >> >> beyond
>> >> >> >> >> the
>> >> >> >> >> > point is treated as unknown to the source cluster and
>> should be
>> >> >> >> >> truncated.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR52. Stop Mirror:
>> >> >> >> >> > > JR52.1 "3. If patterns are provided, the controller
>> removes
>> >> >> >> matching
>> >> >> >> >> > entries from mirror.topics.include or adds them to
>> >> >> >> >> mirror.topics.exclude on
>> >> >> >> >> > the CLUSTER_MIRROR resource."
>> >> >> >> >> > Why is it 'or' and not 'and'?
>> >> >> >> >> >
>> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> >> MirrorMetadataManager
>> >> >> >> >> > discovers new source topics matching the persisted
>> >> include/exclude
>> >> >> >> >> patterns
>> >> >> >> >> > and repeats steps 3 through 7 for each."
>> >> >> >> >> > The broker will send a create topic request to the
>> controller.
>> >> How
>> >> >> >> does
>> >> >> >> >> the
>> >> >> >> >> > controller associate it with the mirror name to create a
>> >> >> >> >> > `MirrorTopicStateChangeRecord`?
>> >> >> >> >> >
>> >> >> >> >> > No, we don’t send a create topic request to the controller
>> >> here.
>> >> >> >> >> Actually
>> >> >> >> >> > we send a StartMirrorTopics request to the controller, which
>> >> >> contains
>> >> >> >> >> the
>> >> >> >> >> > mirror name and topic metadata like NumPartitions, topicId…
>> >> etc.
>> >> >> So
>> >> >> >> the
>> >> >> >> >> > controller can create the non-existed topic as explained in
>> >> JR48,
>> >> >> and
>> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR53. Describe Mirrors:
>> >> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors request
>> >> with
>> >> >> >> >> optional
>> >> >> >> >> > mirror names (empty means all mirrors)."
>> >> >> >> >> > In metadata request, we use a null arrary to represent all
>> >> >> topics. It
>> >> >> >> >> would
>> >> >> >> >> > be useful to be consistent here.
>> >> >> >> >> >
>> >> >> >> >> > OK, KIP updated.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
>> >> brokers
>> >> >> and
>> >> >> >> >> > merges
>> >> >> >> >> > the responses client-side."
>> >> >> >> >> > This is not ideal. Not every broker hosts a mirror
>> partition.
>> >> So
>> >> >> >> sending
>> >> >> >> >> > the request to every broker is wasteful. It's probably
>> better
>> >> to
>> >> >> send
>> >> >> >> >> the
>> >> >> >> >> > request to a single broker, which can then contact the right
>> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >
>> >> >> >> >> > Agree, KIP updated.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are
>> >> MIRRORING,
>> >> >> >> >> PAUSED,
>> >> >> >> >> > STOPPED the only validate states? The state table has other
>> >> states
>> >> >> >> too.
>> >> >> >> >> > Ditto for MirrorPartitionState.
>> >> >> >> >> > The DesiredState field in MirrorTopicStateChangeRecord is to
>> >> >> define
>> >> >> >> the
>> >> >> >> >> > target state the user wants. And the desired state users can
>> >> >> assign
>> >> >> >> is
>> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired
>> state
>> >> as
>> >> >> >> >> STOPPING
>> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
>> intermediate
>> >> >> >> states.
>> >> >> >> >> >
>> >> >> >> >> > About MirrorPartitionState record, it’ll record any state in
>> >> the
>> >> >> >> state
>> >> >> >> >> > table that the current partition is. This is the real state
>> for
>> >> >> each
>> >> >> >> >> > partition. It’s like in the kubernetes world, when we change
>> >> the
>> >> >> >> replica
>> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired state), it
>> >> won’t
>> >> >> >> >> increase
>> >> >> >> >> > to 5 immediately. The current replica (MirrorPartitionState
>> >> >> record)
>> >> >> >> is
>> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Thank you,
>> >> >> >> >> > Luke
>> >> >> >> >> >
>> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
>> >> >> dev@kafka.apache.org
>> >> >> >> >
>> >> >> >> >> > wrote:
>> >> >> >> >> >
>> >> >> >> >> >> Hi, Fede,
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks for the reply. A few more comments.
>> >> >> >> >> >>
>> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror partition cache
>> >> >> lives in
>> >> >> >> >> >> MirrorMetadataManager. It is populated in two ways: by
>> >> replaying
>> >> >> >> >> >> __mirror_state partitions this broker leads, and by
>> fetching
>> >> >> state
>> >> >> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> >> >> Hmm, a given broker only caches the mirror partition state
>> for
>> >> >> >> >> partitions
>> >> >> >> >> >> hashed into the __mirror_state partition hosted on this
>> >> broker,
>> >> >> >> right?
>> >> >> >> >> Why
>> >> >> >> >> >> does it need to fetch the state from other remote
>> >> coordinators?
>> >> >> >> >> >>
>> >> >> >> >> >> JR36. I am still not quite sure why we need
>> MirrorLeaderEpoch
>> >> in
>> >> >> the
>> >> >> >> >> Fetch
>> >> >> >> >> >> request. In the current Fetch request, in addition to
>> >> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> >> it has another field LastFetchedEpoch, which captures the
>> last
>> >> >> >> leader
>> >> >> >> >> >> epoch
>> >> >> >> >> >> in the data and is used to resolve the inconsistency
>> between
>> >> the
>> >> >> >> >> follower
>> >> >> >> >> >> and the leader. Does it achieve the same purpose as
>> >> >> >> MirrorLeaderEpoch?
>> >> >> >> >> >>
>> >> >> >> >> >> JR48. "The NumPartitions field is needed so the controller
>> can
>> >> >> >> create
>> >> >> >> >> the
>> >> >> >> >> >> destination topic if it does not already exist. When a
>> topic
>> >> is
>> >> >> >> being
>> >> >> >> >> >> mirrored for the first time, the destination cluster has no
>> >> >> >> knowledge
>> >> >> >> >> >> of the source topic's partition count, so the request must
>> >> carry
>> >> >> >> it."
>> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is started, the
>> >> >> source
>> >> >> >> >> topic
>> >> >> >> >> >> may not exist yet. When the source topic is created later,
>> we
>> >> can
>> >> >> >> start
>> >> >> >> >> >> mirroring without requiring the user to provide the number
>> of
>> >> >> >> >> partitions.
>> >> >> >> >> >>
>> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need the
>> last
>> >> >> mirror
>> >> >> >> >> epoch
>> >> >> >> >> >> for truncation when starting a mirror. When a mirror is
>> >> stopped
>> >> >> at
>> >> >> >> the
>> >> >> >> >> >> source, we already bump up the leader epoch and write some
>> >> >> >> additional
>> >> >> >> >> >> records for cleanup. It seems that we can just use the
>> >> >> >> DivergingEpoch
>> >> >> >> >> >> returned in the initial mirror Fetch request to detect the
>> >> >> diverging
>> >> >> >> >> epoch
>> >> >> >> >> >> and use that for truncation at the mirror destination.
>> >> >> >> >> >>
>> >> >> >> >> >> JR52. Stop Mirror:
>> >> >> >> >> >> JR52.1 "3. If patterns are provided, the controller removes
>> >> >> matching
>> >> >> >> >> >> entries from mirror.topics.include or adds them to
>> >> >> >> >> mirror.topics.exclude
>> >> >> >> >> >> on
>> >> >> >> >> >> the CLUSTER_MIRROR resource."
>> >> >> >> >> >> Why is it 'or' and not 'and'?
>> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> MirrorMetadataManager
>> >> >> >> >> >> discovers new source topics matching the persisted
>> >> >> include/exclude
>> >> >> >> >> >> patterns
>> >> >> >> >> >> and repeats steps 3 through 7 for each."
>> >> >> >> >> >> The broker will send a create topic request to the
>> controller.
>> >> >> How
>> >> >> >> does
>> >> >> >> >> >> the
>> >> >> >> >> >> controller associate it with the mirror name to create a
>> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
>> >> >> >> >> >>
>> >> >> >> >> >> JR53. Describe Mirrors:
>> >> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors request
>> >> with
>> >> >> >> >> optional
>> >> >> >> >> >> mirror names (empty means all mirrors)."
>> >> >> >> >> >> In metadata request, we use a null arrary to represent all
>> >> >> topics.
>> >> >> >> It
>> >> >> >> >> >> would
>> >> >> >> >> >> be useful to be consistent here.
>> >> >> >> >> >> JR53.2 "2. The admin client fans out the request to all
>> >> brokers
>> >> >> and
>> >> >> >> >> merges
>> >> >> >> >> >> the responses client-side."
>> >> >> >> >> >> This is not ideal. Not every broker hosts a mirror
>> partition.
>> >> So
>> >> >> >> >> sending
>> >> >> >> >> >> the request to every broker is wasteful. It's probably
>> better
>> >> to
>> >> >> >> send
>> >> >> >> >> the
>> >> >> >> >> >> request to a single broker, which can then contact the
>> right
>> >> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >> >>
>> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are
>> >> MIRRORING,
>> >> >> >> PAUSED,
>> >> >> >> >> >> STOPPED the only validate states? The state table has other
>> >> >> states
>> >> >> >> too.
>> >> >> >> >> >> Ditto for MirrorPartitionState.
>> >> >> >> >> >>
>> >> >> >> >> >> Jun
>> >> >> >> >> >>
>> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
>> >> >> >> fedevaleri@gmail.com>
>> >> >> >> >> >> wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> > Hi Andrew, yes, validation is called at the start of
>> >> >> createTopic.
>> >> >> >> If
>> >> >> >> >> >> > the topic ID is invalid, the request fails with
>> >> >> INVALID_REQUEST.
>> >> >> >> If
>> >> >> >> >> >> > the topic ID is already used by a different topic name,
>> it
>> >> >> fails
>> >> >> >> with
>> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual
>> >> topic
>> >> >> in
>> >> >> >> the
>> >> >> >> >> >> > batch fails; other topics in the same CreateTopics
>> request
>> >> >> >> continue
>> >> >> >> >> >> > processing normally. Added this information to the KIP.
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> >
>> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
>> >> >> >> >> >> schofielaj@apache.org>
>> >> >> >> >> >> > wrote:
>> >> >> >> >> >> > >
>> >> >> >> >> >> > > Hi Fede,
>> >> >> >> >> >> > > One more small question.
>> >> >> >> >> >> > >
>> >> >> >> >> >> > > AS37: If the controller fails validation for the
>> >> >> >> MirrorInfo.TopicId
>> >> >> >> >> >> > added to the CreateTopic request, what does it do? I
>> suspect
>> >> >> that
>> >> >> >> it
>> >> >> >> >> >> fails
>> >> >> >> >> >> > the request with a particular error code.
>> >> >> >> >> >> > >
>> >> >> >> >> >> > > Thanks,
>> >> >> >> >> >> > > Andrew
>> >> >> >> >> >> > >
>> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> >> >> >> >> >> > > > Hi all,
>> >> >> >> >> >> > > >
>> >> >> >> >> >> > > > We updated the KIP to include Coordinator Runtime
>> >> >> >> configurations
>> >> >> >> >> and
>> >> >> >> >> >> > > > the coordinator state transition validation
>> paragraph.
>> >> The
>> >> >> >> latter
>> >> >> >> >> >> > > > covers the per-partition epoch fencing mechanism
>> (leader
>> >> >> epoch
>> >> >> >> >> and
>> >> >> >> >> >> > > > state epoch) that complements the existing per-topic
>> >> >> >> StateOffset
>> >> >> >> >> >> > > > fencing for lifecycle operations.
>> >> >> >> >> >> > > >
>> >> >> >> >> >> > > > Thanks,
>> >> >> >> >> >> > > > Fede
>> >> >> >> >> >> > > >
>> >> >> >> >> >> >
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for the good comments.
Answering them below.
> JR59.1 Could you add the RPC to the workflow? The broker first needs to
issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
mirror name and then issue DescribeClusterMirrors to get the LME.
Yes, you’re right. The workflow is updated to make them clear.
> JR71. mirror.support.unclean.leader.election : The name is confusing. It's
also used in LOG_TRUNCATION, which is not related to unclean leader
election. Could we use a better name?
I personally think the name should be fine because even though we use in
“LOG_TRUNCATION” state, the config is to allow us to know if users want to
support unclean leader election, if so, we wait for all replicas to catch
up. If we want to explicitly show the intention of the config, how about
“mirror.full.replica.convergence.required”? I’m open to other suggestions.
> JR72. DescribeClusterMirrors
> JR72.1 A user may specify just the topic name when describing a mirror.
Should we include the topic name in the request?
Hmm.. we didn’t think about the use case. But yes, if users only care about
the state of a topic, they don’t need the info for other topics. I’ve
combined with JR72.2, added “TopicPartitions” filed into
DescribeClusterMirrorsRequest, and by default is null, which means all
topic partitions under the requested mirrors will be returned. If set,
we’ll query the mirrors and return the topic partitions that users
expected. KIP is updated.
> JR72.2 LastMirrorEpochLookups is a bit weird since the response returns
either mirror state or mirror lag and LastMirrorEpochLookups doesn't match
either.
Yes, after the recent refactor, I think the LastMirrorEpochLookups can be
merged into one level up, combining with the topic name filter in above
comment, which will allow users to query down to the partition level. The
LastMirrorEpochLookups will query along with normal mirror state by the
readMirrorState from the coordinator. The difference is that after
receiving the response, it will filter out the mirrors that don’t match the
requested sourceClusterId, and then add the lastMirrorEpoch into
DescribeClusterMirrorResponse. KIP is updated.
> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
Right. Added.
> JR72.4 "includeMirrorLags: source offset, destination offset, lag"
DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
should be omitted.
Sorry, I removed the “SourceOffset” and “DestinationOffset” fields from
DescribeClusterMirrorsResponse because I added them into ReadMirrorOffsets
(ex-ReadMirrorLags) API. But I forgot we still need them in the
DescribeClusterMirrorsResponse so that the adminClient can receive the
results. KIP is updated.
> JR73. MirrorPartitionState record, LastMirrorEpochs record and
MirrorPartitionStateRequest: The values for type and name are incorrect.
Ah, that’s my bad. Fixed.
> JR74. ListClusterMirrorsRequest
> JR74.1 When is DesiredMirrorStateFilter used? There is no option to
specify
the desired state in the mirror-topic command.
Yes, referred to kafka-consumer-groups.sh, I added the `--state` option to
allow users to set the state when listMirror.
> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
Yes, KIP updated.
> JR75. ReadMirrorLags
> JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
include lag?
Indeed. Updated.
> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
Yes, added
> JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to be
consistent.
Thanks for pointing it out. KIP is updated.
> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
Yes, we should validate the mirror name before writing the data. KIP is
updated.
> JR77. Delete Mirror: "3. The controller verifies that no topics are in
MIRRORING or PAUSED state."
Is this necessary since step 2 has verified all partitions are in STOPPED
state, which can only happen if the topics are in STOPPED state.
It’s true, but there could be some delay between “actual state” and
“desired state”. In step (2), we verify all partitions’ “actual state” are
in STOPPED state. But the current desired state could be MIRRORING, it’s
just that the mirror partition leaders haven’t got the metadata update yet.
So, the controller still needs to verify the desired state is not in
MIRRORING or PAUSED. I found we didn’t specify that the controller is
actually verifying the “desired state”, not “actual state”. And this
controller’s desired state validation also applies in
startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics.
I’ve updated the KIP to make it clear.
> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is
bumped,
ABORT markers are appended for all ongoing transactions, last mirror epochs
are persisted, and a MIRROR_PID_RESET record appended."
This describes the state for STOPPED, not STOPPING.
No, these are the tasks during STOPPING state, like what we described in
“Stop Mirror Topics” workflow. If we do these in STOPPED state, we can’t
safely move the readonly topic into writable because not all required tasks
are done. I think I know why it confuses you, it’s because all the tasks
are in the completed state. I’ve rephrased it to make it clear.
> JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
timeout field?
Make sense. Added into `DescribeClusterMirrors` request, as well as
startMirrorTopics/stopMirrorTopics/pauseMirrorTopics/resumeMirrorTopics/deleteClusterMirrors
because we will query the partition states from coordinator for state
validation in these APIs.
> JR80. PauseMirrorTopicsOptions: Should it support patterns? The command
line seems to suppor it.
Sorry, it’s a little confusing here. The “pattern” in
startMirrorTopicsOptions or stopMirrorTopicsOptions is to update the
“mirror.topics.include” and “mirror.topics.exclude” configuration for topic
pattern discovery periodically. For the PauseMirrorTopics and
ResumeMirrorTopics in the clusterMirrorCommand, users can provide regex in
the “--topics” option, and the clusterMirrorCommand will listTopic in the
destination cluster and match with the patterns users provided, and then
feeds the matched topics into the PauseMirrorTopics/ResumeMirrorTopics
requests. So in short, yes, PauseMirrorTopics/ResumeMirrorTopics supports
regex in the ClusterMirrorComamnd, but we don’t need to add the “patterns”.
> JR81. AdminClient
> JR81.1 ListClusterMirrors: We need to add the option to specify different
filters supported in the request.
We already have the `--mirror` option. Added `--state` and
`--sourceClusterId` options to support the other filters.
> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
request doesn't support this option. Could we add other options supported
in the request?
Currently, we have “includeAuthorizedOperations” in DescribeCluster,
DescribeConsumerGroups, DescribeTopics, DescribeShareGroups,
DescribeStreamsGroups requests. But none of them include the option in the
tool to allow users to enable it. I think the reason is that we don’t want
to show this to users on the tool output. But having this field, we allow
the advanced users to invoke admin API with includeAuthorizedOperations
enabled to get the results they want.
> JR81.3 LeaderStateDescription: Should it be named
MirrorPartitionStateDescription? Also, could we make the fields consistent
with those in the response?
OK. KIP is updated.
Thank you,
Luke
On Thu, Aug 13, 2026 at 8:04 AM Jun Rao via dev <de...@kafka.apache.org>
wrote:
> Hi, Luke,
>
> Thanks for the reply.
>
> JR59.1 Could you add the RPC to the workflow? The broker first needs to
> issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
> mirror name and then issue DescribeClusterMirrors to get the LME.
>
> JR71. mirror.support.unclean.leader.election : The name is confusing. It's
> also used in LOG_TRUNCATION, which is not related to unclean leader
> election. Could we use a better name?
>
> JR72. DescribeClusterMirrors
> JR72.1 A user may specify just the topic name when describing a mirror.
> Should we include the topic name in the request?
> JR72.2 LastMirrorEpochLookups is a bit weird since the response returns
> either mirror state or mirror lag and LastMirrorEpochLookups doesn't match
> either.
> JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
> JR72.4 "includeMirrorLags: source offset, destination offset, lag"
> DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
> should be omitted.
>
> JR73. MirrorPartitionState record, LastMirrorEpochs record and
> MirrorPartitionStateRequest: The values for type and name are incorrect.
>
> JR74. ListClusterMirrorsRequest
> JR74.1 When is DesiredMirrorStateFilter used? There is no option to specify
> the desired state in the mirror-topic command.
> JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
>
> JR75. ReadMirrorLags
> JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
> include lag?
> JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
> JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to be
> consistent.
>
> JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
>
> JR77. Delete Mirror: "3. The controller verifies that no topics are in
> MIRRORING or PAUSED state."
> Is this necessary since step 2 has verified all partitions are in STOPPED
> state, which can only happen if the topics are in STOPPED state.
>
> JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is bumped,
> ABORT markers are appended for all ongoing transactions, last mirror epochs
> are persisted, and a MIRROR_PID_RESET record appended."
> This describes the state for STOPPED, not STOPPING.
>
> JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
> timeout field?
>
> JR80. PauseMirrorTopicsOptions: Should it support patterns? The command
> line seems to suppor it.
>
> JR81. AdminClient
> JR81.1 ListClusterMirrors: We need to add the option to specify different
> filters supported in the request.
> JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
> request doesn't support this option. Could we add other options supported
> in the request?
> JR81.3 LeaderStateDescription: Should it be named
> MirrorPartitionStateDescription? Also, could we make the fields consistent
> with those in the response?
>
> Jun
>
> On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the good comments again.
> > Answering them below.
> >
> > > JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election
> is
> > enabled and the destination leader has log truncation, the state will
> move
> > from MIRRORING to this state."
> > This still suggests that divergeEpoch is only handled if
> > mirror.support.unclean.leader.election
> > is enabled.
> >
> >
> > OK, KIP is updated to make it clear.
> >
> > > JR55. Should the mirror state and the mirror lags always be exposed
> > together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
> > from DescribeClusterMirrorsRequest. If not, it's better to use two
> separate
> > RPCs.
> >
> >
> > No, we sometimes only need MirrorState, and sometimes need both. So I’ll
> > keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
> > DescribeClusterMirrorsRequest. And we will use ReadMirrorStates and
> > ReadMirrorLags to get each info.
> >
> >
> > > DescribeClusterMirrorsRequest should be used in the Describe Mirror
> flow,
> > right? Currently it's missing. Also, I assume the admin client refers to
> > the one in the kafka-cluster-mirrors tool. But it seems that sometimes it
> > also refers to the admin client on the broker? Perhaps we can use 'the
> > tool' instead to make it clear.
> >
> > Yes, I missed the DescribeClusterMirrorsRequest. Added.
> > And sorry for the confusion. I’d like to keep the admin client as the one
> > invoking from the tool (or user’s direct admin client API invocation). On
> > the broker side, we actually use InterBrokerSendThread to invoke these
> RPCs
> > or get directly from MetadataImage. I’ve updated the KIP to make it
> clear.
> >
> >
> > > JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
> > instead of just to the controller?
> >
> > Yes, we only need to issue to the controller. Updated to the
> > BumpLeaderEpochs API.
> >
> >
> > > JR59. Start Mirror Topics:
> > > JR59.1 "During this state, the broker queries the source cluster for
> last
> > mirror epochs".
> > Could you describe which RPC is issued to find the LME?
> >
> > We’re using describeClusterMirror API via admin client here.
> >
> > > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
> > doesn't
> > contain a replication factor.
> >
> > We use “default.replication.factor” when topic creation, so the
> > INVALID_REPLICATION_FACTOR will be thrown like normal topic creation.
> Also,
> > because we require the replicas for the partition assignment to be all
> > active and not in fenced or controlled shutdown(JR35), we’ll throw
> > INVALID_REPLICATION_FACTOR if some replicas are fenced.
> >
> >
> > > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
> > different topic name"
> > Should we also return this error when the topic name is used as a regular
> > topic?
> >
> > There are 2 errors we will throw when topic id <-> topic name mismatch.
> > Suppose provided topic id: ID_A, topic name: NAME_A
> > 1. Destination cluster exists topic id: ID_A, but topic name is: NAME_B
> > TOPIC_ALREADY_EXISTS will be thrown
> > 2. Destination cluster exists topic id: ID_B, but topic name is: NAME_A
> > INCONSISTENT_TOPIC_ID will be thrown
> >
> > Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS error for
> > these 2 cases makes more sense. KIP updated.
> >
> >
> > > JR60. CreateClusterMirrorRequest: When will it receive an
> > INVALID_CLUSTER_MIRROR_STATES error not covered by
> > CLUSTER_MIRROR_ALREADY_EXISTS?
> >
> >
> > Sorry, createClusterMirrorRequest doesn’t throw
> > INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate the
> mirror
> > states when creating a cluster mirror. We only care if the “mirror name”
> > exists or not. KIP updated.
> >
> >
> > > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just
> remove
> > MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> > CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
> > specific state causing the error.
> >
> > I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following errors:
> > a. MIRROR_TOPIC_ALREADY_PAUSED
> > b. MIRROR_TOPIC_NOT_PAUSED,
> > c. MIRROR_TOPIC_ALREADY_STOPPED,
> > d. CLUSTER_MIRROR_NOT_EMPTY
> > e. MIRROR_TOPIC_NOT_STOPPED
> >
> > KIP is updated.
> >
> >
> > > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
> requests
> > (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics). Could
> we
> > make them consistent?
> >
> >
> > Good catch. The updated errors for the new added RPC are this:
> >
> > CreateClusterMirror: NOT_CONTROLLER
> > StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> > StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> > PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> > ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> > DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> >
> >
> > > JR63. StopMirrorTopics:
> > > JR63.1 "The broker validates that all target topic partitions are in
> > either
> > LOG_TRUNCATION or MIRRORING or FAILED state."
> > Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
> >
> > Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is updated.
> >
> > > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
> PAUSED
> > -> STOPPED is valid?
> >
> > Yes, it is not correct. KIP updated to remove this error.
> >
> > > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
> > state"
> > This is weird. The reason to issue StopMirrorTopics is that it's not in
> > stopped state.
> >
> > Indeed. KIP is updated to only include INVALID_CLUSTER_MIRROR_STATES
> > error, and will be thrown when states are not in MIRRORING or PAUSED
> state.
> >
> > > JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's derived
> > from SourceOffset and DestinationOffset.
> >
> > True. Removed.
> >
> > > JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
> > could just set the partition level state to STOPPED.
> >
> > Nice catch. We don’t use it at all. That is some old leftover field we
> > forgot to remove. KIP updated.
> >
> > > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent
> to
> > the controller?
> >
> > No, these 2 only send to the broker. KIP updated.
> >
> >
> > > JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch? It
> > seems both are used for fencing and fencing based on the latter is
> > stronger.
> >
> > Yes, basically the StateEpoch can fence for most cases. The LeaderEpoch
> > fencing is used for cases like:
> >
> > 1. b1 is the current leader for foo-0, preparing write state from
> > MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
> > 2. leadership change to b2 for foo-0, LE changes to 2.
> > 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1 }.
> > 4. The coordinator receives the read from b2 in step (3), the highest
> > lastSeen LE for foo-0 updates to 2 in coordinator.
> > 5. The writes state from b1 in step (1) arrives the coordinator, but it
> > rejects b1 writes because the highest lastSeen LE is 2, which is > 1 in
> the
> > request.
> > 6. b2 writes state from MIRRORING to PAUSING in LE: 2, stateEpoch: 1 and
> > succeeds.
> >
> > The leaderEpoch fence is to avoid any stale broker writes after the new
> > leader epoch read. This cannot be protected by the stateEpoch. The
> > stateEpoch can prevent the write with older stateEpoch. So in the
> situation
> > above, the stateEpoch cannot determine if we should accept b1 writes or
> > not, but leaderEpoch can. By the way, we borrowed this concept from
> > KIP-932:
> >
> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
> > <
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$
> >
> >
> >
> > > JR68. Could we make the usage of topicId and topicName in the RPCs
> > consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> > BumpLeaderEpochsReponse uses TopicName.
> >
> > Yes, we have made:
> > 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use TopicName.
> > 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse use
> > TopicName.
> > 3. MirrorTopicStateChangeRecord uses TopicName because we will do
> > validation before writing the metadata log.
> >
> >
> > > JR69. MIRROR_PID_RESET:
> > > JR69.1 Could you specify the value of the control record's type?
> >
> > It’s MIRROR_PID_RESET(7). KIP updated.
> >
> > > JR69.2 How is the SourceClusterId field used?
> >
> > I’m also surprised to see this field in the MirrorPidResetRecord. I
> traced
> > back to the internal PR we added for this field, and confirmed it is not
> > used since it was added. Sorry for the confusion and nice catch! KIP
> > updated.
> >
> > > JR70. MirrorPartitionState: type should be "data" not
> "coordinator-key",
> > right? Ditto for LastMirrorEpochs.
> > We store both MirrorPartitionState and LastMirrorEpochs records into
> > internal __mirror_state topic, and rely on Coordinator Runtime to
> > write/read the records, like how `OffsetCommit` and `GroupMetadata` work.
> > So I think we should keep them as "coordinator-key"/"coordinator-value"
> > type.
> >
> >
> > Thank you,
> > Luke
> >
> > On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Luke,
> >>
> >> Thanks for the reply.
> >>
> >> JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election is
> >> enabled and the destination leader has log truncation, the state will
> move
> >> from MIRRORING to this state."
> >> This still suggests that divergeEpoch is only handled if
> >> mirror.support.unclean.leader.election
> >> is enabled.
> >>
> >> JR55. Should the mirror state and the mirror lags always be exposed
> >> together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
> >> from DescribeClusterMirrorsRequest. If not, it's better to use two
> >> separate
> >> RPCs.
> >>
> >> DescribeClusterMirrorsRequest should be used in the Describe Mirror
> flow,
> >> right? Currently it's missing. Also, I assume the admin client refers to
> >> the one in the kafka-cluster-mirrors tool. But it seems that sometimes
> it
> >> also refers to the admin client on the broker? Perhaps we can use 'the
> >> tool' instead to make it clear.
> >>
> >> JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
> >> instead of just to the controller?
> >>
> >> JR59. Start Mirror Topics:
> >> JR59.1 "During this state, the broker queries the source cluster for
> last
> >> mirror epochs".
> >> Could you describe which RPC is issued to find the LME?
> >> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
> >> doesn't
> >> contain a replication factor.
> >> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
> >> different topic name"
> >> Should we also return this error when the topic name is used as a
> regular
> >> topic?
> >>
> >> JR60. CreateClusterMirrorRequest: When will it receive an
> >> INVALID_CLUSTER_MIRROR_STATES error not covered by
> >> CLUSTER_MIRROR_ALREADY_EXISTS?
> >>
> >> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just remove
> >> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> >> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
> >> specific state causing the error.
> >>
> >> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some
> requests
> >> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics). Could
> we
> >> make them consistent?
> >>
> >> JR63. StopMirrorTopics:
> >> JR63.1 "The broker validates that all target topic partitions are in
> >> either
> >> LOG_TRUNCATION or MIRRORING or FAILED state."
> >> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
> >> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since
> PAUSED
> >> -> STOPPED is valid?
> >> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
> >> state"
> >> This is weird. The reason to issue StopMirrorTopics is that it's not in
> >> stopped state.
> >>
> >> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's derived
> >> from SourceOffset and DestinationOffset.
> >>
> >> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
> >> could just set the partition level state to STOPPED.
> >>
> >> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent to
> >> the controller?
> >>
> >> JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch? It
> >> seems both are used for fencing and fencing based on the latter is
> >> stronger.
> >>
> >> JR68. Could we make the usage of topicId and topicName in the RPCs
> >> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> >> BumpLeaderEpochsReponse uses TopicName.
> >>
> >> JR69. MIRROR_PID_RESET:
> >> JR69.1 Could you specify the value of the control record's type?
> >> JR69.2 How is the SourceClusterId field used?
> >>
> >> JR70. MirrorPartitionState: type should be "data" not "coordinator-key",
> >> right? Ditto for LastMirrorEpochs.
> >>
> >> Jun
> >>
> >> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thanks for the comments.
> >> > Answering them below.
> >> >
> >> > > JR36. Sounds good. Should we remove
> >> MirrorLeaderEpochExceededException?
> >> >
> >> > Sorry, it’s a confusing name. The MirrorLeaderEpochExceededException
> is
> >> > used and thrown when the fetched mirror batch leader epoch > local
> >> leader
> >> > epoch in destination cluster. And then we’ll stop mirroring and move
> the
> >> > partition to EPOCH_FENSING state to wait for local leader epoch bump.
> I
> >> > think since we don’t have the concept of `MirrorLeaderEpoch` anymore,
> >> the
> >> > `MirrorLeaderEpochExceededException` should not cause confusion now.
> >> I’ll
> >> > keep the name for the exception.
> >> >
> >> >
> >> > > JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
> >> > mirror
> >> > fetch request issued to the source cluster, independent of the value
> of
> >> > mirror.support.unclean.leader.election? It's needed to detect
> divergence
> >> > during failover. However, if it's also set when
> >> > mirror.support.unclean.leader.election=false,
> >> > an unclean leader election in the source cluster can halt the
> mirroring
> >> > since we don’t do anything with the returned divergeEpoch in that mode
> >> > and continue
> >> > the next fetch.
> >> >
> >> > > Do we plan to always set lastFetchedEpoch in the mirror
> >> > fetch request issued to the source cluster, independent of the value
> of
> >> > Mirror.support.unclean.leader.election?
> >> >
> >> > Yes, we always set lastFetchedEpoch in the mirror fetch request.
> >> >
> >> > > However, if it's also set when
> >> > mirror.support.unclean.leader.election=false,
> >> > an unclean leader election in the source cluster can halt the
> mirroring
> >> > since we don’t do anything with the returned divergeEpoch in that mode
> >> > and continue the next fetch.
> >> >
> >> > Sorry I didn’t make it clear. When
> >> > “mirror.support.unclean.leader.election=false” and a “divergeEpoch” is
> >> set
> >> > in fetch response, the leader node will handle the divergeEpoch and
> then
> >> > continue the next fetch as usual. The difference with
> >> > “mirror.support.unclean.leader.election=true” is that “should we wait
> >> for
> >> > ALL replicas caught up (complete truncation)”. Without the waiting,
> it’s
> >> > possible the inconsistent log issue in KAFKA-18723 could happen
> between
> >> > leader and follower nodes in the destination cluster because we don’t
> >> have
> >> > “MirrorLeaderEpoch” concept now to reject higher leader epoch batches.
> >> That
> >> > is, we won't halt the mirroring if "divergeEpoch" is set but
> >> > “mirror.support.unclean.leader.election=false”.
> >> >
> >> >
> >> > > JR55. The DescribeClusterMirrorsRequest still returns both the
> mirror
> >> > partition state and the mirror partition leader state. The problem is
> >> that
> >> > if a caller only needs one of the state, the other state becomes
> >> redundant.
> >> > For example, when starting the mirror, to obtain LME, we only need the
> >> > mirror partition state. Here is an alternative design: We expose the
> >> mirror
> >> > partition state through ReadMirrorState. The client will use the
> >> > FindCoordinatorRequest
> >> > to find the right mirror partition coordinator to issue that request.
> We
> >> > expose the mirror partition leader state through the
> >> ReadClusterMirrorLags
> >> > request. The client can use the metadata request to find the partition
> >> > leader to issue that request.
> >> >
> >> > > In the describing mirror flow, the admin client first issues the
> >> > ListMirrorRequest to find all the mirrors and the topics being
> >> mirrored. It
> >> > then issues the metadata request to find the partitions for those
> topics
> >> > and issues a FindCoordinatorRequest to find the mirror partition state
> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
> partition
> >> > state coordinator and ReadClusterMirrorLags to the partition leader.
> >> >
> >> > > In the starting mirror flow, the partition leader in the destination
> >> > cluster issues a ListMirrorRequest to find the mirror name based on
> its
> >> > cluster id. It then issues a FindCoordinatorRequest to find the mirror
> >> > partition state coordinators. Finally, it issues ReadMirrorState to
> the
> >> > mirror partition state coordinator to get the LME.
> >> >
> >> > > We can also improve the ListMirror request to accept a filter based
> on
> >> > mirror name, source cluster ID and topic name.
> >> >
> >> >
> >> > Answers:
> >> >
> >> > > The DescribeClusterMirrorsRequest still returns both the mirror
> >> > partition state and the mirror partition leader state. The problem is
> >> that
> >> > if a caller only needs one of the state, the other state becomes
> >> redundant.
> >> > For example, when starting the mirror, to obtain LME, we only need the
> >> > mirror partition state.
> >> >
> >> > Good point!
> >> >
> >> >
> >> > > Here is an alternative design: We expose the mirror
> >> > partition state through ReadMirrorState. The client will use the
> >> > FindCoordinatorRequest
> >> > to find the right mirror partition coordinator to issue that request.
> We
> >> > expose the mirror partition leader state through the
> >> ReadClusterMirrorLags
> >> > request. The client can use the metadata request to find the partition
> >> > leader to issue that request.
> >> >
> >> >
> >> > Sounds good. But we try not to expose
> >> > ReadMirrorState/ReadClusterMirrorLags directly to the admin client
> >> because
> >> > users might get confused about these 2 requests, and they are also
> both
> >> > related to describeClusterMirror. I think we can add 2 additional
> >> fields to
> >> > describeClusterMirrorRequest: “includeMirrorStates” and
> >> > “includeMirrorLags”. So the admin client can include either one or
> both
> >> of
> >> > them in the DescrbieClusterMirrorRequest based on the provided
> options.
> >> KIP
> >> > is updated.
> >> >
> >> >
> >> > > In the describing mirror flow, the admin client first issues the
> >> > ListMirrorRequest to find all the mirrors and the topics being
> >> mirrored. It
> >> > then issues the metadata request to find the partitions for those
> topics
> >> > and issues a FindCoordinatorRequest to find the mirror partition state
> >> > coordinators. Finally, it issues ReadMirrorState to the mirror
> partition
> >> > state coordinator and ReadClusterMirrorLags to the partition leader.
> >> >
> >> > Makes sense. KIP is updated.
> >> >
> >> >
> >> > > In the starting mirror flow, the partition leader in the destination
> >> > cluster issues a ListMirrorRequest to find the mirror name based on
> its
> >> > cluster id. It then issues a FindCoordinatorRequest to find the mirror
> >> > partition state coordinators. Finally, it issues ReadMirrorState to
> the
> >> > mirror partition state coordinator to get the LME.
> >> >
> >> > Make sense. So we can send DescribeClusterMirrorRequest with
> >> > “includeMirrorStates” only.
> >> >
> >> > > We can also improve the ListMirror request to accept a filter based
> on
> >> > mirror name, source cluster ID and topic name.
> >> >
> >> > Good suggestion, just like what ListGroupsRequest does.
> >> > Added "SourceClusterIdFilter", "MirrorNameFilter",
> >> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP
> >> updated.
> >> >
> >> >
> >> > > JR56. Could you list the ACL needed for each new RPC?
> >> >
> >> >
> >> > Yes, it’s already listed in “Security Control” section.
> >> >
> >> >
> >> > > JR57. In the command workflow, it sometimes says "The user sends"
> and
> >> > some
> >> > other times says "The admin client sends". It's better to use "The
> admin
> >> > client" consistently in all those places.
> >> >
> >> > Agree. KIP updated.
> >> >
> >> >
> >> > Thank you,
> >> > Luke
> >> >
> >> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <de...@kafka.apache.org>
> >> > wrote:
> >> >
> >> >> Hi, Luke,
> >> >>
> >> >> Thanks for the reply.
> >> >>
> >> >> JR36. Sounds good. Should we remove
> MirrorLeaderEpochExceededException?
> >> >>
> >> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
> >> mirror
> >> >> fetch request issued to the source cluster, independent of the value
> of
> >> >> mirror.support.unclean.leader.election? It's needed to detect
> >> divergence
> >> >> during failover. However, if it's also set when
> >> >> mirror.support.unclean.leader.election=false,
> >> >> an unclean leader election in the source cluster can halt the
> mirroring
> >> >> since we don’t do anything with the returned divergeEpoch in that
> mode
> >> >> and continue
> >> >> the next fetch.
> >> >>
> >> >> JR55. The DescribeClusterMirrorsRequest still returns both the mirror
> >> >> partition state and the mirror partition leader state. The problem is
> >> that
> >> >> if a caller only needs one of the state, the other state becomes
> >> >> redundant.
> >> >> For example, when starting the mirror, to obtain LME, we only need
> the
> >> >> mirror partition state. Here is an alternative design: We expose the
> >> >> mirror
> >> >> partition state through ReadMirrorState. The client will use the
> >> >> FindCoordinatorRequest
> >> >> to find the right mirror partition coordinator to issue that request.
> >> We
> >> >> expose the mirror partition leader state through the
> >> ReadClusterMirrorLags
> >> >> request. The client can use the metadata request to find the
> partition
> >> >> leader to issue that request.
> >> >>
> >> >> In the describing mirror flow, the admin client first issues the
> >> >> ListMirrorRequest to find all the mirrors and the topics being
> >> mirrored.
> >> >> It
> >> >> then issues the metadata request to find the partitions for those
> >> topics
> >> >> and issues a FindCoordinatorRequest to find the mirror partition
> state
> >> >> coordinators. Finally, it issues ReadMirrorState to the mirror
> >> partition
> >> >> state coordinator and ReadClusterMirrorLags to the partition leader.
> >> >>
> >> >> In the starting mirror flow, the partition leader in the destination
> >> >> cluster issues a ListMirrorRequest to find the mirror name based on
> its
> >> >> cluster id. It then issues a FindCoordinatorRequest to find the
> mirror
> >> >> partition state coordinators. Finally, it issues ReadMirrorState to
> the
> >> >> mirror partition state coordinator to get the LME.
> >> >>
> >> >> We can also improve the ListMirror request to accept a filter based
> on
> >> >> mirror name, source cluster ID and topic name.
> >> >>
> >> >> JR56. Could you list the ACL needed for each new RPC?
> >> >>
> >> >> JR57. In the command workflow, it sometimes says "The user sends" and
> >> some
> >> >> other times says "The admin client sends". It's better to use "The
> >> admin
> >> >> client" consistently in all those places.
> >> >>
> >> >> Jun
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com> wrote:
> >> >>
> >> >> > Hi Jun,
> >> >> >
> >> >> > Thanks for your comments.
> >> >> > Answering them below.
> >> >> >
> >> >> >
> >> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> >> specifically
> >> >> > designed for the unclean leader election support in Cluster
> Mirroring
> >> >> > because the unclean leader election will break the assumption that
> >> the
> >> >> > committed data won’t be truncated."
> >> >> > If the source cluster has an unclean leader election, the
> destination
> >> >> > cluster will detect the inconsistency and truncate the log at the
> >> >> > destination leader. We can just
> >> >> > extend mirror.support.unclean.leader.election to cover this case,
> >> i.e.,
> >> >> we
> >> >> > wait until all replicas' log end offsets are reduced to the target
> >> >> offset
> >> >> > before resuming mirroring. This avoids the inconsistency.
> >> >> >
> >> >> > If the destination cluster has an unclean leader election, it will
> >> >> > re-fetch the same data from the source. There won't be any
> >> inconsistency
> >> >> > either.
> >> >> > So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >
> >> >> >
> >> >> > Yes, that works, too. I agree this is better because the unclean
> >> leader
> >> >> > election is rare, we don't want to occupy a field just for it.
> >> >> >
> >> >> > So to achieve this, we have to:
> >> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
> >> >> > 2. When the destination cluster leader completes a log truncation
> >> after
> >> >> a
> >> >> > fetch response due to divergeEpoch, we know it must be caused by
> >> unclean
> >> >> > leader election in the source cluster because we only fetch
> committed
> >> >> > records. If “mirror.support.unclean.leader.election=false”, we
> don’t
> >> do
> >> >> > anything and continue the next fetch. If
> >> >> > “mirror.support.unclean.leader.election=true”, we remove the mirror
> >> >> fetcher
> >> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
> >> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be
> >> caught
> >> >> up
> >> >> > into ISR and reduced to the expected LEO.
> >> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
> >> >> >
> >> >> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch API.
> >> >> > KIP is updated.
> >> >> >
> >> >> >
> >> >> > > JR51. Make sense. To truncate based on LME, we need to issue
> >> another
> >> >> RPC
> >> >> > to translate the LME to an offset. It would be useful to include
> >> that in
> >> >> > the workflow description.
> >> >> >
> >> >> > About the additional RPC to translate the LME to the offset (i.e.
> >> >> > OffsetForLeaderEpochRequest), actually we don’t rely on this to do
> >> the
> >> >> > offset translation. We use the local leader epoch cache in the
> >> >> destination
> >> >> > cluster to translate the epoch to save the RPC call. This works
> >> because
> >> >> of
> >> >> > an invariant the LME truncation protocol maintains: for any leader
> >> >> epoch ≤
> >> >> > LME, that epoch's data belongs to exactly one cluster, either it
> >> >> originated
> >> >> > there, or it was mirrored. This holds even across repeated mirror
> ->
> >> >> > reverse-mirror -> mirror cycles, since each direction's
> >> LOG_TRUNCATION
> >> >> > phase re-establishes the invariant before that cluster starts
> >> mirroring.
> >> >> > With this invariant, the replication protocol can find out the
> >> diverge
> >> >> > point and truncate the log correctly when mirroring. For example:
> >> >> >
> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> > foo-0 leader log foo-0 leader log
> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> > Offset 2: A (LE: 0)
> >> >> > Offset 3: A (LE: 1)
> >> >> >
> >> >> > Cluster A crashes while Cluster B mirrors to offset 1. Failover to
> B
> >> and
> >> >> > store the LME=0 in cluster B. When cluster A recovers and starts
> the
> >> >> > reverse mirroring, it’ll query the LME from B and get back LME: 0.
> >> Then
> >> >> > cluster A gets the end offset for epoch 0 from leaderEpochCache,
> and
> >> >> gets
> >> >> > offset 2. So offset 3 is truncated. The log is like this now:
> >> >> >
> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> > foo-0 leader log foo-0 leader log
> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
> >> >> > Offset 3: B (LE: 10)
> >> >> >
> >> >> > When mirror starts, cluster A sends the fetch request with {
> >> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the divergingEpoch
> >> check
> >> >> in
> >> >> > the cluster B queries the endOffset for leader epoch 0, and found
> the
> >> >> end
> >> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns
> fetch
> >> >> > response with divergingEpoch info to cluster A for log truncation.
> In
> >> >> the
> >> >> > end, offset 2 in cluster A will be truncated and then log converges
> >> with
> >> >> > cluster B.
> >> >> >
> >> >> >
> >> >> >
> >> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
> >> >> > because it mixes querying the mirror partition state with querying
> >> the
> >> >> > mirror partition leader. Another option is to use
> >> >> ReadMirrorStatesRequest
> >> >> > to query the mirror partition state and a new request to query the
> >> >> mirror
> >> >> > partition leader. If we want the adminClient to query the right
> >> >> ClusterMirrorCoordinator
> >> >> > directly, we can extend the FindCoordinatorRequest to support that.
> >> >> >
> >> >> >
> >> >> > Good suggestion. Yes, we need to fan out to all brokers because we
> >> have
> >> >> to
> >> >> > collect info from mirror partition leaders and mirror coordinators.
> >> If
> >> >> we
> >> >> > create a new API to contact mirror partition leaders, it will make
> >> the
> >> >> API
> >> >> > cleaner and clearer. So we will do this:
> >> >> >
> >> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields from
> >> >> > DescribeClusterMirror request/response
> >> >> >
> >> >> > b. Add a new API: ReadClusterMirrorLags API to request for
> >> SourceOffset,
> >> >> > DestinationOffset, and Lag info.
> >> >> >
> >> >> > The workflow for admin client describeClusterMirror will be like
> >> this:
> >> >> > 1. adminClient sends describeClusterMirror to the leastLoadedNode
> >> broker
> >> >> > 2. The received broker collects the current mirrored partitions,
> and
> >> >> > groups them by coordinator node and by partition leader node.
> >> >> > 3. The broker sends ReadMirrorState request to the coordinator
> nodes
> >> to
> >> >> > get the mirror state related info
> >> >> > 4. The broker sends ReadClusterMirrorLags request to the mirror
> >> >> partition
> >> >> > leader nodes to get the mirror lag, source offset, destination
> offset
> >> >> > (which updates on every fetch response).
> >> >> > 5. The broker merge all the response into describeClusterMirror
> >> response
> >> >> > and send to admin client
> >> >> >
> >> >> > KIP is updated. Thanks for the suggestion.
> >> >> >
> >> >> >
> >> >> > > JR55. For __mirror_state topic, could we describe whether it is
> >> >> > compacted?
> >> >> >
> >> >> > Yes, it is a compacted topic. I made it much clearer in the
> >> >> > “ClusterMirrorCoordinator” section.
> >> >> >
> >> >> >
> >> >> > Thank you,
> >> >> > Luke
> >> >> >
> >> >> >
> >> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <
> dev@kafka.apache.org
> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> Hi, Luke,
> >> >> >>
> >> >> >> Thanks for the reply.
> >> >> >>
> >> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> >> specifically
> >> >> >> designed for the unclean leader election support in Cluster
> >> Mirroring
> >> >> >> because the unclean leader election will break the assumption that
> >> the
> >> >> >> committed data won’t be truncated."
> >> >> >> If the source cluster has an unclean leader election, the
> >> destination
> >> >> >> cluster will detect the inconsistency and truncate the log at the
> >> >> >> destination leader. We can just
> >> >> >> extend mirror.support.unclean.leader.election to cover this case,
> >> >> i.e., we
> >> >> >> wait until all replicas' log end offsets are reduced to the target
> >> >> offset
> >> >> >> before resuming mirroring. This avoids the inconsistency.
> >> >> >>
> >> >> >> If the destination cluster has an unclean leader election, it will
> >> >> >> re-fetch
> >> >> >> the same data from the source. There won't be any inconsistency
> >> either.
> >> >> >>
> >> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
> >> >> >>
> >> >> >> JR51. Make sense. To truncate based on LME, we need to issue
> another
> >> >> RPC
> >> >> >> to
> >> >> >> translate the LME to an offset. It would be useful to include that
> >> in
> >> >> the
> >> >> >> workflow description.
> >> >> >>
> >> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
> >> >> because
> >> >> >> it mixes querying the mirror partition state with querying the
> >> mirror
> >> >> >> partition leader. Another option is to use ReadMirrorStatesRequest
> >> to
> >> >> >> query
> >> >> >> the mirror partition state and a new request to query the mirror
> >> >> partition
> >> >> >> leader. If we want the adminClient to query the right
> >> >> >> ClusterMirrorCoordinator
> >> >> >> directly, we can extend the FindCoordinatorRequest to support
> that.
> >> >> >>
> >> >> >> JR55. For __mirror_state topic, could we describe whether it is
> >> >> compacted?
> >> >> >>
> >> >> >> Jun
> >> >> >>
> >> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com>
> wrote:
> >> >> >>
> >> >> >> > Hi Jun,
> >> >> >> >
> >> >> >> > Thanks for the comments.
> >> >> >> > Answering them below:
> >> >> >> >
> >> >> >> > > JR31.3 It makes sense for the new partition leader to query
> the
> >> >> >> current
> >> >> >> > mirror state from the local or remote MirrorMetadataManager that
> >> owns
> >> >> >> the
> >> >> >> > partition's state. However, the wording in the KIP is confusing.
> >> >> >> > MirrorMetadataManager
> >> >> >> > mirror partition state cache should be populated only by one
> way,
> >> >> which
> >> >> >> is
> >> >> >> > replaying __mirror_state partitions this broker leads. By saying
> >> >> "and by
> >> >> >> > fetching state from remote coordinators via ReadMirrorStates
> >> RPCs.",
> >> >> >> it's
> >> >> >> > as if the mirror partition state cache on each broker stores the
> >> >> state
> >> >> >> for
> >> >> >> > all mirror partitions.
> >> >> >> >
> >> >> >> > You’re right. KIP is updated to make it clear.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR36. I understand the intention now, but I still don't think
> we
> >> >> need
> >> >> >> > MirrorLeaderEpoch. Let's
> >> >> >> > assume the common case: no unclean leader election in the source
> >> >> cluster
> >> >> >> > during mirroring. Mirroring only fetches the committed data from
> >> the
> >> >> >> > source, which never changes if obtained a second time. In your
> >> >> example
> >> >> >> > image, follower broker 1 in the destination cluster can never
> >> obtain
> >> >> the
> >> >> >> > record B(0) because it's never committed in the source and thus
> is
> >> >> never
> >> >> >> > exposed to the destination leader. It's possible for broker 1 to
> >> >> first
> >> >> >> get
> >> >> >> > D(1) which is truncated later when broker 0 follows a new
> leader.
> >> But
> >> >> >> the
> >> >> >> > new leader will refetch the same D(1) from the source cluster.
> So,
> >> >> it's
> >> >> >> ok
> >> >> >> > for broker 1 to keep D(1).
> >> >> >> >
> >> >> >> >
> >> >> >> > Yes, you are correct. Because mirroring only fetches the
> committed
> >> >> data
> >> >> >> > from the source, the MirrorLeaderEpoch is unnecessary for normal
> >> >> >> mirroring.
> >> >> >> > Thanks for the clarification. So I think we can say, the
> >> >> >> > “MirrorLeaderEpoch” is specifically designed for the unclean
> >> leader
> >> >> >> > election support in Cluster Mirroring because the unclean leader
> >> >> >> election
> >> >> >> > will break the assumption that the committed data won’t be
> >> truncated.
> >> >> >> >
> >> >> >> > Given that, we could gate the MirrorLeaderEpoch field by only
> >> >> setting it
> >> >> >> > in the Fetch request when
> >> >> "mirror.support.unclean.leader.election=true"
> >> >> >> > (false by default), and skip it entirely for normal mirroring.
> >> What
> >> >> do
> >> >> >> you
> >> >> >> > think?
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR51. Sounds good. Why do we choose to store the last mirror
> >> epoch
> >> >> >> > instead
> >> >> >> > of a last mirror offset?
> >> >> >> >
> >> >> >> > Yes, originally we stored the “last mirror offset” instead of
> >> “last
> >> >> >> mirror
> >> >> >> > epoch”. But later, we found it doesn’t work for unclean leader
> >> >> election
> >> >> >> > cases. (yes, `unclean leader election` again :) ) For example:
> >> >> >> >
> >> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >> >
> >> >> >> > Cluster B is mirroring from A. And A crashes and failover to B.
> B
> >> >> stores
> >> >> >> > the “last mirror offset” as 1.
> >> >> >> > When cluster A recovers, somehow the unclean leader election
> >> happens,
> >> >> >> log
> >> >> >> > is empty in the new leader, and then some more leadership
> changes
> >> to
> >> >> >> make
> >> >> >> > the leader epoch become 10. Then it starts the reverse mirror
> >> from B.
> >> >> >> The
> >> >> >> > log looks like this:
> >> >> >> >
> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> >> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> >> >> >> > Offset 2: pid_res
> (LE:
> >> >> 10)
> >> >> >> >
> >> >> >> > When A starts mirroring from B, log is truncated to last mirror
> >> >> offset:
> >> >> >> 1
> >> >> >> > (no-op), and then fetch from cluster B. The 1st fetch request is
> >> like
> >> >> >> this:
> >> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2,
> …
> >> }.
> >> >> The
> >> >> >> > divergingEpoch check in the cluster B queries the endOffset for
> >> >> leader
> >> >> >> > epoch 10 (in Partition#readRecords), and found the end offset
> for
> >> LE
> >> >> 10
> >> >> >> is
> >> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch validation
> >> >> passes
> >> >> >> and
> >> >> >> > returns offset 2 to cluster A. In short, relying on “last mirror
> >> >> offset”
> >> >> >> > cannot resolve the log diverge issue in unclean leader election
> >> case.
> >> >> >> >
> >> >> >> > If instead, we truncate until “last mirror epoch”: 0, the
> offset 0
> >> >> and 1
> >> >> >> > in cluster A will be truncated as expected and then log converge
> >> with
> >> >> >> > cluster B correctly in the end.
> >> >> >> >
> >> >> >> >
> >> >> >> > Also, about JR53.2:
> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
> brokers
> >> and
> >> >> >> > merges
> >> >> >> > the responses client-side."
> >> >> >> > This is not ideal. Not every broker hosts a mirror partition. So
> >> >> sending
> >> >> >> > the request to every broker is wasteful. It's probably better to
> >> send
> >> >> >> the
> >> >> >> > request to a single broker, which can then contact the right
> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >
> >> >> >> > While working on this, I found we need to add one more tagged
> >> field
> >> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema because
> >> >> >> otherwise
> >> >> >> > the receiving broker doesn’t know if this is the request from
> >> client
> >> >> >> and it
> >> >> >> > needs to contact other brokers, or this is just a forwarded
> >> request.
> >> >> The
> >> >> >> > workflow is like this:
> >> >> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest to a
> >> >> >> > leastLoaded broker, which doesn’t contain the "Forwarded" field.
> >> >> >> > 2. The broker receives the request and sends
> >> >> >> DescribeClusterMirrorsRequest
> >> >> >> > with "Forwarded=true" to the responsible brokers and waits for
> the
> >> >> >> response
> >> >> >> > async.
> >> >> >> > 3. When all responses are received, merge the results and
> respond
> >> to
> >> >> the
> >> >> >> > client.
> >> >> >> >
> >> >> >> > Note: we cannot use ReadMirrorStatesRequest here because it
> >> doesn’t
> >> >> >> > contain the lag info in the coordinator.
> >> >> >> >
> >> >> >> > The alternative solution is the adminClient sends
> >> >> >> > DescribeClusterMirrorsRequest to the “responsible brokers only”
> >> (via
> >> >> >> > describeTopics), instead of ALL brokers, and merge the results
> in
> >> the
> >> >> >> > client side. This solution doesn’t need to add a new field in
> the
> >> >> >> > DescribeClusterMirrorsRequest and still can avoid unnecessary
> >> >> requests
> >> >> >> > sent.
> >> >> >> >
> >> >> >> > My preference is to keep the merge logic on the admin client
> side,
> >> >> which
> >> >> >> > saves the overhead in the broker side.
> >> >> >> > What do you think?
> >> >> >> >
> >> >> >> >
> >> >> >> > Thank you,
> >> >> >> > Luke
> >> >> >> >
> >> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
> >> dev@kafka.apache.org
> >> >> >
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Hi, Luke,
> >> >> >> >>
> >> >> >> >> Thanks for the reply.
> >> >> >> >>
> >> >> >> >> JR31.3 It makes sense for the new partition leader to query the
> >> >> current
> >> >> >> >> mirror state from the local or remote MirrorMetadataManager
> that
> >> >> owns
> >> >> >> the
> >> >> >> >> partition's state. However, the wording in the KIP is
> confusing.
> >> >> >> >> MirrorMetadataManager
> >> >> >> >> mirror partition state cache should be populated only by one
> way,
> >> >> >> which is
> >> >> >> >> replaying __mirror_state partitions this broker leads. By
> saying
> >> >> "and
> >> >> >> by
> >> >> >> >> fetching state from remote coordinators via ReadMirrorStates
> >> RPCs.",
> >> >> >> it's
> >> >> >> >> as if the mirror partition state cache on each broker stores
> the
> >> >> state
> >> >> >> for
> >> >> >> >> all mirror partitions.
> >> >> >> >>
> >> >> >> >> JR36. I understand the intention now, but I still don't think
> we
> >> >> need
> >> >> >> >> MirrorLeaderEpoch. Let's
> >> >> >> >> assume the common case: no unclean leader election in the
> source
> >> >> >> cluster
> >> >> >> >> during mirroring. Mirroring only fetches the committed data
> from
> >> the
> >> >> >> >> source, which never changes if obtained a second time. In your
> >> >> example
> >> >> >> >> image, follower broker 1 in the destination cluster can never
> >> obtain
> >> >> >> the
> >> >> >> >> record B(0) because it's never committed in the source and thus
> >> is
> >> >> >> never
> >> >> >> >> exposed to the destination leader. It's possible for broker 1
> to
> >> >> first
> >> >> >> get
> >> >> >> >> D(1) which is truncated later when broker 0 follows a new
> leader.
> >> >> But
> >> >> >> the
> >> >> >> >> new leader will refetch the same D(1) from the source cluster.
> >> So,
> >> >> >> it's ok
> >> >> >> >> for broker 1 to keep D(1).
> >> >> >> >>
> >> >> >> >> JR51. Sounds good. Why do we choose to store the last mirror
> >> epoch
> >> >> >> instead
> >> >> >> >> of a last mirror offset?
> >> >> >> >>
> >> >> >> >> Jun
> >> >> >> >>
> >> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com>
> >> wrote:
> >> >> >> >>
> >> >> >> >> > Hi Jun,
> >> >> >> >> >
> >> >> >> >> > Thanks for the great comments again.
> >> >> >> >> > Answering your question below.
> >> >> >> >> >
> >> >> >> >> > > JR31.3 "Regarding state caching: the mirror partition cache
> >> >> lives
> >> >> >> in
> >> >> >> >> > MirrorMetadataManager. It is populated in two ways: by
> >> replaying
> >> >> >> >> > __mirror_state partitions this broker leads, and by fetching
> >> state
> >> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> >> > Hmm, a given broker only caches the mirror partition state
> for
> >> >> >> >> partitions
> >> >> >> >> > hashed into the __mirror_state partition hosted on this
> broker,
> >> >> >> right?
> >> >> >> >> Why
> >> >> >> >> > does it need to fetch the state from other remote
> coordinators?
> >> >> >> >> >
> >> >> >> >> > > a given broker only caches the mirror partition state for
> >> >> >> partitions
> >> >> >> >> > hashed into the __mirror_state partition hosted on this
> broker,
> >> >> >> right?
> >> >> >> >> >
> >> >> >> >> > Right.
> >> >> >> >> >
> >> >> >> >> > > Why does it need to fetch the state from other remote
> >> >> coordinators?
> >> >> >> >> >
> >> >> >> >> > Because the data partition leader needs to know the current
> >> mirror
> >> >> >> state
> >> >> >> >> > of the partitions it leads. On a leadership change, the
> >> metadata
> >> >> >> delta
> >> >> >> >> is
> >> >> >> >> > published to the MirrorMetadataManager, which queries the
> >> current
> >> >> >> mirror
> >> >> >> >> > state for its led partitions from the coordinator (local or
> >> >> remote),
> >> >> >> and
> >> >> >> >> > processes tasks based on that state. This is the reason why
> it
> >> >> >> fetches
> >> >> >> >> the
> >> >> >> >> > state from other remote coordinator. This part of cache will
> be
> >> >> >> cleared
> >> >> >> >> > after the leadership change.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR36. I am still not quite sure why we need
> >> MirrorLeaderEpoch in
> >> >> >> the
> >> >> >> >> > Fetch
> >> >> >> >> > request. In the current Fetch request, in addition to
> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> > it has another field LastFetchedEpoch, which captures the
> last
> >> >> leader
> >> >> >> >> epoch
> >> >> >> >> > in the data and is used to resolve the inconsistency between
> >> the
> >> >> >> >> follower
> >> >> >> >> > and the leader. Does it achieve the same purpose as
> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to do the
> >> >> >> validation
> >> >> >> >> to
> >> >> >> >> > avoid the issue in KAFKA-18723. If we reject any fetched
> >> batches
> >> >> with
> >> >> >> >> epoch
> >> >> >> >> > higher than the LastFetchedEpoch, it means the follower can
> >> never
> >> >> >> accept
> >> >> >> >> > new appended batches with higher epoch in the leader node.
> >> >> >> >> >
> >> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes in
> >> the
> >> >> >> fetch
> >> >> >> >> > API:
> >> >> >> >> > 1. The receiver (leader node) uses it to validate it matches
> >> the
> >> >> >> leader
> >> >> >> >> > epoch in the leader node
> >> >> >> >> > 2. The sender (follower node) uses it to validate the epoch
> in
> >> the
> >> >> >> >> fetched
> >> >> >> >> > batches are all <= this value.
> >> >> >> >> >
> >> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still serving for
> >> >> >> followers
> >> >> >> >> in
> >> >> >> >> > the destination cluster. For purpose (2), my understanding of
> >> the
> >> >> >> >> > validation for KAFKA-18723 is like: the follower knows the
> >> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader node
> >> should
> >> >> >> >> contain in
> >> >> >> >> > the log batches when fetch request is sent. Any log batch
> >> beyond
> >> >> this
> >> >> >> >> epoch
> >> >> >> >> > in the fetch response means the follower’s metadata is stale.
> >> The
> >> >> >> >> batches
> >> >> >> >> > need to be rejected and follower needs to update metadata for
> >> next
> >> >> >> >> fetch.
> >> >> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch
> serves
> >> >> this
> >> >> >> >> purpose
> >> >> >> >> > to let the followers know the highest epoch in the leader
> node
> >> >> log in
> >> >> >> >> each
> >> >> >> >> > fetch response. Once the info is stale in the follower side,
> it
> >> >> >> updates
> >> >> >> >> it
> >> >> >> >> > from fetch response, and send next fetch request with correct
> >> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is
> possible
> >> >> like
> >> >> >> >> > KAFKA-18723.
> >> >> >> >> >
> >> >> >> >> > I tried not to add new field in Fetch API, but this is the
> best
> >> >> >> >> solution I
> >> >> >> >> > can think of.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR48. "The NumPartitions field is needed so the controller
> >> can
> >> >> >> create
> >> >> >> >> the
> >> >> >> >> > destination topic if it does not already exist. When a topic
> is
> >> >> being
> >> >> >> >> > mirrored for the first time, the destination cluster has no
> >> >> knowledge
> >> >> >> >> > of the source topic's partition count, so the request must
> >> carry
> >> >> it."
> >> >> >> >> > Hmm, this seems inconsistent. When a mirror is started, the
> >> source
> >> >> >> topic
> >> >> >> >> > may not exist yet. When the source topic is created later, we
> >> can
> >> >> >> start
> >> >> >> >> > mirroring without requiring the user to provide the number of
> >> >> >> >> partitions.
> >> >> >> >> >
> >> >> >> >> > > When the source topic is created later, we can start
> >> mirroring
> >> >> >> without
> >> >> >> >> > requiring the user to provide the number of partitions.
> >> >> >> >> >
> >> >> >> >> > This is because this happens in periodical sync up in the
> >> >> destination
> >> >> >> >> > cluster, and the numPartitions and other topic info in the
> >> source
> >> >> >> >> cluster
> >> >> >> >> > is already retrieved during the sync up process. Also, if
> users
> >> >> are
> >> >> >> >> > creating mirror topics via admin API, this info can also be
> >> >> retrieved
> >> >> >> >> > inside the admin client if not provided.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR51. Start Mirror: I am wondering if we truly need the
> last
> >> >> mirror
> >> >> >> >> epoch
> >> >> >> >> > for truncation when starting a mirror. When a mirror is
> >> stopped at
> >> >> >> the
> >> >> >> >> > source, we already bump up the leader epoch and write some
> >> >> additional
> >> >> >> >> > records for cleanup. It seems that we can just use the
> >> >> DivergingEpoch
> >> >> >> >> > returned in the initial mirror Fetch request to detect the
> >> >> diverging
> >> >> >> >> epoch
> >> >> >> >> > and use that for truncation at the mirror destination.
> >> >> >> >> >
> >> >> >> >> > That’s a good question. I thought about this before but
> >> >> >> unfortunately it
> >> >> >> >> > doesn’t work. The reason we need the last mirror epoch record
> >> is
> >> >> >> because
> >> >> >> >> > the destination cluster might not completely sync with the
> >> source
> >> >> >> >> cluster
> >> >> >> >> > before STOPPED. And this un-mirrored data might confuse the
> >> >> >> replication
> >> >> >> >> > protocol when reverse mirror. For example,
> >> >> >> >> >
> >> >> >> >> > Cluster A (source) —> Cluster B (dest)
> >> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> >> >> > Offset 2: from A (LE: 10)
> >> >> >> >> >
> >> >> >> >> > While cluster B mirrors until offset 1, the cluster A crashes
> >> and
> >> >> >> >> failover
> >> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
> >> >> MIRROR_PID_RESET
> >> >> >> >> with
> >> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from cluster
> B,
> >> >> the
> >> >> >> log
> >> >> >> >> > will be like this:
> >> >> >> >> >
> >> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> >> > foo-0 leader log: foo-0 leader log:
> >> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
> >> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
> >> >> >> >> >
> >> >> >> >> > As you can see, cluster A (new destination cluster) doesn’t
> >> know
> >> >> the
> >> >> >> >> > offset 2 is not belonging to the cluster B, which causes
> >> >> inconsistent
> >> >> >> >> logs.
> >> >> >> >> > So like what we described in the KIP, the last mirror epoch
> is
> >> the
> >> >> >> >> > synchronization point between source and destination. Any
> data
> >> >> beyond
> >> >> >> >> the
> >> >> >> >> > point is treated as unknown to the source cluster and should
> be
> >> >> >> >> truncated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR52. Stop Mirror:
> >> >> >> >> > > JR52.1 "3. If patterns are provided, the controller removes
> >> >> >> matching
> >> >> >> >> > entries from mirror.topics.include or adds them to
> >> >> >> >> mirror.topics.exclude on
> >> >> >> >> > the CLUSTER_MIRROR resource."
> >> >> >> >> > Why is it 'or' and not 'and'?
> >> >> >> >> >
> >> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> >> MirrorMetadataManager
> >> >> >> >> > discovers new source topics matching the persisted
> >> include/exclude
> >> >> >> >> patterns
> >> >> >> >> > and repeats steps 3 through 7 for each."
> >> >> >> >> > The broker will send a create topic request to the
> controller.
> >> How
> >> >> >> does
> >> >> >> >> the
> >> >> >> >> > controller associate it with the mirror name to create a
> >> >> >> >> > `MirrorTopicStateChangeRecord`?
> >> >> >> >> >
> >> >> >> >> > No, we don’t send a create topic request to the controller
> >> here.
> >> >> >> >> Actually
> >> >> >> >> > we send a StartMirrorTopics request to the controller, which
> >> >> contains
> >> >> >> >> the
> >> >> >> >> > mirror name and topic metadata like NumPartitions, topicId…
> >> etc.
> >> >> So
> >> >> >> the
> >> >> >> >> > controller can create the non-existed topic as explained in
> >> JR48,
> >> >> and
> >> >> >> >> > associate it to MirrorTopicStateChangeRecord.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR53. Describe Mirrors:
> >> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors request
> >> with
> >> >> >> >> optional
> >> >> >> >> > mirror names (empty means all mirrors)."
> >> >> >> >> > In metadata request, we use a null arrary to represent all
> >> >> topics. It
> >> >> >> >> would
> >> >> >> >> > be useful to be consistent here.
> >> >> >> >> >
> >> >> >> >> > OK, KIP updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
> >> brokers
> >> >> and
> >> >> >> >> > merges
> >> >> >> >> > the responses client-side."
> >> >> >> >> > This is not ideal. Not every broker hosts a mirror partition.
> >> So
> >> >> >> sending
> >> >> >> >> > the request to every broker is wasteful. It's probably better
> >> to
> >> >> send
> >> >> >> >> the
> >> >> >> >> > request to a single broker, which can then contact the right
> >> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >
> >> >> >> >> > Agree, KIP updated.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are
> >> MIRRORING,
> >> >> >> >> PAUSED,
> >> >> >> >> > STOPPED the only validate states? The state table has other
> >> states
> >> >> >> too.
> >> >> >> >> > Ditto for MirrorPartitionState.
> >> >> >> >> > The DesiredState field in MirrorTopicStateChangeRecord is to
> >> >> define
> >> >> >> the
> >> >> >> >> > target state the user wants. And the desired state users can
> >> >> assign
> >> >> >> is
> >> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state
> >> as
> >> >> >> >> STOPPING
> >> >> >> >> > or LOG_TRUNCATION… etc because they are more like
> intermediate
> >> >> >> states.
> >> >> >> >> >
> >> >> >> >> > About MirrorPartitionState record, it’ll record any state in
> >> the
> >> >> >> state
> >> >> >> >> > table that the current partition is. This is the real state
> for
> >> >> each
> >> >> >> >> > partition. It’s like in the kubernetes world, when we change
> >> the
> >> >> >> replica
> >> >> >> >> > field of the deployment yaml from 3 -> 5 (desired state), it
> >> won’t
> >> >> >> >> increase
> >> >> >> >> > to 5 immediately. The current replica (MirrorPartitionState
> >> >> record)
> >> >> >> is
> >> >> >> >> > still 3, then 4 (maybe), and finally 5.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Thank you,
> >> >> >> >> > Luke
> >> >> >> >> >
> >> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
> >> >> dev@kafka.apache.org
> >> >> >> >
> >> >> >> >> > wrote:
> >> >> >> >> >
> >> >> >> >> >> Hi, Fede,
> >> >> >> >> >>
> >> >> >> >> >> Thanks for the reply. A few more comments.
> >> >> >> >> >>
> >> >> >> >> >> JR31.3 "Regarding state caching: the mirror partition cache
> >> >> lives in
> >> >> >> >> >> MirrorMetadataManager. It is populated in two ways: by
> >> replaying
> >> >> >> >> >> __mirror_state partitions this broker leads, and by fetching
> >> >> state
> >> >> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> >> >> Hmm, a given broker only caches the mirror partition state
> for
> >> >> >> >> partitions
> >> >> >> >> >> hashed into the __mirror_state partition hosted on this
> >> broker,
> >> >> >> right?
> >> >> >> >> Why
> >> >> >> >> >> does it need to fetch the state from other remote
> >> coordinators?
> >> >> >> >> >>
> >> >> >> >> >> JR36. I am still not quite sure why we need
> MirrorLeaderEpoch
> >> in
> >> >> the
> >> >> >> >> Fetch
> >> >> >> >> >> request. In the current Fetch request, in addition to
> >> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> >> it has another field LastFetchedEpoch, which captures the
> last
> >> >> >> leader
> >> >> >> >> >> epoch
> >> >> >> >> >> in the data and is used to resolve the inconsistency between
> >> the
> >> >> >> >> follower
> >> >> >> >> >> and the leader. Does it achieve the same purpose as
> >> >> >> MirrorLeaderEpoch?
> >> >> >> >> >>
> >> >> >> >> >> JR48. "The NumPartitions field is needed so the controller
> can
> >> >> >> create
> >> >> >> >> the
> >> >> >> >> >> destination topic if it does not already exist. When a topic
> >> is
> >> >> >> being
> >> >> >> >> >> mirrored for the first time, the destination cluster has no
> >> >> >> knowledge
> >> >> >> >> >> of the source topic's partition count, so the request must
> >> carry
> >> >> >> it."
> >> >> >> >> >> Hmm, this seems inconsistent. When a mirror is started, the
> >> >> source
> >> >> >> >> topic
> >> >> >> >> >> may not exist yet. When the source topic is created later,
> we
> >> can
> >> >> >> start
> >> >> >> >> >> mirroring without requiring the user to provide the number
> of
> >> >> >> >> partitions.
> >> >> >> >> >>
> >> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need the last
> >> >> mirror
> >> >> >> >> epoch
> >> >> >> >> >> for truncation when starting a mirror. When a mirror is
> >> stopped
> >> >> at
> >> >> >> the
> >> >> >> >> >> source, we already bump up the leader epoch and write some
> >> >> >> additional
> >> >> >> >> >> records for cleanup. It seems that we can just use the
> >> >> >> DivergingEpoch
> >> >> >> >> >> returned in the initial mirror Fetch request to detect the
> >> >> diverging
> >> >> >> >> epoch
> >> >> >> >> >> and use that for truncation at the mirror destination.
> >> >> >> >> >>
> >> >> >> >> >> JR52. Stop Mirror:
> >> >> >> >> >> JR52.1 "3. If patterns are provided, the controller removes
> >> >> matching
> >> >> >> >> >> entries from mirror.topics.include or adds them to
> >> >> >> >> mirror.topics.exclude
> >> >> >> >> >> on
> >> >> >> >> >> the CLUSTER_MIRROR resource."
> >> >> >> >> >> Why is it 'or' and not 'and'?
> >> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> MirrorMetadataManager
> >> >> >> >> >> discovers new source topics matching the persisted
> >> >> include/exclude
> >> >> >> >> >> patterns
> >> >> >> >> >> and repeats steps 3 through 7 for each."
> >> >> >> >> >> The broker will send a create topic request to the
> controller.
> >> >> How
> >> >> >> does
> >> >> >> >> >> the
> >> >> >> >> >> controller associate it with the mirror name to create a
> >> >> >> >> >> `MirrorTopicStateChangeRecord`?
> >> >> >> >> >>
> >> >> >> >> >> JR53. Describe Mirrors:
> >> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors request
> >> with
> >> >> >> >> optional
> >> >> >> >> >> mirror names (empty means all mirrors)."
> >> >> >> >> >> In metadata request, we use a null arrary to represent all
> >> >> topics.
> >> >> >> It
> >> >> >> >> >> would
> >> >> >> >> >> be useful to be consistent here.
> >> >> >> >> >> JR53.2 "2. The admin client fans out the request to all
> >> brokers
> >> >> and
> >> >> >> >> merges
> >> >> >> >> >> the responses client-side."
> >> >> >> >> >> This is not ideal. Not every broker hosts a mirror
> partition.
> >> So
> >> >> >> >> sending
> >> >> >> >> >> the request to every broker is wasteful. It's probably
> better
> >> to
> >> >> >> send
> >> >> >> >> the
> >> >> >> >> >> request to a single broker, which can then contact the right
> >> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >> >>
> >> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are
> >> MIRRORING,
> >> >> >> PAUSED,
> >> >> >> >> >> STOPPED the only validate states? The state table has other
> >> >> states
> >> >> >> too.
> >> >> >> >> >> Ditto for MirrorPartitionState.
> >> >> >> >> >>
> >> >> >> >> >> Jun
> >> >> >> >> >>
> >> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
> >> >> >> fedevaleri@gmail.com>
> >> >> >> >> >> wrote:
> >> >> >> >> >>
> >> >> >> >> >> > Hi Andrew, yes, validation is called at the start of
> >> >> createTopic.
> >> >> >> If
> >> >> >> >> >> > the topic ID is invalid, the request fails with
> >> >> INVALID_REQUEST.
> >> >> >> If
> >> >> >> >> >> > the topic ID is already used by a different topic name, it
> >> >> fails
> >> >> >> with
> >> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual
> >> topic
> >> >> in
> >> >> >> the
> >> >> >> >> >> > batch fails; other topics in the same CreateTopics request
> >> >> >> continue
> >> >> >> >> >> > processing normally. Added this information to the KIP.
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> >
> >> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
> >> >> >> >> >> schofielaj@apache.org>
> >> >> >> >> >> > wrote:
> >> >> >> >> >> > >
> >> >> >> >> >> > > Hi Fede,
> >> >> >> >> >> > > One more small question.
> >> >> >> >> >> > >
> >> >> >> >> >> > > AS37: If the controller fails validation for the
> >> >> >> MirrorInfo.TopicId
> >> >> >> >> >> > added to the CreateTopic request, what does it do? I
> suspect
> >> >> that
> >> >> >> it
> >> >> >> >> >> fails
> >> >> >> >> >> > the request with a particular error code.
> >> >> >> >> >> > >
> >> >> >> >> >> > > Thanks,
> >> >> >> >> >> > > Andrew
> >> >> >> >> >> > >
> >> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> >> >> >> >> >> > > > Hi all,
> >> >> >> >> >> > > >
> >> >> >> >> >> > > > We updated the KIP to include Coordinator Runtime
> >> >> >> configurations
> >> >> >> >> and
> >> >> >> >> >> > > > the coordinator state transition validation paragraph.
> >> The
> >> >> >> latter
> >> >> >> >> >> > > > covers the per-partition epoch fencing mechanism
> (leader
> >> >> epoch
> >> >> >> >> and
> >> >> >> >> >> > > > state epoch) that complements the existing per-topic
> >> >> >> StateOffset
> >> >> >> >> >> > > > fencing for lifecycle operations.
> >> >> >> >> >> > > >
> >> >> >> >> >> > > > Thanks,
> >> >> >> >> >> > > > Fede
> >> >> >> >> >> > > >
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply.
JR59.1 Could you add the RPC to the workflow? The broker first needs to
issue ListClusterMirrorsRequest with SourceClusterIdFilter to get the
mirror name and then issue DescribeClusterMirrors to get the LME.
JR71. mirror.support.unclean.leader.election : The name is confusing. It's
also used in LOG_TRUNCATION, which is not related to unclean leader
election. Could we use a better name?
JR72. DescribeClusterMirrors
JR72.1 A user may specify just the topic name when describing a mirror.
Should we include the topic name in the request?
JR72.2 LastMirrorEpochLookups is a bit weird since the response returns
either mirror state or mirror lag and LastMirrorEpochLookups doesn't match
either.
JR72.3 Should it return UNKNOWN_CLUSTER_MIRROR too?
JR72.4 "includeMirrorLags: source offset, destination offset, lag"
DescribeClusterMirrorsRequest doesn't include those fields. Also, lag
should be omitted.
JR73. MirrorPartitionState record, LastMirrorEpochs record and
MirrorPartitionStateRequest: The values for type and name are incorrect.
JR74. ListClusterMirrorsRequest
JR74.1 When is DesiredMirrorStateFilter used? There is no option to specify
the desired state in the mirror-topic command.
JR74.2 Should it return CLUSTER_MIRROR_AUTHORIZATION_FAILED too?
JR75. ReadMirrorLags
JR75.1 Should it be named ReadMirrorOffsets since the response doesn't
include lag?
JR75.2 Should it return UNKNOWN_CLUSTER_MIRROR too?
JR75.3 Some places refer to this as ReadClusterMirrorLags. We need to be
consistent.
JR76. WriteMirrorStates: Should it return UNKNOWN_CLUSTER_MIRROR too?
JR77. Delete Mirror: "3. The controller verifies that no topics are in
MIRRORING or PAUSED state."
Is this necessary since step 2 has verified all partitions are in STOPPED
state, which can only happen if the topics are in STOPPED state.
JR78. STOPPING: "The mirror fetcher is removed, the leader epoch is bumped,
ABORT markers are appended for all ongoing transactions, last mirror epochs
are persisted, and a MIRROR_PID_RESET record appended."
This describes the state for STOPPED, not STOPPING.
JR79. RPCs like DescribeClusterMirrors can take time. Should we add a
timeout field?
JR80. PauseMirrorTopicsOptions: Should it support patterns? The command
line seems to suppor it.
JR81. AdminClient
JR81.1 ListClusterMirrors: We need to add the option to specify different
filters supported in the request.
JR81.2 DescribeClusterMirrorsOptions.includeAuthorizedOperations: The
request doesn't support this option. Could we add other options supported
in the request?
JR81.3 LeaderStateDescription: Should it be named
MirrorPartitionStateDescription? Also, could we make the fields consistent
with those in the response?
Jun
On Wed, Aug 12, 2026 at 5:36 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the good comments again.
> Answering them below.
>
> > JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election is
> enabled and the destination leader has log truncation, the state will move
> from MIRRORING to this state."
> This still suggests that divergeEpoch is only handled if
> mirror.support.unclean.leader.election
> is enabled.
>
>
> OK, KIP is updated to make it clear.
>
> > JR55. Should the mirror state and the mirror lags always be exposed
> together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
> from DescribeClusterMirrorsRequest. If not, it's better to use two separate
> RPCs.
>
>
> No, we sometimes only need MirrorState, and sometimes need both. So I’ll
> keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
> DescribeClusterMirrorsRequest. And we will use ReadMirrorStates and
> ReadMirrorLags to get each info.
>
>
> > DescribeClusterMirrorsRequest should be used in the Describe Mirror flow,
> right? Currently it's missing. Also, I assume the admin client refers to
> the one in the kafka-cluster-mirrors tool. But it seems that sometimes it
> also refers to the admin client on the broker? Perhaps we can use 'the
> tool' instead to make it clear.
>
> Yes, I missed the DescribeClusterMirrorsRequest. Added.
> And sorry for the confusion. I’d like to keep the admin client as the one
> invoking from the tool (or user’s direct admin client API invocation). On
> the broker side, we actually use InterBrokerSendThread to invoke these RPCs
> or get directly from MetadataImage. I’ve updated the KIP to make it clear.
>
>
> > JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
> instead of just to the controller?
>
> Yes, we only need to issue to the controller. Updated to the
> BumpLeaderEpochs API.
>
>
> > JR59. Start Mirror Topics:
> > JR59.1 "During this state, the broker queries the source cluster for last
> mirror epochs".
> Could you describe which RPC is issued to find the LME?
>
> We’re using describeClusterMirror API via admin client here.
>
> > JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
> doesn't
> contain a replication factor.
>
> We use “default.replication.factor” when topic creation, so the
> INVALID_REPLICATION_FACTOR will be thrown like normal topic creation. Also,
> because we require the replicas for the partition assignment to be all
> active and not in fenced or controlled shutdown(JR35), we’ll throw
> INVALID_REPLICATION_FACTOR if some replicas are fenced.
>
>
> > JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
> different topic name"
> Should we also return this error when the topic name is used as a regular
> topic?
>
> There are 2 errors we will throw when topic id <-> topic name mismatch.
> Suppose provided topic id: ID_A, topic name: NAME_A
> 1. Destination cluster exists topic id: ID_A, but topic name is: NAME_B
> TOPIC_ALREADY_EXISTS will be thrown
> 2. Destination cluster exists topic id: ID_B, but topic name is: NAME_A
> INCONSISTENT_TOPIC_ID will be thrown
>
> Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS error for
> these 2 cases makes more sense. KIP updated.
>
>
> > JR60. CreateClusterMirrorRequest: When will it receive an
> INVALID_CLUSTER_MIRROR_STATES error not covered by
> CLUSTER_MIRROR_ALREADY_EXISTS?
>
>
> Sorry, createClusterMirrorRequest doesn’t throw
> INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate the mirror
> states when creating a cluster mirror. We only care if the “mirror name”
> exists or not. KIP updated.
>
>
> > JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just remove
> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
> specific state causing the error.
>
> I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following errors:
> a. MIRROR_TOPIC_ALREADY_PAUSED
> b. MIRROR_TOPIC_NOT_PAUSED,
> c. MIRROR_TOPIC_ALREADY_STOPPED,
> d. CLUSTER_MIRROR_NOT_EMPTY
> e. MIRROR_TOPIC_NOT_STOPPED
>
> KIP is updated.
>
>
> > JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some requests
> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics). Could we
> make them consistent?
>
>
> Good catch. The updated errors for the new added RPC are this:
>
> CreateClusterMirror: NOT_CONTROLLER
> StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
>
>
> > JR63. StopMirrorTopics:
> > JR63.1 "The broker validates that all target topic partitions are in
> either
> LOG_TRUNCATION or MIRRORING or FAILED state."
> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
>
> Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is updated.
>
> > JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since PAUSED
> -> STOPPED is valid?
>
> Yes, it is not correct. KIP updated to remove this error.
>
> > JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
> state"
> This is weird. The reason to issue StopMirrorTopics is that it's not in
> stopped state.
>
> Indeed. KIP is updated to only include INVALID_CLUSTER_MIRROR_STATES
> error, and will be thrown when states are not in MIRRORING or PAUSED state.
>
> > JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's derived
> from SourceOffset and DestinationOffset.
>
> True. Removed.
>
> > JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
> could just set the partition level state to STOPPED.
>
> Nice catch. We don’t use it at all. That is some old leftover field we
> forgot to remove. KIP updated.
>
> > JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent to
> the controller?
>
> No, these 2 only send to the broker. KIP updated.
>
>
> > JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch? It
> seems both are used for fencing and fencing based on the latter is
> stronger.
>
> Yes, basically the StateEpoch can fence for most cases. The LeaderEpoch
> fencing is used for cases like:
>
> 1. b1 is the current leader for foo-0, preparing write state from
> MIRRORING to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
> 2. leadership change to b2 for foo-0, LE changes to 2.
> 3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1 }.
> 4. The coordinator receives the read from b2 in step (3), the highest
> lastSeen LE for foo-0 updates to 2 in coordinator.
> 5. The writes state from b1 in step (1) arrives the coordinator, but it
> rejects b1 writes because the highest lastSeen LE is 2, which is > 1 in the
> request.
> 6. b2 writes state from MIRRORING to PAUSING in LE: 2, stateEpoch: 1 and
> succeeds.
>
> The leaderEpoch fence is to avoid any stale broker writes after the new
> leader epoch read. This cannot be protected by the stateEpoch. The
> stateEpoch can prevent the write with older stateEpoch. So in the situation
> above, the stateEpoch cannot determine if we should accept b1 writes or
> not, but leaderEpoch can. By the way, we borrowed this concept from
> KIP-932:
> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
> <https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932*Queues*for*Kafka*KIP932*3AQueuesforKafka-WriteShareGroupStateAPI__;KysrIyU!!Ayb5sqE7!rM45DU3iceu-9W4moXR1xuj6XCrozML7X6eIkpxep5RCI9NA3jAcpJzCnEx76Q83EhQko0_Ye9r9bPQ$>
>
>
> > JR68. Could we make the usage of topicId and topicName in the RPCs
> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> BumpLeaderEpochsReponse uses TopicName.
>
> Yes, we have made:
> 1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use TopicName.
> 2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse use
> TopicName.
> 3. MirrorTopicStateChangeRecord uses TopicName because we will do
> validation before writing the metadata log.
>
>
> > JR69. MIRROR_PID_RESET:
> > JR69.1 Could you specify the value of the control record's type?
>
> It’s MIRROR_PID_RESET(7). KIP updated.
>
> > JR69.2 How is the SourceClusterId field used?
>
> I’m also surprised to see this field in the MirrorPidResetRecord. I traced
> back to the internal PR we added for this field, and confirmed it is not
> used since it was added. Sorry for the confusion and nice catch! KIP
> updated.
>
> > JR70. MirrorPartitionState: type should be "data" not "coordinator-key",
> right? Ditto for LastMirrorEpochs.
> We store both MirrorPartitionState and LastMirrorEpochs records into
> internal __mirror_state topic, and rely on Coordinator Runtime to
> write/read the records, like how `OffsetCommit` and `GroupMetadata` work.
> So I think we should keep them as "coordinator-key"/"coordinator-value"
> type.
>
>
> Thank you,
> Luke
>
> On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Luke,
>>
>> Thanks for the reply.
>>
>> JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election is
>> enabled and the destination leader has log truncation, the state will move
>> from MIRRORING to this state."
>> This still suggests that divergeEpoch is only handled if
>> mirror.support.unclean.leader.election
>> is enabled.
>>
>> JR55. Should the mirror state and the mirror lags always be exposed
>> together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
>> from DescribeClusterMirrorsRequest. If not, it's better to use two
>> separate
>> RPCs.
>>
>> DescribeClusterMirrorsRequest should be used in the Describe Mirror flow,
>> right? Currently it's missing. Also, I assume the admin client refers to
>> the one in the kafka-cluster-mirrors tool. But it seems that sometimes it
>> also refers to the admin client on the broker? Perhaps we can use 'the
>> tool' instead to make it clear.
>>
>> JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
>> instead of just to the controller?
>>
>> JR59. Start Mirror Topics:
>> JR59.1 "During this state, the broker queries the source cluster for last
>> mirror epochs".
>> Could you describe which RPC is issued to find the LME?
>> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
>> doesn't
>> contain a replication factor.
>> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
>> different topic name"
>> Should we also return this error when the topic name is used as a regular
>> topic?
>>
>> JR60. CreateClusterMirrorRequest: When will it receive an
>> INVALID_CLUSTER_MIRROR_STATES error not covered by
>> CLUSTER_MIRROR_ALREADY_EXISTS?
>>
>> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just remove
>> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
>> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
>> specific state causing the error.
>>
>> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some requests
>> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics). Could we
>> make them consistent?
>>
>> JR63. StopMirrorTopics:
>> JR63.1 "The broker validates that all target topic partitions are in
>> either
>> LOG_TRUNCATION or MIRRORING or FAILED state."
>> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
>> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since PAUSED
>> -> STOPPED is valid?
>> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
>> state"
>> This is weird. The reason to issue StopMirrorTopics is that it's not in
>> stopped state.
>>
>> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's derived
>> from SourceOffset and DestinationOffset.
>>
>> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
>> could just set the partition level state to STOPPED.
>>
>> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent to
>> the controller?
>>
>> JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch? It
>> seems both are used for fencing and fencing based on the latter is
>> stronger.
>>
>> JR68. Could we make the usage of topicId and topicName in the RPCs
>> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
>> BumpLeaderEpochsReponse uses TopicName.
>>
>> JR69. MIRROR_PID_RESET:
>> JR69.1 Could you specify the value of the control record's type?
>> JR69.2 How is the SourceClusterId field used?
>>
>> JR70. MirrorPartitionState: type should be "data" not "coordinator-key",
>> right? Ditto for LastMirrorEpochs.
>>
>> Jun
>>
>> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for the comments.
>> > Answering them below.
>> >
>> > > JR36. Sounds good. Should we remove
>> MirrorLeaderEpochExceededException?
>> >
>> > Sorry, it’s a confusing name. The MirrorLeaderEpochExceededException is
>> > used and thrown when the fetched mirror batch leader epoch > local
>> leader
>> > epoch in destination cluster. And then we’ll stop mirroring and move the
>> > partition to EPOCH_FENSING state to wait for local leader epoch bump. I
>> > think since we don’t have the concept of `MirrorLeaderEpoch` anymore,
>> the
>> > `MirrorLeaderEpochExceededException` should not cause confusion now.
>> I’ll
>> > keep the name for the exception.
>> >
>> >
>> > > JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
>> > mirror
>> > fetch request issued to the source cluster, independent of the value of
>> > mirror.support.unclean.leader.election? It's needed to detect divergence
>> > during failover. However, if it's also set when
>> > mirror.support.unclean.leader.election=false,
>> > an unclean leader election in the source cluster can halt the mirroring
>> > since we don’t do anything with the returned divergeEpoch in that mode
>> > and continue
>> > the next fetch.
>> >
>> > > Do we plan to always set lastFetchedEpoch in the mirror
>> > fetch request issued to the source cluster, independent of the value of
>> > Mirror.support.unclean.leader.election?
>> >
>> > Yes, we always set lastFetchedEpoch in the mirror fetch request.
>> >
>> > > However, if it's also set when
>> > mirror.support.unclean.leader.election=false,
>> > an unclean leader election in the source cluster can halt the mirroring
>> > since we don’t do anything with the returned divergeEpoch in that mode
>> > and continue the next fetch.
>> >
>> > Sorry I didn’t make it clear. When
>> > “mirror.support.unclean.leader.election=false” and a “divergeEpoch” is
>> set
>> > in fetch response, the leader node will handle the divergeEpoch and then
>> > continue the next fetch as usual. The difference with
>> > “mirror.support.unclean.leader.election=true” is that “should we wait
>> for
>> > ALL replicas caught up (complete truncation)”. Without the waiting, it’s
>> > possible the inconsistent log issue in KAFKA-18723 could happen between
>> > leader and follower nodes in the destination cluster because we don’t
>> have
>> > “MirrorLeaderEpoch” concept now to reject higher leader epoch batches.
>> That
>> > is, we won't halt the mirroring if "divergeEpoch" is set but
>> > “mirror.support.unclean.leader.election=false”.
>> >
>> >
>> > > JR55. The DescribeClusterMirrorsRequest still returns both the mirror
>> > partition state and the mirror partition leader state. The problem is
>> that
>> > if a caller only needs one of the state, the other state becomes
>> redundant.
>> > For example, when starting the mirror, to obtain LME, we only need the
>> > mirror partition state. Here is an alternative design: We expose the
>> mirror
>> > partition state through ReadMirrorState. The client will use the
>> > FindCoordinatorRequest
>> > to find the right mirror partition coordinator to issue that request. We
>> > expose the mirror partition leader state through the
>> ReadClusterMirrorLags
>> > request. The client can use the metadata request to find the partition
>> > leader to issue that request.
>> >
>> > > In the describing mirror flow, the admin client first issues the
>> > ListMirrorRequest to find all the mirrors and the topics being
>> mirrored. It
>> > then issues the metadata request to find the partitions for those topics
>> > and issues a FindCoordinatorRequest to find the mirror partition state
>> > coordinators. Finally, it issues ReadMirrorState to the mirror partition
>> > state coordinator and ReadClusterMirrorLags to the partition leader.
>> >
>> > > In the starting mirror flow, the partition leader in the destination
>> > cluster issues a ListMirrorRequest to find the mirror name based on its
>> > cluster id. It then issues a FindCoordinatorRequest to find the mirror
>> > partition state coordinators. Finally, it issues ReadMirrorState to the
>> > mirror partition state coordinator to get the LME.
>> >
>> > > We can also improve the ListMirror request to accept a filter based on
>> > mirror name, source cluster ID and topic name.
>> >
>> >
>> > Answers:
>> >
>> > > The DescribeClusterMirrorsRequest still returns both the mirror
>> > partition state and the mirror partition leader state. The problem is
>> that
>> > if a caller only needs one of the state, the other state becomes
>> redundant.
>> > For example, when starting the mirror, to obtain LME, we only need the
>> > mirror partition state.
>> >
>> > Good point!
>> >
>> >
>> > > Here is an alternative design: We expose the mirror
>> > partition state through ReadMirrorState. The client will use the
>> > FindCoordinatorRequest
>> > to find the right mirror partition coordinator to issue that request. We
>> > expose the mirror partition leader state through the
>> ReadClusterMirrorLags
>> > request. The client can use the metadata request to find the partition
>> > leader to issue that request.
>> >
>> >
>> > Sounds good. But we try not to expose
>> > ReadMirrorState/ReadClusterMirrorLags directly to the admin client
>> because
>> > users might get confused about these 2 requests, and they are also both
>> > related to describeClusterMirror. I think we can add 2 additional
>> fields to
>> > describeClusterMirrorRequest: “includeMirrorStates” and
>> > “includeMirrorLags”. So the admin client can include either one or both
>> of
>> > them in the DescrbieClusterMirrorRequest based on the provided options.
>> KIP
>> > is updated.
>> >
>> >
>> > > In the describing mirror flow, the admin client first issues the
>> > ListMirrorRequest to find all the mirrors and the topics being
>> mirrored. It
>> > then issues the metadata request to find the partitions for those topics
>> > and issues a FindCoordinatorRequest to find the mirror partition state
>> > coordinators. Finally, it issues ReadMirrorState to the mirror partition
>> > state coordinator and ReadClusterMirrorLags to the partition leader.
>> >
>> > Makes sense. KIP is updated.
>> >
>> >
>> > > In the starting mirror flow, the partition leader in the destination
>> > cluster issues a ListMirrorRequest to find the mirror name based on its
>> > cluster id. It then issues a FindCoordinatorRequest to find the mirror
>> > partition state coordinators. Finally, it issues ReadMirrorState to the
>> > mirror partition state coordinator to get the LME.
>> >
>> > Make sense. So we can send DescribeClusterMirrorRequest with
>> > “includeMirrorStates” only.
>> >
>> > > We can also improve the ListMirror request to accept a filter based on
>> > mirror name, source cluster ID and topic name.
>> >
>> > Good suggestion, just like what ListGroupsRequest does.
>> > Added "SourceClusterIdFilter", "MirrorNameFilter",
>> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP
>> updated.
>> >
>> >
>> > > JR56. Could you list the ACL needed for each new RPC?
>> >
>> >
>> > Yes, it’s already listed in “Security Control” section.
>> >
>> >
>> > > JR57. In the command workflow, it sometimes says "The user sends" and
>> > some
>> > other times says "The admin client sends". It's better to use "The admin
>> > client" consistently in all those places.
>> >
>> > Agree. KIP updated.
>> >
>> >
>> > Thank you,
>> > Luke
>> >
>> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <de...@kafka.apache.org>
>> > wrote:
>> >
>> >> Hi, Luke,
>> >>
>> >> Thanks for the reply.
>> >>
>> >> JR36. Sounds good. Should we remove MirrorLeaderEpochExceededException?
>> >>
>> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
>> mirror
>> >> fetch request issued to the source cluster, independent of the value of
>> >> mirror.support.unclean.leader.election? It's needed to detect
>> divergence
>> >> during failover. However, if it's also set when
>> >> mirror.support.unclean.leader.election=false,
>> >> an unclean leader election in the source cluster can halt the mirroring
>> >> since we don’t do anything with the returned divergeEpoch in that mode
>> >> and continue
>> >> the next fetch.
>> >>
>> >> JR55. The DescribeClusterMirrorsRequest still returns both the mirror
>> >> partition state and the mirror partition leader state. The problem is
>> that
>> >> if a caller only needs one of the state, the other state becomes
>> >> redundant.
>> >> For example, when starting the mirror, to obtain LME, we only need the
>> >> mirror partition state. Here is an alternative design: We expose the
>> >> mirror
>> >> partition state through ReadMirrorState. The client will use the
>> >> FindCoordinatorRequest
>> >> to find the right mirror partition coordinator to issue that request.
>> We
>> >> expose the mirror partition leader state through the
>> ReadClusterMirrorLags
>> >> request. The client can use the metadata request to find the partition
>> >> leader to issue that request.
>> >>
>> >> In the describing mirror flow, the admin client first issues the
>> >> ListMirrorRequest to find all the mirrors and the topics being
>> mirrored.
>> >> It
>> >> then issues the metadata request to find the partitions for those
>> topics
>> >> and issues a FindCoordinatorRequest to find the mirror partition state
>> >> coordinators. Finally, it issues ReadMirrorState to the mirror
>> partition
>> >> state coordinator and ReadClusterMirrorLags to the partition leader.
>> >>
>> >> In the starting mirror flow, the partition leader in the destination
>> >> cluster issues a ListMirrorRequest to find the mirror name based on its
>> >> cluster id. It then issues a FindCoordinatorRequest to find the mirror
>> >> partition state coordinators. Finally, it issues ReadMirrorState to the
>> >> mirror partition state coordinator to get the LME.
>> >>
>> >> We can also improve the ListMirror request to accept a filter based on
>> >> mirror name, source cluster ID and topic name.
>> >>
>> >> JR56. Could you list the ACL needed for each new RPC?
>> >>
>> >> JR57. In the command workflow, it sometimes says "The user sends" and
>> some
>> >> other times says "The admin client sends". It's better to use "The
>> admin
>> >> client" consistently in all those places.
>> >>
>> >> Jun
>> >>
>> >>
>> >>
>> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com> wrote:
>> >>
>> >> > Hi Jun,
>> >> >
>> >> > Thanks for your comments.
>> >> > Answering them below.
>> >> >
>> >> >
>> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> specifically
>> >> > designed for the unclean leader election support in Cluster Mirroring
>> >> > because the unclean leader election will break the assumption that
>> the
>> >> > committed data won’t be truncated."
>> >> > If the source cluster has an unclean leader election, the destination
>> >> > cluster will detect the inconsistency and truncate the log at the
>> >> > destination leader. We can just
>> >> > extend mirror.support.unclean.leader.election to cover this case,
>> i.e.,
>> >> we
>> >> > wait until all replicas' log end offsets are reduced to the target
>> >> offset
>> >> > before resuming mirroring. This avoids the inconsistency.
>> >> >
>> >> > If the destination cluster has an unclean leader election, it will
>> >> > re-fetch the same data from the source. There won't be any
>> inconsistency
>> >> > either.
>> >> > So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >
>> >> >
>> >> > Yes, that works, too. I agree this is better because the unclean
>> leader
>> >> > election is rare, we don't want to occupy a field just for it.
>> >> >
>> >> > So to achieve this, we have to:
>> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
>> >> > 2. When the destination cluster leader completes a log truncation
>> after
>> >> a
>> >> > fetch response due to divergeEpoch, we know it must be caused by
>> unclean
>> >> > leader election in the source cluster because we only fetch committed
>> >> > records. If “mirror.support.unclean.leader.election=false”, we don’t
>> do
>> >> > anything and continue the next fetch. If
>> >> > “mirror.support.unclean.leader.election=true”, we remove the mirror
>> >> fetcher
>> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
>> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be
>> caught
>> >> up
>> >> > into ISR and reduced to the expected LEO.
>> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
>> >> >
>> >> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch API.
>> >> > KIP is updated.
>> >> >
>> >> >
>> >> > > JR51. Make sense. To truncate based on LME, we need to issue
>> another
>> >> RPC
>> >> > to translate the LME to an offset. It would be useful to include
>> that in
>> >> > the workflow description.
>> >> >
>> >> > About the additional RPC to translate the LME to the offset (i.e.
>> >> > OffsetForLeaderEpochRequest), actually we don’t rely on this to do
>> the
>> >> > offset translation. We use the local leader epoch cache in the
>> >> destination
>> >> > cluster to translate the epoch to save the RPC call. This works
>> because
>> >> of
>> >> > an invariant the LME truncation protocol maintains: for any leader
>> >> epoch ≤
>> >> > LME, that epoch's data belongs to exactly one cluster, either it
>> >> originated
>> >> > there, or it was mirrored. This holds even across repeated mirror ->
>> >> > reverse-mirror -> mirror cycles, since each direction's
>> LOG_TRUNCATION
>> >> > phase re-establishes the invariant before that cluster starts
>> mirroring.
>> >> > With this invariant, the replication protocol can find out the
>> diverge
>> >> > point and truncate the log correctly when mirroring. For example:
>> >> >
>> >> > Cluster A (source) --> Cluster B (dest)
>> >> > foo-0 leader log foo-0 leader log
>> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> > Offset 2: A (LE: 0)
>> >> > Offset 3: A (LE: 1)
>> >> >
>> >> > Cluster A crashes while Cluster B mirrors to offset 1. Failover to B
>> and
>> >> > store the LME=0 in cluster B. When cluster A recovers and starts the
>> >> > reverse mirroring, it’ll query the LME from B and get back LME: 0.
>> Then
>> >> > cluster A gets the end offset for epoch 0 from leaderEpochCache, and
>> >> gets
>> >> > offset 2. So offset 3 is truncated. The log is like this now:
>> >> >
>> >> > Cluster A (dest) <-- Cluster B (source)
>> >> > foo-0 leader log foo-0 leader log
>> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
>> >> > Offset 3: B (LE: 10)
>> >> >
>> >> > When mirror starts, cluster A sends the fetch request with {
>> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the divergingEpoch
>> check
>> >> in
>> >> > the cluster B queries the endOffset for leader epoch 0, and found the
>> >> end
>> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns fetch
>> >> > response with divergingEpoch info to cluster A for log truncation. In
>> >> the
>> >> > end, offset 2 in cluster A will be truncated and then log converges
>> with
>> >> > cluster B.
>> >> >
>> >> >
>> >> >
>> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
>> >> > because it mixes querying the mirror partition state with querying
>> the
>> >> > mirror partition leader. Another option is to use
>> >> ReadMirrorStatesRequest
>> >> > to query the mirror partition state and a new request to query the
>> >> mirror
>> >> > partition leader. If we want the adminClient to query the right
>> >> ClusterMirrorCoordinator
>> >> > directly, we can extend the FindCoordinatorRequest to support that.
>> >> >
>> >> >
>> >> > Good suggestion. Yes, we need to fan out to all brokers because we
>> have
>> >> to
>> >> > collect info from mirror partition leaders and mirror coordinators.
>> If
>> >> we
>> >> > create a new API to contact mirror partition leaders, it will make
>> the
>> >> API
>> >> > cleaner and clearer. So we will do this:
>> >> >
>> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields from
>> >> > DescribeClusterMirror request/response
>> >> >
>> >> > b. Add a new API: ReadClusterMirrorLags API to request for
>> SourceOffset,
>> >> > DestinationOffset, and Lag info.
>> >> >
>> >> > The workflow for admin client describeClusterMirror will be like
>> this:
>> >> > 1. adminClient sends describeClusterMirror to the leastLoadedNode
>> broker
>> >> > 2. The received broker collects the current mirrored partitions, and
>> >> > groups them by coordinator node and by partition leader node.
>> >> > 3. The broker sends ReadMirrorState request to the coordinator nodes
>> to
>> >> > get the mirror state related info
>> >> > 4. The broker sends ReadClusterMirrorLags request to the mirror
>> >> partition
>> >> > leader nodes to get the mirror lag, source offset, destination offset
>> >> > (which updates on every fetch response).
>> >> > 5. The broker merge all the response into describeClusterMirror
>> response
>> >> > and send to admin client
>> >> >
>> >> > KIP is updated. Thanks for the suggestion.
>> >> >
>> >> >
>> >> > > JR55. For __mirror_state topic, could we describe whether it is
>> >> > compacted?
>> >> >
>> >> > Yes, it is a compacted topic. I made it much clearer in the
>> >> > “ClusterMirrorCoordinator” section.
>> >> >
>> >> >
>> >> > Thank you,
>> >> > Luke
>> >> >
>> >> >
>> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <dev@kafka.apache.org
>> >
>> >> > wrote:
>> >> >
>> >> >> Hi, Luke,
>> >> >>
>> >> >> Thanks for the reply.
>> >> >>
>> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is
>> specifically
>> >> >> designed for the unclean leader election support in Cluster
>> Mirroring
>> >> >> because the unclean leader election will break the assumption that
>> the
>> >> >> committed data won’t be truncated."
>> >> >> If the source cluster has an unclean leader election, the
>> destination
>> >> >> cluster will detect the inconsistency and truncate the log at the
>> >> >> destination leader. We can just
>> >> >> extend mirror.support.unclean.leader.election to cover this case,
>> >> i.e., we
>> >> >> wait until all replicas' log end offsets are reduced to the target
>> >> offset
>> >> >> before resuming mirroring. This avoids the inconsistency.
>> >> >>
>> >> >> If the destination cluster has an unclean leader election, it will
>> >> >> re-fetch
>> >> >> the same data from the source. There won't be any inconsistency
>> either.
>> >> >>
>> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
>> >> >>
>> >> >> JR51. Make sense. To truncate based on LME, we need to issue another
>> >> RPC
>> >> >> to
>> >> >> translate the LME to an offset. It would be useful to include that
>> in
>> >> the
>> >> >> workflow description.
>> >> >>
>> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
>> >> because
>> >> >> it mixes querying the mirror partition state with querying the
>> mirror
>> >> >> partition leader. Another option is to use ReadMirrorStatesRequest
>> to
>> >> >> query
>> >> >> the mirror partition state and a new request to query the mirror
>> >> partition
>> >> >> leader. If we want the adminClient to query the right
>> >> >> ClusterMirrorCoordinator
>> >> >> directly, we can extend the FindCoordinatorRequest to support that.
>> >> >>
>> >> >> JR55. For __mirror_state topic, could we describe whether it is
>> >> compacted?
>> >> >>
>> >> >> Jun
>> >> >>
>> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com> wrote:
>> >> >>
>> >> >> > Hi Jun,
>> >> >> >
>> >> >> > Thanks for the comments.
>> >> >> > Answering them below:
>> >> >> >
>> >> >> > > JR31.3 It makes sense for the new partition leader to query the
>> >> >> current
>> >> >> > mirror state from the local or remote MirrorMetadataManager that
>> owns
>> >> >> the
>> >> >> > partition's state. However, the wording in the KIP is confusing.
>> >> >> > MirrorMetadataManager
>> >> >> > mirror partition state cache should be populated only by one way,
>> >> which
>> >> >> is
>> >> >> > replaying __mirror_state partitions this broker leads. By saying
>> >> "and by
>> >> >> > fetching state from remote coordinators via ReadMirrorStates
>> RPCs.",
>> >> >> it's
>> >> >> > as if the mirror partition state cache on each broker stores the
>> >> state
>> >> >> for
>> >> >> > all mirror partitions.
>> >> >> >
>> >> >> > You’re right. KIP is updated to make it clear.
>> >> >> >
>> >> >> >
>> >> >> > > JR36. I understand the intention now, but I still don't think we
>> >> need
>> >> >> > MirrorLeaderEpoch. Let's
>> >> >> > assume the common case: no unclean leader election in the source
>> >> cluster
>> >> >> > during mirroring. Mirroring only fetches the committed data from
>> the
>> >> >> > source, which never changes if obtained a second time. In your
>> >> example
>> >> >> > image, follower broker 1 in the destination cluster can never
>> obtain
>> >> the
>> >> >> > record B(0) because it's never committed in the source and thus is
>> >> never
>> >> >> > exposed to the destination leader. It's possible for broker 1 to
>> >> first
>> >> >> get
>> >> >> > D(1) which is truncated later when broker 0 follows a new leader.
>> But
>> >> >> the
>> >> >> > new leader will refetch the same D(1) from the source cluster. So,
>> >> it's
>> >> >> ok
>> >> >> > for broker 1 to keep D(1).
>> >> >> >
>> >> >> >
>> >> >> > Yes, you are correct. Because mirroring only fetches the committed
>> >> data
>> >> >> > from the source, the MirrorLeaderEpoch is unnecessary for normal
>> >> >> mirroring.
>> >> >> > Thanks for the clarification. So I think we can say, the
>> >> >> > “MirrorLeaderEpoch” is specifically designed for the unclean
>> leader
>> >> >> > election support in Cluster Mirroring because the unclean leader
>> >> >> election
>> >> >> > will break the assumption that the committed data won’t be
>> truncated.
>> >> >> >
>> >> >> > Given that, we could gate the MirrorLeaderEpoch field by only
>> >> setting it
>> >> >> > in the Fetch request when
>> >> "mirror.support.unclean.leader.election=true"
>> >> >> > (false by default), and skip it entirely for normal mirroring.
>> What
>> >> do
>> >> >> you
>> >> >> > think?
>> >> >> >
>> >> >> >
>> >> >> > > JR51. Sounds good. Why do we choose to store the last mirror
>> epoch
>> >> >> > instead
>> >> >> > of a last mirror offset?
>> >> >> >
>> >> >> > Yes, originally we stored the “last mirror offset” instead of
>> “last
>> >> >> mirror
>> >> >> > epoch”. But later, we found it doesn’t work for unclean leader
>> >> election
>> >> >> > cases. (yes, `unclean leader election` again :) ) For example:
>> >> >> >
>> >> >> > Cluster A (source) --> Cluster B (dest)
>> >> >> > foo-0 leader log foo-0 leader log
>> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >> >
>> >> >> > Cluster B is mirroring from A. And A crashes and failover to B. B
>> >> stores
>> >> >> > the “last mirror offset” as 1.
>> >> >> > When cluster A recovers, somehow the unclean leader election
>> happens,
>> >> >> log
>> >> >> > is empty in the new leader, and then some more leadership changes
>> to
>> >> >> make
>> >> >> > the leader epoch become 10. Then it starts the reverse mirror
>> from B.
>> >> >> The
>> >> >> > log looks like this:
>> >> >> >
>> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> > foo-0 leader log foo-0 leader log
>> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
>> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
>> >> >> > Offset 2: pid_res (LE:
>> >> 10)
>> >> >> >
>> >> >> > When A starts mirroring from B, log is truncated to last mirror
>> >> offset:
>> >> >> 1
>> >> >> > (no-op), and then fetch from cluster B. The 1st fetch request is
>> like
>> >> >> this:
>> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2, …
>> }.
>> >> The
>> >> >> > divergingEpoch check in the cluster B queries the endOffset for
>> >> leader
>> >> >> > epoch 10 (in Partition#readRecords), and found the end offset for
>> LE
>> >> 10
>> >> >> is
>> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch validation
>> >> passes
>> >> >> and
>> >> >> > returns offset 2 to cluster A. In short, relying on “last mirror
>> >> offset”
>> >> >> > cannot resolve the log diverge issue in unclean leader election
>> case.
>> >> >> >
>> >> >> > If instead, we truncate until “last mirror epoch”: 0, the offset 0
>> >> and 1
>> >> >> > in cluster A will be truncated as expected and then log converge
>> with
>> >> >> > cluster B correctly in the end.
>> >> >> >
>> >> >> >
>> >> >> > Also, about JR53.2:
>> >> >> > > JR53.2 "2. The admin client fans out the request to all brokers
>> and
>> >> >> > merges
>> >> >> > the responses client-side."
>> >> >> > This is not ideal. Not every broker hosts a mirror partition. So
>> >> sending
>> >> >> > the request to every broker is wasteful. It's probably better to
>> send
>> >> >> the
>> >> >> > request to a single broker, which can then contact the right
>> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >
>> >> >> > While working on this, I found we need to add one more tagged
>> field
>> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema because
>> >> >> otherwise
>> >> >> > the receiving broker doesn’t know if this is the request from
>> client
>> >> >> and it
>> >> >> > needs to contact other brokers, or this is just a forwarded
>> request.
>> >> The
>> >> >> > workflow is like this:
>> >> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest to a
>> >> >> > leastLoaded broker, which doesn’t contain the "Forwarded" field.
>> >> >> > 2. The broker receives the request and sends
>> >> >> DescribeClusterMirrorsRequest
>> >> >> > with "Forwarded=true" to the responsible brokers and waits for the
>> >> >> response
>> >> >> > async.
>> >> >> > 3. When all responses are received, merge the results and respond
>> to
>> >> the
>> >> >> > client.
>> >> >> >
>> >> >> > Note: we cannot use ReadMirrorStatesRequest here because it
>> doesn’t
>> >> >> > contain the lag info in the coordinator.
>> >> >> >
>> >> >> > The alternative solution is the adminClient sends
>> >> >> > DescribeClusterMirrorsRequest to the “responsible brokers only”
>> (via
>> >> >> > describeTopics), instead of ALL brokers, and merge the results in
>> the
>> >> >> > client side. This solution doesn’t need to add a new field in the
>> >> >> > DescribeClusterMirrorsRequest and still can avoid unnecessary
>> >> requests
>> >> >> > sent.
>> >> >> >
>> >> >> > My preference is to keep the merge logic on the admin client side,
>> >> which
>> >> >> > saves the overhead in the broker side.
>> >> >> > What do you think?
>> >> >> >
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Luke
>> >> >> >
>> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
>> dev@kafka.apache.org
>> >> >
>> >> >> > wrote:
>> >> >> >
>> >> >> >> Hi, Luke,
>> >> >> >>
>> >> >> >> Thanks for the reply.
>> >> >> >>
>> >> >> >> JR31.3 It makes sense for the new partition leader to query the
>> >> current
>> >> >> >> mirror state from the local or remote MirrorMetadataManager that
>> >> owns
>> >> >> the
>> >> >> >> partition's state. However, the wording in the KIP is confusing.
>> >> >> >> MirrorMetadataManager
>> >> >> >> mirror partition state cache should be populated only by one way,
>> >> >> which is
>> >> >> >> replaying __mirror_state partitions this broker leads. By saying
>> >> "and
>> >> >> by
>> >> >> >> fetching state from remote coordinators via ReadMirrorStates
>> RPCs.",
>> >> >> it's
>> >> >> >> as if the mirror partition state cache on each broker stores the
>> >> state
>> >> >> for
>> >> >> >> all mirror partitions.
>> >> >> >>
>> >> >> >> JR36. I understand the intention now, but I still don't think we
>> >> need
>> >> >> >> MirrorLeaderEpoch. Let's
>> >> >> >> assume the common case: no unclean leader election in the source
>> >> >> cluster
>> >> >> >> during mirroring. Mirroring only fetches the committed data from
>> the
>> >> >> >> source, which never changes if obtained a second time. In your
>> >> example
>> >> >> >> image, follower broker 1 in the destination cluster can never
>> obtain
>> >> >> the
>> >> >> >> record B(0) because it's never committed in the source and thus
>> is
>> >> >> never
>> >> >> >> exposed to the destination leader. It's possible for broker 1 to
>> >> first
>> >> >> get
>> >> >> >> D(1) which is truncated later when broker 0 follows a new leader.
>> >> But
>> >> >> the
>> >> >> >> new leader will refetch the same D(1) from the source cluster.
>> So,
>> >> >> it's ok
>> >> >> >> for broker 1 to keep D(1).
>> >> >> >>
>> >> >> >> JR51. Sounds good. Why do we choose to store the last mirror
>> epoch
>> >> >> instead
>> >> >> >> of a last mirror offset?
>> >> >> >>
>> >> >> >> Jun
>> >> >> >>
>> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com>
>> wrote:
>> >> >> >>
>> >> >> >> > Hi Jun,
>> >> >> >> >
>> >> >> >> > Thanks for the great comments again.
>> >> >> >> > Answering your question below.
>> >> >> >> >
>> >> >> >> > > JR31.3 "Regarding state caching: the mirror partition cache
>> >> lives
>> >> >> in
>> >> >> >> > MirrorMetadataManager. It is populated in two ways: by
>> replaying
>> >> >> >> > __mirror_state partitions this broker leads, and by fetching
>> state
>> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> > Hmm, a given broker only caches the mirror partition state for
>> >> >> >> partitions
>> >> >> >> > hashed into the __mirror_state partition hosted on this broker,
>> >> >> right?
>> >> >> >> Why
>> >> >> >> > does it need to fetch the state from other remote coordinators?
>> >> >> >> >
>> >> >> >> > > a given broker only caches the mirror partition state for
>> >> >> partitions
>> >> >> >> > hashed into the __mirror_state partition hosted on this broker,
>> >> >> right?
>> >> >> >> >
>> >> >> >> > Right.
>> >> >> >> >
>> >> >> >> > > Why does it need to fetch the state from other remote
>> >> coordinators?
>> >> >> >> >
>> >> >> >> > Because the data partition leader needs to know the current
>> mirror
>> >> >> state
>> >> >> >> > of the partitions it leads. On a leadership change, the
>> metadata
>> >> >> delta
>> >> >> >> is
>> >> >> >> > published to the MirrorMetadataManager, which queries the
>> current
>> >> >> mirror
>> >> >> >> > state for its led partitions from the coordinator (local or
>> >> remote),
>> >> >> and
>> >> >> >> > processes tasks based on that state. This is the reason why it
>> >> >> fetches
>> >> >> >> the
>> >> >> >> > state from other remote coordinator. This part of cache will be
>> >> >> cleared
>> >> >> >> > after the leadership change.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR36. I am still not quite sure why we need
>> MirrorLeaderEpoch in
>> >> >> the
>> >> >> >> > Fetch
>> >> >> >> > request. In the current Fetch request, in addition to
>> >> >> >> CurrentLeaderEpoch,
>> >> >> >> > it has another field LastFetchedEpoch, which captures the last
>> >> leader
>> >> >> >> epoch
>> >> >> >> > in the data and is used to resolve the inconsistency between
>> the
>> >> >> >> follower
>> >> >> >> > and the leader. Does it achieve the same purpose as
>> >> >> MirrorLeaderEpoch?
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to do the
>> >> >> validation
>> >> >> >> to
>> >> >> >> > avoid the issue in KAFKA-18723. If we reject any fetched
>> batches
>> >> with
>> >> >> >> epoch
>> >> >> >> > higher than the LastFetchedEpoch, it means the follower can
>> never
>> >> >> accept
>> >> >> >> > new appended batches with higher epoch in the leader node.
>> >> >> >> >
>> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes in
>> the
>> >> >> fetch
>> >> >> >> > API:
>> >> >> >> > 1. The receiver (leader node) uses it to validate it matches
>> the
>> >> >> leader
>> >> >> >> > epoch in the leader node
>> >> >> >> > 2. The sender (follower node) uses it to validate the epoch in
>> the
>> >> >> >> fetched
>> >> >> >> > batches are all <= this value.
>> >> >> >> >
>> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still serving for
>> >> >> followers
>> >> >> >> in
>> >> >> >> > the destination cluster. For purpose (2), my understanding of
>> the
>> >> >> >> > validation for KAFKA-18723 is like: the follower knows the
>> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader node
>> should
>> >> >> >> contain in
>> >> >> >> > the log batches when fetch request is sent. Any log batch
>> beyond
>> >> this
>> >> >> >> epoch
>> >> >> >> > in the fetch response means the follower’s metadata is stale.
>> The
>> >> >> >> batches
>> >> >> >> > need to be rejected and follower needs to update metadata for
>> next
>> >> >> >> fetch.
>> >> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves
>> >> this
>> >> >> >> purpose
>> >> >> >> > to let the followers know the highest epoch in the leader node
>> >> log in
>> >> >> >> each
>> >> >> >> > fetch response. Once the info is stale in the follower side, it
>> >> >> updates
>> >> >> >> it
>> >> >> >> > from fetch response, and send next fetch request with correct
>> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is possible
>> >> like
>> >> >> >> > KAFKA-18723.
>> >> >> >> >
>> >> >> >> > I tried not to add new field in Fetch API, but this is the best
>> >> >> >> solution I
>> >> >> >> > can think of.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR48. "The NumPartitions field is needed so the controller
>> can
>> >> >> create
>> >> >> >> the
>> >> >> >> > destination topic if it does not already exist. When a topic is
>> >> being
>> >> >> >> > mirrored for the first time, the destination cluster has no
>> >> knowledge
>> >> >> >> > of the source topic's partition count, so the request must
>> carry
>> >> it."
>> >> >> >> > Hmm, this seems inconsistent. When a mirror is started, the
>> source
>> >> >> topic
>> >> >> >> > may not exist yet. When the source topic is created later, we
>> can
>> >> >> start
>> >> >> >> > mirroring without requiring the user to provide the number of
>> >> >> >> partitions.
>> >> >> >> >
>> >> >> >> > > When the source topic is created later, we can start
>> mirroring
>> >> >> without
>> >> >> >> > requiring the user to provide the number of partitions.
>> >> >> >> >
>> >> >> >> > This is because this happens in periodical sync up in the
>> >> destination
>> >> >> >> > cluster, and the numPartitions and other topic info in the
>> source
>> >> >> >> cluster
>> >> >> >> > is already retrieved during the sync up process. Also, if users
>> >> are
>> >> >> >> > creating mirror topics via admin API, this info can also be
>> >> retrieved
>> >> >> >> > inside the admin client if not provided.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR51. Start Mirror: I am wondering if we truly need the last
>> >> mirror
>> >> >> >> epoch
>> >> >> >> > for truncation when starting a mirror. When a mirror is
>> stopped at
>> >> >> the
>> >> >> >> > source, we already bump up the leader epoch and write some
>> >> additional
>> >> >> >> > records for cleanup. It seems that we can just use the
>> >> DivergingEpoch
>> >> >> >> > returned in the initial mirror Fetch request to detect the
>> >> diverging
>> >> >> >> epoch
>> >> >> >> > and use that for truncation at the mirror destination.
>> >> >> >> >
>> >> >> >> > That’s a good question. I thought about this before but
>> >> >> unfortunately it
>> >> >> >> > doesn’t work. The reason we need the last mirror epoch record
>> is
>> >> >> because
>> >> >> >> > the destination cluster might not completely sync with the
>> source
>> >> >> >> cluster
>> >> >> >> > before STOPPED. And this un-mirrored data might confuse the
>> >> >> replication
>> >> >> >> > protocol when reverse mirror. For example,
>> >> >> >> >
>> >> >> >> > Cluster A (source) —> Cluster B (dest)
>> >> >> >> > foo-0 leader log foo-0 leader log
>> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> >> >> > Offset 2: from A (LE: 10)
>> >> >> >> >
>> >> >> >> > While cluster B mirrors until offset 1, the cluster A crashes
>> and
>> >> >> >> failover
>> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
>> >> MIRROR_PID_RESET
>> >> >> >> with
>> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from cluster B,
>> >> the
>> >> >> log
>> >> >> >> > will be like this:
>> >> >> >> >
>> >> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> >> > foo-0 leader log: foo-0 leader log:
>> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
>> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
>> >> >> >> >
>> >> >> >> > As you can see, cluster A (new destination cluster) doesn’t
>> know
>> >> the
>> >> >> >> > offset 2 is not belonging to the cluster B, which causes
>> >> inconsistent
>> >> >> >> logs.
>> >> >> >> > So like what we described in the KIP, the last mirror epoch is
>> the
>> >> >> >> > synchronization point between source and destination. Any data
>> >> beyond
>> >> >> >> the
>> >> >> >> > point is treated as unknown to the source cluster and should be
>> >> >> >> truncated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR52. Stop Mirror:
>> >> >> >> > > JR52.1 "3. If patterns are provided, the controller removes
>> >> >> matching
>> >> >> >> > entries from mirror.topics.include or adds them to
>> >> >> >> mirror.topics.exclude on
>> >> >> >> > the CLUSTER_MIRROR resource."
>> >> >> >> > Why is it 'or' and not 'and'?
>> >> >> >> >
>> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> >> MirrorMetadataManager
>> >> >> >> > discovers new source topics matching the persisted
>> include/exclude
>> >> >> >> patterns
>> >> >> >> > and repeats steps 3 through 7 for each."
>> >> >> >> > The broker will send a create topic request to the controller.
>> How
>> >> >> does
>> >> >> >> the
>> >> >> >> > controller associate it with the mirror name to create a
>> >> >> >> > `MirrorTopicStateChangeRecord`?
>> >> >> >> >
>> >> >> >> > No, we don’t send a create topic request to the controller
>> here.
>> >> >> >> Actually
>> >> >> >> > we send a StartMirrorTopics request to the controller, which
>> >> contains
>> >> >> >> the
>> >> >> >> > mirror name and topic metadata like NumPartitions, topicId…
>> etc.
>> >> So
>> >> >> the
>> >> >> >> > controller can create the non-existed topic as explained in
>> JR48,
>> >> and
>> >> >> >> > associate it to MirrorTopicStateChangeRecord.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR53. Describe Mirrors:
>> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors request
>> with
>> >> >> >> optional
>> >> >> >> > mirror names (empty means all mirrors)."
>> >> >> >> > In metadata request, we use a null arrary to represent all
>> >> topics. It
>> >> >> >> would
>> >> >> >> > be useful to be consistent here.
>> >> >> >> >
>> >> >> >> > OK, KIP updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
>> brokers
>> >> and
>> >> >> >> > merges
>> >> >> >> > the responses client-side."
>> >> >> >> > This is not ideal. Not every broker hosts a mirror partition.
>> So
>> >> >> sending
>> >> >> >> > the request to every broker is wasteful. It's probably better
>> to
>> >> send
>> >> >> >> the
>> >> >> >> > request to a single broker, which can then contact the right
>> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >
>> >> >> >> > Agree, KIP updated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are
>> MIRRORING,
>> >> >> >> PAUSED,
>> >> >> >> > STOPPED the only validate states? The state table has other
>> states
>> >> >> too.
>> >> >> >> > Ditto for MirrorPartitionState.
>> >> >> >> > The DesiredState field in MirrorTopicStateChangeRecord is to
>> >> define
>> >> >> the
>> >> >> >> > target state the user wants. And the desired state users can
>> >> assign
>> >> >> is
>> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state
>> as
>> >> >> >> STOPPING
>> >> >> >> > or LOG_TRUNCATION… etc because they are more like intermediate
>> >> >> states.
>> >> >> >> >
>> >> >> >> > About MirrorPartitionState record, it’ll record any state in
>> the
>> >> >> state
>> >> >> >> > table that the current partition is. This is the real state for
>> >> each
>> >> >> >> > partition. It’s like in the kubernetes world, when we change
>> the
>> >> >> replica
>> >> >> >> > field of the deployment yaml from 3 -> 5 (desired state), it
>> won’t
>> >> >> >> increase
>> >> >> >> > to 5 immediately. The current replica (MirrorPartitionState
>> >> record)
>> >> >> is
>> >> >> >> > still 3, then 4 (maybe), and finally 5.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Thank you,
>> >> >> >> > Luke
>> >> >> >> >
>> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
>> >> dev@kafka.apache.org
>> >> >> >
>> >> >> >> > wrote:
>> >> >> >> >
>> >> >> >> >> Hi, Fede,
>> >> >> >> >>
>> >> >> >> >> Thanks for the reply. A few more comments.
>> >> >> >> >>
>> >> >> >> >> JR31.3 "Regarding state caching: the mirror partition cache
>> >> lives in
>> >> >> >> >> MirrorMetadataManager. It is populated in two ways: by
>> replaying
>> >> >> >> >> __mirror_state partitions this broker leads, and by fetching
>> >> state
>> >> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> >> Hmm, a given broker only caches the mirror partition state for
>> >> >> >> partitions
>> >> >> >> >> hashed into the __mirror_state partition hosted on this
>> broker,
>> >> >> right?
>> >> >> >> Why
>> >> >> >> >> does it need to fetch the state from other remote
>> coordinators?
>> >> >> >> >>
>> >> >> >> >> JR36. I am still not quite sure why we need MirrorLeaderEpoch
>> in
>> >> the
>> >> >> >> Fetch
>> >> >> >> >> request. In the current Fetch request, in addition to
>> >> >> >> CurrentLeaderEpoch,
>> >> >> >> >> it has another field LastFetchedEpoch, which captures the last
>> >> >> leader
>> >> >> >> >> epoch
>> >> >> >> >> in the data and is used to resolve the inconsistency between
>> the
>> >> >> >> follower
>> >> >> >> >> and the leader. Does it achieve the same purpose as
>> >> >> MirrorLeaderEpoch?
>> >> >> >> >>
>> >> >> >> >> JR48. "The NumPartitions field is needed so the controller can
>> >> >> create
>> >> >> >> the
>> >> >> >> >> destination topic if it does not already exist. When a topic
>> is
>> >> >> being
>> >> >> >> >> mirrored for the first time, the destination cluster has no
>> >> >> knowledge
>> >> >> >> >> of the source topic's partition count, so the request must
>> carry
>> >> >> it."
>> >> >> >> >> Hmm, this seems inconsistent. When a mirror is started, the
>> >> source
>> >> >> >> topic
>> >> >> >> >> may not exist yet. When the source topic is created later, we
>> can
>> >> >> start
>> >> >> >> >> mirroring without requiring the user to provide the number of
>> >> >> >> partitions.
>> >> >> >> >>
>> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need the last
>> >> mirror
>> >> >> >> epoch
>> >> >> >> >> for truncation when starting a mirror. When a mirror is
>> stopped
>> >> at
>> >> >> the
>> >> >> >> >> source, we already bump up the leader epoch and write some
>> >> >> additional
>> >> >> >> >> records for cleanup. It seems that we can just use the
>> >> >> DivergingEpoch
>> >> >> >> >> returned in the initial mirror Fetch request to detect the
>> >> diverging
>> >> >> >> epoch
>> >> >> >> >> and use that for truncation at the mirror destination.
>> >> >> >> >>
>> >> >> >> >> JR52. Stop Mirror:
>> >> >> >> >> JR52.1 "3. If patterns are provided, the controller removes
>> >> matching
>> >> >> >> >> entries from mirror.topics.include or adds them to
>> >> >> >> mirror.topics.exclude
>> >> >> >> >> on
>> >> >> >> >> the CLUSTER_MIRROR resource."
>> >> >> >> >> Why is it 'or' and not 'and'?
>> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> MirrorMetadataManager
>> >> >> >> >> discovers new source topics matching the persisted
>> >> include/exclude
>> >> >> >> >> patterns
>> >> >> >> >> and repeats steps 3 through 7 for each."
>> >> >> >> >> The broker will send a create topic request to the controller.
>> >> How
>> >> >> does
>> >> >> >> >> the
>> >> >> >> >> controller associate it with the mirror name to create a
>> >> >> >> >> `MirrorTopicStateChangeRecord`?
>> >> >> >> >>
>> >> >> >> >> JR53. Describe Mirrors:
>> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors request
>> with
>> >> >> >> optional
>> >> >> >> >> mirror names (empty means all mirrors)."
>> >> >> >> >> In metadata request, we use a null arrary to represent all
>> >> topics.
>> >> >> It
>> >> >> >> >> would
>> >> >> >> >> be useful to be consistent here.
>> >> >> >> >> JR53.2 "2. The admin client fans out the request to all
>> brokers
>> >> and
>> >> >> >> merges
>> >> >> >> >> the responses client-side."
>> >> >> >> >> This is not ideal. Not every broker hosts a mirror partition.
>> So
>> >> >> >> sending
>> >> >> >> >> the request to every broker is wasteful. It's probably better
>> to
>> >> >> send
>> >> >> >> the
>> >> >> >> >> request to a single broker, which can then contact the right
>> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >> >>
>> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are
>> MIRRORING,
>> >> >> PAUSED,
>> >> >> >> >> STOPPED the only validate states? The state table has other
>> >> states
>> >> >> too.
>> >> >> >> >> Ditto for MirrorPartitionState.
>> >> >> >> >>
>> >> >> >> >> Jun
>> >> >> >> >>
>> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
>> >> >> fedevaleri@gmail.com>
>> >> >> >> >> wrote:
>> >> >> >> >>
>> >> >> >> >> > Hi Andrew, yes, validation is called at the start of
>> >> createTopic.
>> >> >> If
>> >> >> >> >> > the topic ID is invalid, the request fails with
>> >> INVALID_REQUEST.
>> >> >> If
>> >> >> >> >> > the topic ID is already used by a different topic name, it
>> >> fails
>> >> >> with
>> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual
>> topic
>> >> in
>> >> >> the
>> >> >> >> >> > batch fails; other topics in the same CreateTopics request
>> >> >> continue
>> >> >> >> >> > processing normally. Added this information to the KIP.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
>> >> >> >> >> schofielaj@apache.org>
>> >> >> >> >> > wrote:
>> >> >> >> >> > >
>> >> >> >> >> > > Hi Fede,
>> >> >> >> >> > > One more small question.
>> >> >> >> >> > >
>> >> >> >> >> > > AS37: If the controller fails validation for the
>> >> >> MirrorInfo.TopicId
>> >> >> >> >> > added to the CreateTopic request, what does it do? I suspect
>> >> that
>> >> >> it
>> >> >> >> >> fails
>> >> >> >> >> > the request with a particular error code.
>> >> >> >> >> > >
>> >> >> >> >> > > Thanks,
>> >> >> >> >> > > Andrew
>> >> >> >> >> > >
>> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> >> >> >> >> > > > Hi all,
>> >> >> >> >> > > >
>> >> >> >> >> > > > We updated the KIP to include Coordinator Runtime
>> >> >> configurations
>> >> >> >> and
>> >> >> >> >> > > > the coordinator state transition validation paragraph.
>> The
>> >> >> latter
>> >> >> >> >> > > > covers the per-partition epoch fencing mechanism (leader
>> >> epoch
>> >> >> >> and
>> >> >> >> >> > > > state epoch) that complements the existing per-topic
>> >> >> StateOffset
>> >> >> >> >> > > > fencing for lifecycle operations.
>> >> >> >> >> > > >
>> >> >> >> >> > > > Thanks,
>> >> >> >> >> > > > Fede
>> >> >> >> >> > > >
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for the good comments again.
Answering them below.
> JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election is
enabled and the destination leader has log truncation, the state will move
from MIRRORING to this state."
This still suggests that divergeEpoch is only handled if
mirror.support.unclean.leader.election
is enabled.
OK, KIP is updated to make it clear.
> JR55. Should the mirror state and the mirror lags always be exposed
together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
from DescribeClusterMirrorsRequest. If not, it's better to use two separate
RPCs.
No, we sometimes only need MirrorState, and sometimes need both. So I’ll
keep “IncludeMirrorState” and “IncludeMirrorLags” fields in
DescribeClusterMirrorsRequest. And we will use ReadMirrorStates and
ReadMirrorLags to get each info.
> DescribeClusterMirrorsRequest should be used in the Describe Mirror flow,
right? Currently it's missing. Also, I assume the admin client refers to
the one in the kafka-cluster-mirrors tool. But it seems that sometimes it
also refers to the admin client on the broker? Perhaps we can use 'the
tool' instead to make it clear.
Yes, I missed the DescribeClusterMirrorsRequest. Added.
And sorry for the confusion. I’d like to keep the admin client as the one
invoking from the tool (or user’s direct admin client API invocation). On
the broker side, we actually use InterBrokerSendThread to invoke these RPCs
or get directly from MetadataImage. I’ve updated the KIP to make it clear.
> JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
instead of just to the controller?
Yes, we only need to issue to the controller. Updated to the
BumpLeaderEpochs API.
> JR59. Start Mirror Topics:
> JR59.1 "During this state, the broker queries the source cluster for last
mirror epochs".
Could you describe which RPC is issued to find the LME?
We’re using describeClusterMirror API via admin client here.
> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request
doesn't
contain a replication factor.
We use “default.replication.factor” when topic creation, so the
INVALID_REPLICATION_FACTOR will be thrown like normal topic creation. Also,
because we require the replicas for the partition assignment to be all
active and not in fenced or controlled shutdown(JR35), we’ll throw
INVALID_REPLICATION_FACTOR if some replicas are fenced.
> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
different topic name"
Should we also return this error when the topic name is used as a regular
topic?
There are 2 errors we will throw when topic id <-> topic name mismatch.
Suppose provided topic id: ID_A, topic name: NAME_A
1. Destination cluster exists topic id: ID_A, but topic name is: NAME_B
TOPIC_ALREADY_EXISTS will be thrown
2. Destination cluster exists topic id: ID_B, but topic name is: NAME_A
INCONSISTENT_TOPIC_ID will be thrown
Hmm.. indeed, it looks like using the same TOPIC_ALREADY_EXISTS error for
these 2 cases makes more sense. KIP updated.
> JR60. CreateClusterMirrorRequest: When will it receive an
INVALID_CLUSTER_MIRROR_STATES error not covered by
CLUSTER_MIRROR_ALREADY_EXISTS?
Sorry, createClusterMirrorRequest doesn’t throw
INVALID_CLUSTER_MIRROR_STATES because we don’t need to validate the mirror
states when creating a cluster mirror. We only care if the “mirror name”
exists or not. KIP updated.
> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just remove
MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
specific state causing the error.
I agree. INVALID_CLUSTER_MIRROR_STATES can cover the following errors:
a. MIRROR_TOPIC_ALREADY_PAUSED
b. MIRROR_TOPIC_NOT_PAUSED,
c. MIRROR_TOPIC_ALREADY_STOPPED,
d. CLUSTER_MIRROR_NOT_EMPTY
e. MIRROR_TOPIC_NOT_STOPPED
KIP is updated.
> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some requests
(e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics). Could we
make them consistent?
Good catch. The updated errors for the new added RPC are this:
CreateClusterMirror: NOT_CONTROLLER
StartMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
StopMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
PauseMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
ResumeMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
DeleteMirrorTopics: NOT_CONTROLLER and UNKNOWN_CLUSTER_MIRROR
> JR63. StopMirrorTopics:
> JR63.1 "The broker validates that all target topic partitions are in
either
LOG_TRUNCATION or MIRRORING or FAILED state."
Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
Yes, nice catch. LOG_TRUNCATION should change to PAUSED. KIP is updated.
> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since PAUSED
-> STOPPED is valid?
Yes, it is not correct. KIP updated to remove this error.
> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped
state"
This is weird. The reason to issue StopMirrorTopics is that it's not in
stopped state.
Indeed. KIP is updated to only include INVALID_CLUSTER_MIRROR_STATES error,
and will be thrown when states are not in MIRRORING or PAUSED state.
> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's derived
from SourceOffset and DestinationOffset.
True. Removed.
> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
could just set the partition level state to STOPPED.
Nice catch. We don’t use it at all. That is some old leftover field we
forgot to remove. KIP updated.
> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent to
the controller?
No, these 2 only send to the broker. KIP updated.
> JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch? It
seems both are used for fencing and fencing based on the latter is stronger.
Yes, basically the StateEpoch can fence for most cases. The LeaderEpoch
fencing is used for cases like:
1. b1 is the current leader for foo-0, preparing write state from MIRRORING
to EPOCH_FENCING in { LE: 1, stateEpoch: 1 }
2. leadership change to b2 for foo-0, LE changes to 2.
3. b2 reads state from coordinator with { LE: 2, stateEpoch: 1 }.
4. The coordinator receives the read from b2 in step (3), the highest
lastSeen LE for foo-0 updates to 2 in coordinator.
5. The writes state from b1 in step (1) arrives the coordinator, but it
rejects b1 writes because the highest lastSeen LE is 2, which is > 1 in the
request.
6. b2 writes state from MIRRORING to PAUSING in LE: 2, stateEpoch: 1 and
succeeds.
The leaderEpoch fence is to avoid any stale broker writes after the new
leader epoch read. This cannot be protected by the stateEpoch. The
stateEpoch can prevent the write with older stateEpoch. So in the situation
above, the stateEpoch cannot determine if we should accept b1 writes or
not, but leaderEpoch can. By the way, we borrowed this concept from
KIP-932:
https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka#KIP932%3AQueuesforKafka-WriteShareGroupStateAPI
> JR68. Could we make the usage of topicId and topicName in the RPCs
consistent? Also, BumpLeaderEpochsRequest uses topicId, but
BumpLeaderEpochsReponse uses TopicName.
Yes, we have made:
1. BumpLeaderEpochsRequest/BumpLeaderEpochsResponse use TopicName.
2. DescribeClusterMirrorRequest/DescribeClusterMirrorResponse use TopicName.
3. MirrorTopicStateChangeRecord uses TopicName because we will do
validation before writing the metadata log.
> JR69. MIRROR_PID_RESET:
> JR69.1 Could you specify the value of the control record's type?
It’s MIRROR_PID_RESET(7). KIP updated.
> JR69.2 How is the SourceClusterId field used?
I’m also surprised to see this field in the MirrorPidResetRecord. I traced
back to the internal PR we added for this field, and confirmed it is not
used since it was added. Sorry for the confusion and nice catch! KIP
updated.
> JR70. MirrorPartitionState: type should be "data" not "coordinator-key",
right? Ditto for LastMirrorEpochs.
We store both MirrorPartitionState and LastMirrorEpochs records into
internal __mirror_state topic, and rely on Coordinator Runtime to
write/read the records, like how `OffsetCommit` and `GroupMetadata` work.
So I think we should keep them as "coordinator-key"/"coordinator-value"
type.
Thank you,
Luke
On Tue, Aug 11, 2026 at 7:02 AM Jun Rao via dev <de...@kafka.apache.org>
wrote:
> Hi, Luke,
>
> Thanks for the reply.
>
> JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election is
> enabled and the destination leader has log truncation, the state will move
> from MIRRORING to this state."
> This still suggests that divergeEpoch is only handled if
> mirror.support.unclean.leader.election
> is enabled.
>
> JR55. Should the mirror state and the mirror lags always be exposed
> together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
> from DescribeClusterMirrorsRequest. If not, it's better to use two separate
> RPCs.
>
> DescribeClusterMirrorsRequest should be used in the Describe Mirror flow,
> right? Currently it's missing. Also, I assume the admin client refers to
> the one in the kafka-cluster-mirrors tool. But it seems that sometimes it
> also refers to the admin client on the broker? Perhaps we can use 'the
> tool' instead to make it clear.
>
> JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
> instead of just to the controller?
>
> JR59. Start Mirror Topics:
> JR59.1 "During this state, the broker queries the source cluster for last
> mirror epochs".
> Could you describe which RPC is issued to find the LME?
> JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request doesn't
> contain a replication factor.
> JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
> different topic name"
> Should we also return this error when the topic name is used as a regular
> topic?
>
> JR60. CreateClusterMirrorRequest: When will it receive an
> INVALID_CLUSTER_MIRROR_STATES error not covered by
> CLUSTER_MIRROR_ALREADY_EXISTS?
>
> JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just remove
> MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
> CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
> specific state causing the error.
>
> JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some requests
> (e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics). Could we
> make them consistent?
>
> JR63. StopMirrorTopics:
> JR63.1 "The broker validates that all target topic partitions are in either
> LOG_TRUNCATION or MIRRORING or FAILED state."
> Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
> JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since PAUSED
> -> STOPPED is valid?
> JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped state"
> This is weird. The reason to issue StopMirrorTopics is that it's not in
> stopped state.
>
> JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's derived
> from SourceOffset and DestinationOffset.
>
> JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
> could just set the partition level state to STOPPED.
>
> JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent to
> the controller?
>
> JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch? It
> seems both are used for fencing and fencing based on the latter is
> stronger.
>
> JR68. Could we make the usage of topicId and topicName in the RPCs
> consistent? Also, BumpLeaderEpochsRequest uses topicId, but
> BumpLeaderEpochsReponse uses TopicName.
>
> JR69. MIRROR_PID_RESET:
> JR69.1 Could you specify the value of the control record's type?
> JR69.2 How is the SourceClusterId field used?
>
> JR70. MirrorPartitionState: type should be "data" not "coordinator-key",
> right? Ditto for LastMirrorEpochs.
>
> Jun
>
> On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the comments.
> > Answering them below.
> >
> > > JR36. Sounds good. Should we remove MirrorLeaderEpochExceededException?
> >
> > Sorry, it’s a confusing name. The MirrorLeaderEpochExceededException is
> > used and thrown when the fetched mirror batch leader epoch > local leader
> > epoch in destination cluster. And then we’ll stop mirroring and move the
> > partition to EPOCH_FENSING state to wait for local leader epoch bump. I
> > think since we don’t have the concept of `MirrorLeaderEpoch` anymore, the
> > `MirrorLeaderEpochExceededException` should not cause confusion now. I’ll
> > keep the name for the exception.
> >
> >
> > > JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
> > mirror
> > fetch request issued to the source cluster, independent of the value of
> > mirror.support.unclean.leader.election? It's needed to detect divergence
> > during failover. However, if it's also set when
> > mirror.support.unclean.leader.election=false,
> > an unclean leader election in the source cluster can halt the mirroring
> > since we don’t do anything with the returned divergeEpoch in that mode
> > and continue
> > the next fetch.
> >
> > > Do we plan to always set lastFetchedEpoch in the mirror
> > fetch request issued to the source cluster, independent of the value of
> > Mirror.support.unclean.leader.election?
> >
> > Yes, we always set lastFetchedEpoch in the mirror fetch request.
> >
> > > However, if it's also set when
> > mirror.support.unclean.leader.election=false,
> > an unclean leader election in the source cluster can halt the mirroring
> > since we don’t do anything with the returned divergeEpoch in that mode
> > and continue the next fetch.
> >
> > Sorry I didn’t make it clear. When
> > “mirror.support.unclean.leader.election=false” and a “divergeEpoch” is
> set
> > in fetch response, the leader node will handle the divergeEpoch and then
> > continue the next fetch as usual. The difference with
> > “mirror.support.unclean.leader.election=true” is that “should we wait for
> > ALL replicas caught up (complete truncation)”. Without the waiting, it’s
> > possible the inconsistent log issue in KAFKA-18723 could happen between
> > leader and follower nodes in the destination cluster because we don’t
> have
> > “MirrorLeaderEpoch” concept now to reject higher leader epoch batches.
> That
> > is, we won't halt the mirroring if "divergeEpoch" is set but
> > “mirror.support.unclean.leader.election=false”.
> >
> >
> > > JR55. The DescribeClusterMirrorsRequest still returns both the mirror
> > partition state and the mirror partition leader state. The problem is
> that
> > if a caller only needs one of the state, the other state becomes
> redundant.
> > For example, when starting the mirror, to obtain LME, we only need the
> > mirror partition state. Here is an alternative design: We expose the
> mirror
> > partition state through ReadMirrorState. The client will use the
> > FindCoordinatorRequest
> > to find the right mirror partition coordinator to issue that request. We
> > expose the mirror partition leader state through the
> ReadClusterMirrorLags
> > request. The client can use the metadata request to find the partition
> > leader to issue that request.
> >
> > > In the describing mirror flow, the admin client first issues the
> > ListMirrorRequest to find all the mirrors and the topics being mirrored.
> It
> > then issues the metadata request to find the partitions for those topics
> > and issues a FindCoordinatorRequest to find the mirror partition state
> > coordinators. Finally, it issues ReadMirrorState to the mirror partition
> > state coordinator and ReadClusterMirrorLags to the partition leader.
> >
> > > In the starting mirror flow, the partition leader in the destination
> > cluster issues a ListMirrorRequest to find the mirror name based on its
> > cluster id. It then issues a FindCoordinatorRequest to find the mirror
> > partition state coordinators. Finally, it issues ReadMirrorState to the
> > mirror partition state coordinator to get the LME.
> >
> > > We can also improve the ListMirror request to accept a filter based on
> > mirror name, source cluster ID and topic name.
> >
> >
> > Answers:
> >
> > > The DescribeClusterMirrorsRequest still returns both the mirror
> > partition state and the mirror partition leader state. The problem is
> that
> > if a caller only needs one of the state, the other state becomes
> redundant.
> > For example, when starting the mirror, to obtain LME, we only need the
> > mirror partition state.
> >
> > Good point!
> >
> >
> > > Here is an alternative design: We expose the mirror
> > partition state through ReadMirrorState. The client will use the
> > FindCoordinatorRequest
> > to find the right mirror partition coordinator to issue that request. We
> > expose the mirror partition leader state through the
> ReadClusterMirrorLags
> > request. The client can use the metadata request to find the partition
> > leader to issue that request.
> >
> >
> > Sounds good. But we try not to expose
> > ReadMirrorState/ReadClusterMirrorLags directly to the admin client
> because
> > users might get confused about these 2 requests, and they are also both
> > related to describeClusterMirror. I think we can add 2 additional fields
> to
> > describeClusterMirrorRequest: “includeMirrorStates” and
> > “includeMirrorLags”. So the admin client can include either one or both
> of
> > them in the DescrbieClusterMirrorRequest based on the provided options.
> KIP
> > is updated.
> >
> >
> > > In the describing mirror flow, the admin client first issues the
> > ListMirrorRequest to find all the mirrors and the topics being mirrored.
> It
> > then issues the metadata request to find the partitions for those topics
> > and issues a FindCoordinatorRequest to find the mirror partition state
> > coordinators. Finally, it issues ReadMirrorState to the mirror partition
> > state coordinator and ReadClusterMirrorLags to the partition leader.
> >
> > Makes sense. KIP is updated.
> >
> >
> > > In the starting mirror flow, the partition leader in the destination
> > cluster issues a ListMirrorRequest to find the mirror name based on its
> > cluster id. It then issues a FindCoordinatorRequest to find the mirror
> > partition state coordinators. Finally, it issues ReadMirrorState to the
> > mirror partition state coordinator to get the LME.
> >
> > Make sense. So we can send DescribeClusterMirrorRequest with
> > “includeMirrorStates” only.
> >
> > > We can also improve the ListMirror request to accept a filter based on
> > mirror name, source cluster ID and topic name.
> >
> > Good suggestion, just like what ListGroupsRequest does.
> > Added "SourceClusterIdFilter", "MirrorNameFilter",
> > "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP updated.
> >
> >
> > > JR56. Could you list the ACL needed for each new RPC?
> >
> >
> > Yes, it’s already listed in “Security Control” section.
> >
> >
> > > JR57. In the command workflow, it sometimes says "The user sends" and
> > some
> > other times says "The admin client sends". It's better to use "The admin
> > client" consistently in all those places.
> >
> > Agree. KIP updated.
> >
> >
> > Thank you,
> > Luke
> >
> > On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Luke,
> >>
> >> Thanks for the reply.
> >>
> >> JR36. Sounds good. Should we remove MirrorLeaderEpochExceededException?
> >>
> >> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
> mirror
> >> fetch request issued to the source cluster, independent of the value of
> >> mirror.support.unclean.leader.election? It's needed to detect divergence
> >> during failover. However, if it's also set when
> >> mirror.support.unclean.leader.election=false,
> >> an unclean leader election in the source cluster can halt the mirroring
> >> since we don’t do anything with the returned divergeEpoch in that mode
> >> and continue
> >> the next fetch.
> >>
> >> JR55. The DescribeClusterMirrorsRequest still returns both the mirror
> >> partition state and the mirror partition leader state. The problem is
> that
> >> if a caller only needs one of the state, the other state becomes
> >> redundant.
> >> For example, when starting the mirror, to obtain LME, we only need the
> >> mirror partition state. Here is an alternative design: We expose the
> >> mirror
> >> partition state through ReadMirrorState. The client will use the
> >> FindCoordinatorRequest
> >> to find the right mirror partition coordinator to issue that request. We
> >> expose the mirror partition leader state through the
> ReadClusterMirrorLags
> >> request. The client can use the metadata request to find the partition
> >> leader to issue that request.
> >>
> >> In the describing mirror flow, the admin client first issues the
> >> ListMirrorRequest to find all the mirrors and the topics being mirrored.
> >> It
> >> then issues the metadata request to find the partitions for those topics
> >> and issues a FindCoordinatorRequest to find the mirror partition state
> >> coordinators. Finally, it issues ReadMirrorState to the mirror partition
> >> state coordinator and ReadClusterMirrorLags to the partition leader.
> >>
> >> In the starting mirror flow, the partition leader in the destination
> >> cluster issues a ListMirrorRequest to find the mirror name based on its
> >> cluster id. It then issues a FindCoordinatorRequest to find the mirror
> >> partition state coordinators. Finally, it issues ReadMirrorState to the
> >> mirror partition state coordinator to get the LME.
> >>
> >> We can also improve the ListMirror request to accept a filter based on
> >> mirror name, source cluster ID and topic name.
> >>
> >> JR56. Could you list the ACL needed for each new RPC?
> >>
> >> JR57. In the command workflow, it sometimes says "The user sends" and
> some
> >> other times says "The admin client sends". It's better to use "The admin
> >> client" consistently in all those places.
> >>
> >> Jun
> >>
> >>
> >>
> >> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thanks for your comments.
> >> > Answering them below.
> >> >
> >> >
> >> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is
> specifically
> >> > designed for the unclean leader election support in Cluster Mirroring
> >> > because the unclean leader election will break the assumption that the
> >> > committed data won’t be truncated."
> >> > If the source cluster has an unclean leader election, the destination
> >> > cluster will detect the inconsistency and truncate the log at the
> >> > destination leader. We can just
> >> > extend mirror.support.unclean.leader.election to cover this case,
> i.e.,
> >> we
> >> > wait until all replicas' log end offsets are reduced to the target
> >> offset
> >> > before resuming mirroring. This avoids the inconsistency.
> >> >
> >> > If the destination cluster has an unclean leader election, it will
> >> > re-fetch the same data from the source. There won't be any
> inconsistency
> >> > either.
> >> > So, I think we can avoid adding MirrorLeaderEpoch.
> >> >
> >> >
> >> > Yes, that works, too. I agree this is better because the unclean
> leader
> >> > election is rare, we don't want to occupy a field just for it.
> >> >
> >> > So to achieve this, we have to:
> >> > 1. add one more state: “ULE_LOG_TRUNCATION”,
> >> > 2. When the destination cluster leader completes a log truncation
> after
> >> a
> >> > fetch response due to divergeEpoch, we know it must be caused by
> unclean
> >> > leader election in the source cluster because we only fetch committed
> >> > records. If “mirror.support.unclean.leader.election=false”, we don’t
> do
> >> > anything and continue the next fetch. If
> >> > “mirror.support.unclean.leader.election=true”, we remove the mirror
> >> fetcher
> >> > and move the partition to “ULE_LOG_TRUNCATION” state.
> >> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be
> caught
> >> up
> >> > into ISR and reduced to the expected LEO.
> >> > 4. Move to “MIRRORING” state and create the mirror fetcher
> >> >
> >> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch API.
> >> > KIP is updated.
> >> >
> >> >
> >> > > JR51. Make sense. To truncate based on LME, we need to issue another
> >> RPC
> >> > to translate the LME to an offset. It would be useful to include that
> in
> >> > the workflow description.
> >> >
> >> > About the additional RPC to translate the LME to the offset (i.e.
> >> > OffsetForLeaderEpochRequest), actually we don’t rely on this to do the
> >> > offset translation. We use the local leader epoch cache in the
> >> destination
> >> > cluster to translate the epoch to save the RPC call. This works
> because
> >> of
> >> > an invariant the LME truncation protocol maintains: for any leader
> >> epoch ≤
> >> > LME, that epoch's data belongs to exactly one cluster, either it
> >> originated
> >> > there, or it was mirrored. This holds even across repeated mirror ->
> >> > reverse-mirror -> mirror cycles, since each direction's LOG_TRUNCATION
> >> > phase re-establishes the invariant before that cluster starts
> mirroring.
> >> > With this invariant, the replication protocol can find out the diverge
> >> > point and truncate the log correctly when mirroring. For example:
> >> >
> >> > Cluster A (source) --> Cluster B (dest)
> >> > foo-0 leader log foo-0 leader log
> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> > Offset 2: A (LE: 0)
> >> > Offset 3: A (LE: 1)
> >> >
> >> > Cluster A crashes while Cluster B mirrors to offset 1. Failover to B
> and
> >> > store the LME=0 in cluster B. When cluster A recovers and starts the
> >> > reverse mirroring, it’ll query the LME from B and get back LME: 0.
> Then
> >> > cluster A gets the end offset for epoch 0 from leaderEpochCache, and
> >> gets
> >> > offset 2. So offset 3 is truncated. The log is like this now:
> >> >
> >> > Cluster A (dest) <-- Cluster B (source)
> >> > foo-0 leader log foo-0 leader log
> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
> >> > Offset 3: B (LE: 10)
> >> >
> >> > When mirror starts, cluster A sends the fetch request with {
> >> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the divergingEpoch
> check
> >> in
> >> > the cluster B queries the endOffset for leader epoch 0, and found the
> >> end
> >> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns fetch
> >> > response with divergingEpoch info to cluster A for log truncation. In
> >> the
> >> > end, offset 2 in cluster A will be truncated and then log converges
> with
> >> > cluster B.
> >> >
> >> >
> >> >
> >> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
> >> > because it mixes querying the mirror partition state with querying the
> >> > mirror partition leader. Another option is to use
> >> ReadMirrorStatesRequest
> >> > to query the mirror partition state and a new request to query the
> >> mirror
> >> > partition leader. If we want the adminClient to query the right
> >> ClusterMirrorCoordinator
> >> > directly, we can extend the FindCoordinatorRequest to support that.
> >> >
> >> >
> >> > Good suggestion. Yes, we need to fan out to all brokers because we
> have
> >> to
> >> > collect info from mirror partition leaders and mirror coordinators. If
> >> we
> >> > create a new API to contact mirror partition leaders, it will make the
> >> API
> >> > cleaner and clearer. So we will do this:
> >> >
> >> > a. Remove SourceOffset, DestinationOffset, and Lag fields from
> >> > DescribeClusterMirror request/response
> >> >
> >> > b. Add a new API: ReadClusterMirrorLags API to request for
> SourceOffset,
> >> > DestinationOffset, and Lag info.
> >> >
> >> > The workflow for admin client describeClusterMirror will be like this:
> >> > 1. adminClient sends describeClusterMirror to the leastLoadedNode
> broker
> >> > 2. The received broker collects the current mirrored partitions, and
> >> > groups them by coordinator node and by partition leader node.
> >> > 3. The broker sends ReadMirrorState request to the coordinator nodes
> to
> >> > get the mirror state related info
> >> > 4. The broker sends ReadClusterMirrorLags request to the mirror
> >> partition
> >> > leader nodes to get the mirror lag, source offset, destination offset
> >> > (which updates on every fetch response).
> >> > 5. The broker merge all the response into describeClusterMirror
> response
> >> > and send to admin client
> >> >
> >> > KIP is updated. Thanks for the suggestion.
> >> >
> >> >
> >> > > JR55. For __mirror_state topic, could we describe whether it is
> >> > compacted?
> >> >
> >> > Yes, it is a compacted topic. I made it much clearer in the
> >> > “ClusterMirrorCoordinator” section.
> >> >
> >> >
> >> > Thank you,
> >> > Luke
> >> >
> >> >
> >> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <de...@kafka.apache.org>
> >> > wrote:
> >> >
> >> >> Hi, Luke,
> >> >>
> >> >> Thanks for the reply.
> >> >>
> >> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
> >> >> designed for the unclean leader election support in Cluster Mirroring
> >> >> because the unclean leader election will break the assumption that
> the
> >> >> committed data won’t be truncated."
> >> >> If the source cluster has an unclean leader election, the destination
> >> >> cluster will detect the inconsistency and truncate the log at the
> >> >> destination leader. We can just
> >> >> extend mirror.support.unclean.leader.election to cover this case,
> >> i.e., we
> >> >> wait until all replicas' log end offsets are reduced to the target
> >> offset
> >> >> before resuming mirroring. This avoids the inconsistency.
> >> >>
> >> >> If the destination cluster has an unclean leader election, it will
> >> >> re-fetch
> >> >> the same data from the source. There won't be any inconsistency
> either.
> >> >>
> >> >> So, I think we can avoid adding MirrorLeaderEpoch.
> >> >>
> >> >> JR51. Make sense. To truncate based on LME, we need to issue another
> >> RPC
> >> >> to
> >> >> translate the LME to an offset. It would be useful to include that in
> >> the
> >> >> workflow description.
> >> >>
> >> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
> >> because
> >> >> it mixes querying the mirror partition state with querying the mirror
> >> >> partition leader. Another option is to use ReadMirrorStatesRequest to
> >> >> query
> >> >> the mirror partition state and a new request to query the mirror
> >> partition
> >> >> leader. If we want the adminClient to query the right
> >> >> ClusterMirrorCoordinator
> >> >> directly, we can extend the FindCoordinatorRequest to support that.
> >> >>
> >> >> JR55. For __mirror_state topic, could we describe whether it is
> >> compacted?
> >> >>
> >> >> Jun
> >> >>
> >> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com> wrote:
> >> >>
> >> >> > Hi Jun,
> >> >> >
> >> >> > Thanks for the comments.
> >> >> > Answering them below:
> >> >> >
> >> >> > > JR31.3 It makes sense for the new partition leader to query the
> >> >> current
> >> >> > mirror state from the local or remote MirrorMetadataManager that
> owns
> >> >> the
> >> >> > partition's state. However, the wording in the KIP is confusing.
> >> >> > MirrorMetadataManager
> >> >> > mirror partition state cache should be populated only by one way,
> >> which
> >> >> is
> >> >> > replaying __mirror_state partitions this broker leads. By saying
> >> "and by
> >> >> > fetching state from remote coordinators via ReadMirrorStates
> RPCs.",
> >> >> it's
> >> >> > as if the mirror partition state cache on each broker stores the
> >> state
> >> >> for
> >> >> > all mirror partitions.
> >> >> >
> >> >> > You’re right. KIP is updated to make it clear.
> >> >> >
> >> >> >
> >> >> > > JR36. I understand the intention now, but I still don't think we
> >> need
> >> >> > MirrorLeaderEpoch. Let's
> >> >> > assume the common case: no unclean leader election in the source
> >> cluster
> >> >> > during mirroring. Mirroring only fetches the committed data from
> the
> >> >> > source, which never changes if obtained a second time. In your
> >> example
> >> >> > image, follower broker 1 in the destination cluster can never
> obtain
> >> the
> >> >> > record B(0) because it's never committed in the source and thus is
> >> never
> >> >> > exposed to the destination leader. It's possible for broker 1 to
> >> first
> >> >> get
> >> >> > D(1) which is truncated later when broker 0 follows a new leader.
> But
> >> >> the
> >> >> > new leader will refetch the same D(1) from the source cluster. So,
> >> it's
> >> >> ok
> >> >> > for broker 1 to keep D(1).
> >> >> >
> >> >> >
> >> >> > Yes, you are correct. Because mirroring only fetches the committed
> >> data
> >> >> > from the source, the MirrorLeaderEpoch is unnecessary for normal
> >> >> mirroring.
> >> >> > Thanks for the clarification. So I think we can say, the
> >> >> > “MirrorLeaderEpoch” is specifically designed for the unclean leader
> >> >> > election support in Cluster Mirroring because the unclean leader
> >> >> election
> >> >> > will break the assumption that the committed data won’t be
> truncated.
> >> >> >
> >> >> > Given that, we could gate the MirrorLeaderEpoch field by only
> >> setting it
> >> >> > in the Fetch request when
> >> "mirror.support.unclean.leader.election=true"
> >> >> > (false by default), and skip it entirely for normal mirroring. What
> >> do
> >> >> you
> >> >> > think?
> >> >> >
> >> >> >
> >> >> > > JR51. Sounds good. Why do we choose to store the last mirror
> epoch
> >> >> > instead
> >> >> > of a last mirror offset?
> >> >> >
> >> >> > Yes, originally we stored the “last mirror offset” instead of “last
> >> >> mirror
> >> >> > epoch”. But later, we found it doesn’t work for unclean leader
> >> election
> >> >> > cases. (yes, `unclean leader election` again :) ) For example:
> >> >> >
> >> >> > Cluster A (source) --> Cluster B (dest)
> >> >> > foo-0 leader log foo-0 leader log
> >> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >> >
> >> >> > Cluster B is mirroring from A. And A crashes and failover to B. B
> >> stores
> >> >> > the “last mirror offset” as 1.
> >> >> > When cluster A recovers, somehow the unclean leader election
> happens,
> >> >> log
> >> >> > is empty in the new leader, and then some more leadership changes
> to
> >> >> make
> >> >> > the leader epoch become 10. Then it starts the reverse mirror from
> B.
> >> >> The
> >> >> > log looks like this:
> >> >> >
> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> > foo-0 leader log foo-0 leader log
> >> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> >> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> >> >> > Offset 2: pid_res (LE:
> >> 10)
> >> >> >
> >> >> > When A starts mirroring from B, log is truncated to last mirror
> >> offset:
> >> >> 1
> >> >> > (no-op), and then fetch from cluster B. The 1st fetch request is
> like
> >> >> this:
> >> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2, …
> }.
> >> The
> >> >> > divergingEpoch check in the cluster B queries the endOffset for
> >> leader
> >> >> > epoch 10 (in Partition#readRecords), and found the end offset for
> LE
> >> 10
> >> >> is
> >> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch validation
> >> passes
> >> >> and
> >> >> > returns offset 2 to cluster A. In short, relying on “last mirror
> >> offset”
> >> >> > cannot resolve the log diverge issue in unclean leader election
> case.
> >> >> >
> >> >> > If instead, we truncate until “last mirror epoch”: 0, the offset 0
> >> and 1
> >> >> > in cluster A will be truncated as expected and then log converge
> with
> >> >> > cluster B correctly in the end.
> >> >> >
> >> >> >
> >> >> > Also, about JR53.2:
> >> >> > > JR53.2 "2. The admin client fans out the request to all brokers
> and
> >> >> > merges
> >> >> > the responses client-side."
> >> >> > This is not ideal. Not every broker hosts a mirror partition. So
> >> sending
> >> >> > the request to every broker is wasteful. It's probably better to
> send
> >> >> the
> >> >> > request to a single broker, which can then contact the right
> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >
> >> >> > While working on this, I found we need to add one more tagged field
> >> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema because
> >> >> otherwise
> >> >> > the receiving broker doesn’t know if this is the request from
> client
> >> >> and it
> >> >> > needs to contact other brokers, or this is just a forwarded
> request.
> >> The
> >> >> > workflow is like this:
> >> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest to a
> >> >> > leastLoaded broker, which doesn’t contain the "Forwarded" field.
> >> >> > 2. The broker receives the request and sends
> >> >> DescribeClusterMirrorsRequest
> >> >> > with "Forwarded=true" to the responsible brokers and waits for the
> >> >> response
> >> >> > async.
> >> >> > 3. When all responses are received, merge the results and respond
> to
> >> the
> >> >> > client.
> >> >> >
> >> >> > Note: we cannot use ReadMirrorStatesRequest here because it doesn’t
> >> >> > contain the lag info in the coordinator.
> >> >> >
> >> >> > The alternative solution is the adminClient sends
> >> >> > DescribeClusterMirrorsRequest to the “responsible brokers only”
> (via
> >> >> > describeTopics), instead of ALL brokers, and merge the results in
> the
> >> >> > client side. This solution doesn’t need to add a new field in the
> >> >> > DescribeClusterMirrorsRequest and still can avoid unnecessary
> >> requests
> >> >> > sent.
> >> >> >
> >> >> > My preference is to keep the merge logic on the admin client side,
> >> which
> >> >> > saves the overhead in the broker side.
> >> >> > What do you think?
> >> >> >
> >> >> >
> >> >> > Thank you,
> >> >> > Luke
> >> >> >
> >> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <
> dev@kafka.apache.org
> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> Hi, Luke,
> >> >> >>
> >> >> >> Thanks for the reply.
> >> >> >>
> >> >> >> JR31.3 It makes sense for the new partition leader to query the
> >> current
> >> >> >> mirror state from the local or remote MirrorMetadataManager that
> >> owns
> >> >> the
> >> >> >> partition's state. However, the wording in the KIP is confusing.
> >> >> >> MirrorMetadataManager
> >> >> >> mirror partition state cache should be populated only by one way,
> >> >> which is
> >> >> >> replaying __mirror_state partitions this broker leads. By saying
> >> "and
> >> >> by
> >> >> >> fetching state from remote coordinators via ReadMirrorStates
> RPCs.",
> >> >> it's
> >> >> >> as if the mirror partition state cache on each broker stores the
> >> state
> >> >> for
> >> >> >> all mirror partitions.
> >> >> >>
> >> >> >> JR36. I understand the intention now, but I still don't think we
> >> need
> >> >> >> MirrorLeaderEpoch. Let's
> >> >> >> assume the common case: no unclean leader election in the source
> >> >> cluster
> >> >> >> during mirroring. Mirroring only fetches the committed data from
> the
> >> >> >> source, which never changes if obtained a second time. In your
> >> example
> >> >> >> image, follower broker 1 in the destination cluster can never
> obtain
> >> >> the
> >> >> >> record B(0) because it's never committed in the source and thus is
> >> >> never
> >> >> >> exposed to the destination leader. It's possible for broker 1 to
> >> first
> >> >> get
> >> >> >> D(1) which is truncated later when broker 0 follows a new leader.
> >> But
> >> >> the
> >> >> >> new leader will refetch the same D(1) from the source cluster. So,
> >> >> it's ok
> >> >> >> for broker 1 to keep D(1).
> >> >> >>
> >> >> >> JR51. Sounds good. Why do we choose to store the last mirror epoch
> >> >> instead
> >> >> >> of a last mirror offset?
> >> >> >>
> >> >> >> Jun
> >> >> >>
> >> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com>
> wrote:
> >> >> >>
> >> >> >> > Hi Jun,
> >> >> >> >
> >> >> >> > Thanks for the great comments again.
> >> >> >> > Answering your question below.
> >> >> >> >
> >> >> >> > > JR31.3 "Regarding state caching: the mirror partition cache
> >> lives
> >> >> in
> >> >> >> > MirrorMetadataManager. It is populated in two ways: by replaying
> >> >> >> > __mirror_state partitions this broker leads, and by fetching
> state
> >> >> >> > from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> > Hmm, a given broker only caches the mirror partition state for
> >> >> >> partitions
> >> >> >> > hashed into the __mirror_state partition hosted on this broker,
> >> >> right?
> >> >> >> Why
> >> >> >> > does it need to fetch the state from other remote coordinators?
> >> >> >> >
> >> >> >> > > a given broker only caches the mirror partition state for
> >> >> partitions
> >> >> >> > hashed into the __mirror_state partition hosted on this broker,
> >> >> right?
> >> >> >> >
> >> >> >> > Right.
> >> >> >> >
> >> >> >> > > Why does it need to fetch the state from other remote
> >> coordinators?
> >> >> >> >
> >> >> >> > Because the data partition leader needs to know the current
> mirror
> >> >> state
> >> >> >> > of the partitions it leads. On a leadership change, the metadata
> >> >> delta
> >> >> >> is
> >> >> >> > published to the MirrorMetadataManager, which queries the
> current
> >> >> mirror
> >> >> >> > state for its led partitions from the coordinator (local or
> >> remote),
> >> >> and
> >> >> >> > processes tasks based on that state. This is the reason why it
> >> >> fetches
> >> >> >> the
> >> >> >> > state from other remote coordinator. This part of cache will be
> >> >> cleared
> >> >> >> > after the leadership change.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR36. I am still not quite sure why we need MirrorLeaderEpoch
> in
> >> >> the
> >> >> >> > Fetch
> >> >> >> > request. In the current Fetch request, in addition to
> >> >> >> CurrentLeaderEpoch,
> >> >> >> > it has another field LastFetchedEpoch, which captures the last
> >> leader
> >> >> >> epoch
> >> >> >> > in the data and is used to resolve the inconsistency between the
> >> >> >> follower
> >> >> >> > and the leader. Does it achieve the same purpose as
> >> >> MirrorLeaderEpoch?
> >> >> >> >
> >> >> >> >
> >> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to do the
> >> >> validation
> >> >> >> to
> >> >> >> > avoid the issue in KAFKA-18723. If we reject any fetched batches
> >> with
> >> >> >> epoch
> >> >> >> > higher than the LastFetchedEpoch, it means the follower can
> never
> >> >> accept
> >> >> >> > new appended batches with higher epoch in the leader node.
> >> >> >> >
> >> >> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes in the
> >> >> fetch
> >> >> >> > API:
> >> >> >> > 1. The receiver (leader node) uses it to validate it matches the
> >> >> leader
> >> >> >> > epoch in the leader node
> >> >> >> > 2. The sender (follower node) uses it to validate the epoch in
> the
> >> >> >> fetched
> >> >> >> > batches are all <= this value.
> >> >> >> >
> >> >> >> > For purpose (1), the CurrentLeaderEpoch is still serving for
> >> >> followers
> >> >> >> in
> >> >> >> > the destination cluster. For purpose (2), my understanding of
> the
> >> >> >> > validation for KAFKA-18723 is like: the follower knows the
> >> >> >> > CurrentLeaderEpoch X is the highest epoch the leader node should
> >> >> >> contain in
> >> >> >> > the log batches when fetch request is sent. Any log batch beyond
> >> this
> >> >> >> epoch
> >> >> >> > in the fetch response means the follower’s metadata is stale.
> The
> >> >> >> batches
> >> >> >> > need to be rejected and follower needs to update metadata for
> next
> >> >> >> fetch.
> >> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves
> >> this
> >> >> >> purpose
> >> >> >> > to let the followers know the highest epoch in the leader node
> >> log in
> >> >> >> each
> >> >> >> > fetch response. Once the info is stale in the follower side, it
> >> >> updates
> >> >> >> it
> >> >> >> > from fetch response, and send next fetch request with correct
> >> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is possible
> >> like
> >> >> >> > KAFKA-18723.
> >> >> >> >
> >> >> >> > I tried not to add new field in Fetch API, but this is the best
> >> >> >> solution I
> >> >> >> > can think of.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR48. "The NumPartitions field is needed so the controller can
> >> >> create
> >> >> >> the
> >> >> >> > destination topic if it does not already exist. When a topic is
> >> being
> >> >> >> > mirrored for the first time, the destination cluster has no
> >> knowledge
> >> >> >> > of the source topic's partition count, so the request must carry
> >> it."
> >> >> >> > Hmm, this seems inconsistent. When a mirror is started, the
> source
> >> >> topic
> >> >> >> > may not exist yet. When the source topic is created later, we
> can
> >> >> start
> >> >> >> > mirroring without requiring the user to provide the number of
> >> >> >> partitions.
> >> >> >> >
> >> >> >> > > When the source topic is created later, we can start mirroring
> >> >> without
> >> >> >> > requiring the user to provide the number of partitions.
> >> >> >> >
> >> >> >> > This is because this happens in periodical sync up in the
> >> destination
> >> >> >> > cluster, and the numPartitions and other topic info in the
> source
> >> >> >> cluster
> >> >> >> > is already retrieved during the sync up process. Also, if users
> >> are
> >> >> >> > creating mirror topics via admin API, this info can also be
> >> retrieved
> >> >> >> > inside the admin client if not provided.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR51. Start Mirror: I am wondering if we truly need the last
> >> mirror
> >> >> >> epoch
> >> >> >> > for truncation when starting a mirror. When a mirror is stopped
> at
> >> >> the
> >> >> >> > source, we already bump up the leader epoch and write some
> >> additional
> >> >> >> > records for cleanup. It seems that we can just use the
> >> DivergingEpoch
> >> >> >> > returned in the initial mirror Fetch request to detect the
> >> diverging
> >> >> >> epoch
> >> >> >> > and use that for truncation at the mirror destination.
> >> >> >> >
> >> >> >> > That’s a good question. I thought about this before but
> >> >> unfortunately it
> >> >> >> > doesn’t work. The reason we need the last mirror epoch record is
> >> >> because
> >> >> >> > the destination cluster might not completely sync with the
> source
> >> >> >> cluster
> >> >> >> > before STOPPED. And this un-mirrored data might confuse the
> >> >> replication
> >> >> >> > protocol when reverse mirror. For example,
> >> >> >> >
> >> >> >> > Cluster A (source) —> Cluster B (dest)
> >> >> >> > foo-0 leader log foo-0 leader log
> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> >> > Offset 2: from A (LE: 10)
> >> >> >> >
> >> >> >> > While cluster B mirrors until offset 1, the cluster A crashes
> and
> >> >> >> failover
> >> >> >> > to cluster B. B bumps leader epoch to 10, appends a
> >> MIRROR_PID_RESET
> >> >> >> with
> >> >> >> > LE: 10. Then cluster A starts to reverse mirror from cluster B,
> >> the
> >> >> log
> >> >> >> > will be like this:
> >> >> >> >
> >> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> >> > foo-0 leader log: foo-0 leader log:
> >> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
> >> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
> >> >> >> >
> >> >> >> > As you can see, cluster A (new destination cluster) doesn’t know
> >> the
> >> >> >> > offset 2 is not belonging to the cluster B, which causes
> >> inconsistent
> >> >> >> logs.
> >> >> >> > So like what we described in the KIP, the last mirror epoch is
> the
> >> >> >> > synchronization point between source and destination. Any data
> >> beyond
> >> >> >> the
> >> >> >> > point is treated as unknown to the source cluster and should be
> >> >> >> truncated.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR52. Stop Mirror:
> >> >> >> > > JR52.1 "3. If patterns are provided, the controller removes
> >> >> matching
> >> >> >> > entries from mirror.topics.include or adds them to
> >> >> >> mirror.topics.exclude on
> >> >> >> > the CLUSTER_MIRROR resource."
> >> >> >> > Why is it 'or' and not 'and'?
> >> >> >> >
> >> >> >> > Yes, that should be AND. KIP updated. Thanks.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> >> MirrorMetadataManager
> >> >> >> > discovers new source topics matching the persisted
> include/exclude
> >> >> >> patterns
> >> >> >> > and repeats steps 3 through 7 for each."
> >> >> >> > The broker will send a create topic request to the controller.
> How
> >> >> does
> >> >> >> the
> >> >> >> > controller associate it with the mirror name to create a
> >> >> >> > `MirrorTopicStateChangeRecord`?
> >> >> >> >
> >> >> >> > No, we don’t send a create topic request to the controller here.
> >> >> >> Actually
> >> >> >> > we send a StartMirrorTopics request to the controller, which
> >> contains
> >> >> >> the
> >> >> >> > mirror name and topic metadata like NumPartitions, topicId… etc.
> >> So
> >> >> the
> >> >> >> > controller can create the non-existed topic as explained in
> JR48,
> >> and
> >> >> >> > associate it to MirrorTopicStateChangeRecord.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR53. Describe Mirrors:
> >> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors request
> with
> >> >> >> optional
> >> >> >> > mirror names (empty means all mirrors)."
> >> >> >> > In metadata request, we use a null arrary to represent all
> >> topics. It
> >> >> >> would
> >> >> >> > be useful to be consistent here.
> >> >> >> >
> >> >> >> > OK, KIP updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR53.2 "2. The admin client fans out the request to all
> brokers
> >> and
> >> >> >> > merges
> >> >> >> > the responses client-side."
> >> >> >> > This is not ideal. Not every broker hosts a mirror partition. So
> >> >> sending
> >> >> >> > the request to every broker is wasteful. It's probably better to
> >> send
> >> >> >> the
> >> >> >> > request to a single broker, which can then contact the right
> >> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >
> >> >> >> > Agree, KIP updated.
> >> >> >> >
> >> >> >> >
> >> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are
> MIRRORING,
> >> >> >> PAUSED,
> >> >> >> > STOPPED the only validate states? The state table has other
> states
> >> >> too.
> >> >> >> > Ditto for MirrorPartitionState.
> >> >> >> > The DesiredState field in MirrorTopicStateChangeRecord is to
> >> define
> >> >> the
> >> >> >> > target state the user wants. And the desired state users can
> >> assign
> >> >> is
> >> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as
> >> >> >> STOPPING
> >> >> >> > or LOG_TRUNCATION… etc because they are more like intermediate
> >> >> states.
> >> >> >> >
> >> >> >> > About MirrorPartitionState record, it’ll record any state in the
> >> >> state
> >> >> >> > table that the current partition is. This is the real state for
> >> each
> >> >> >> > partition. It’s like in the kubernetes world, when we change the
> >> >> replica
> >> >> >> > field of the deployment yaml from 3 -> 5 (desired state), it
> won’t
> >> >> >> increase
> >> >> >> > to 5 immediately. The current replica (MirrorPartitionState
> >> record)
> >> >> is
> >> >> >> > still 3, then 4 (maybe), and finally 5.
> >> >> >> >
> >> >> >> >
> >> >> >> > Thank you,
> >> >> >> > Luke
> >> >> >> >
> >> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
> >> dev@kafka.apache.org
> >> >> >
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Hi, Fede,
> >> >> >> >>
> >> >> >> >> Thanks for the reply. A few more comments.
> >> >> >> >>
> >> >> >> >> JR31.3 "Regarding state caching: the mirror partition cache
> >> lives in
> >> >> >> >> MirrorMetadataManager. It is populated in two ways: by
> replaying
> >> >> >> >> __mirror_state partitions this broker leads, and by fetching
> >> state
> >> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> >> Hmm, a given broker only caches the mirror partition state for
> >> >> >> partitions
> >> >> >> >> hashed into the __mirror_state partition hosted on this broker,
> >> >> right?
> >> >> >> Why
> >> >> >> >> does it need to fetch the state from other remote coordinators?
> >> >> >> >>
> >> >> >> >> JR36. I am still not quite sure why we need MirrorLeaderEpoch
> in
> >> the
> >> >> >> Fetch
> >> >> >> >> request. In the current Fetch request, in addition to
> >> >> >> CurrentLeaderEpoch,
> >> >> >> >> it has another field LastFetchedEpoch, which captures the last
> >> >> leader
> >> >> >> >> epoch
> >> >> >> >> in the data and is used to resolve the inconsistency between
> the
> >> >> >> follower
> >> >> >> >> and the leader. Does it achieve the same purpose as
> >> >> MirrorLeaderEpoch?
> >> >> >> >>
> >> >> >> >> JR48. "The NumPartitions field is needed so the controller can
> >> >> create
> >> >> >> the
> >> >> >> >> destination topic if it does not already exist. When a topic is
> >> >> being
> >> >> >> >> mirrored for the first time, the destination cluster has no
> >> >> knowledge
> >> >> >> >> of the source topic's partition count, so the request must
> carry
> >> >> it."
> >> >> >> >> Hmm, this seems inconsistent. When a mirror is started, the
> >> source
> >> >> >> topic
> >> >> >> >> may not exist yet. When the source topic is created later, we
> can
> >> >> start
> >> >> >> >> mirroring without requiring the user to provide the number of
> >> >> >> partitions.
> >> >> >> >>
> >> >> >> >> JR51. Start Mirror: I am wondering if we truly need the last
> >> mirror
> >> >> >> epoch
> >> >> >> >> for truncation when starting a mirror. When a mirror is stopped
> >> at
> >> >> the
> >> >> >> >> source, we already bump up the leader epoch and write some
> >> >> additional
> >> >> >> >> records for cleanup. It seems that we can just use the
> >> >> DivergingEpoch
> >> >> >> >> returned in the initial mirror Fetch request to detect the
> >> diverging
> >> >> >> epoch
> >> >> >> >> and use that for truncation at the mirror destination.
> >> >> >> >>
> >> >> >> >> JR52. Stop Mirror:
> >> >> >> >> JR52.1 "3. If patterns are provided, the controller removes
> >> matching
> >> >> >> >> entries from mirror.topics.include or adds them to
> >> >> >> mirror.topics.exclude
> >> >> >> >> on
> >> >> >> >> the CLUSTER_MIRROR resource."
> >> >> >> >> Why is it 'or' and not 'and'?
> >> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> MirrorMetadataManager
> >> >> >> >> discovers new source topics matching the persisted
> >> include/exclude
> >> >> >> >> patterns
> >> >> >> >> and repeats steps 3 through 7 for each."
> >> >> >> >> The broker will send a create topic request to the controller.
> >> How
> >> >> does
> >> >> >> >> the
> >> >> >> >> controller associate it with the mirror name to create a
> >> >> >> >> `MirrorTopicStateChangeRecord`?
> >> >> >> >>
> >> >> >> >> JR53. Describe Mirrors:
> >> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors request with
> >> >> >> optional
> >> >> >> >> mirror names (empty means all mirrors)."
> >> >> >> >> In metadata request, we use a null arrary to represent all
> >> topics.
> >> >> It
> >> >> >> >> would
> >> >> >> >> be useful to be consistent here.
> >> >> >> >> JR53.2 "2. The admin client fans out the request to all brokers
> >> and
> >> >> >> merges
> >> >> >> >> the responses client-side."
> >> >> >> >> This is not ideal. Not every broker hosts a mirror partition.
> So
> >> >> >> sending
> >> >> >> >> the request to every broker is wasteful. It's probably better
> to
> >> >> send
> >> >> >> the
> >> >> >> >> request to a single broker, which can then contact the right
> >> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >> >>
> >> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
> >> >> PAUSED,
> >> >> >> >> STOPPED the only validate states? The state table has other
> >> states
> >> >> too.
> >> >> >> >> Ditto for MirrorPartitionState.
> >> >> >> >>
> >> >> >> >> Jun
> >> >> >> >>
> >> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
> >> >> fedevaleri@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >> > Hi Andrew, yes, validation is called at the start of
> >> createTopic.
> >> >> If
> >> >> >> >> > the topic ID is invalid, the request fails with
> >> INVALID_REQUEST.
> >> >> If
> >> >> >> >> > the topic ID is already used by a different topic name, it
> >> fails
> >> >> with
> >> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual
> topic
> >> in
> >> >> the
> >> >> >> >> > batch fails; other topics in the same CreateTopics request
> >> >> continue
> >> >> >> >> > processing normally. Added this information to the KIP.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
> >> >> >> >> schofielaj@apache.org>
> >> >> >> >> > wrote:
> >> >> >> >> > >
> >> >> >> >> > > Hi Fede,
> >> >> >> >> > > One more small question.
> >> >> >> >> > >
> >> >> >> >> > > AS37: If the controller fails validation for the
> >> >> MirrorInfo.TopicId
> >> >> >> >> > added to the CreateTopic request, what does it do? I suspect
> >> that
> >> >> it
> >> >> >> >> fails
> >> >> >> >> > the request with a particular error code.
> >> >> >> >> > >
> >> >> >> >> > > Thanks,
> >> >> >> >> > > Andrew
> >> >> >> >> > >
> >> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> >> >> >> >> > > > Hi all,
> >> >> >> >> > > >
> >> >> >> >> > > > We updated the KIP to include Coordinator Runtime
> >> >> configurations
> >> >> >> and
> >> >> >> >> > > > the coordinator state transition validation paragraph.
> The
> >> >> latter
> >> >> >> >> > > > covers the per-partition epoch fencing mechanism (leader
> >> epoch
> >> >> >> and
> >> >> >> >> > > > state epoch) that complements the existing per-topic
> >> >> StateOffset
> >> >> >> >> > > > fencing for lifecycle operations.
> >> >> >> >> > > >
> >> >> >> >> > > > Thanks,
> >> >> >> >> > > > Fede
> >> >> >> >> > > >
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply.
JR51. "ULE_LOG_TRUNCATION When mirror.support.unclean.leader.election is
enabled and the destination leader has log truncation, the state will move
from MIRRORING to this state."
This still suggests that divergeEpoch is only handled if
mirror.support.unclean.leader.election
is enabled.
JR55. Should the mirror state and the mirror lags always be exposed
together? If so, we can remove IncludeMirrorState and IncludeMirrorLags
from DescribeClusterMirrorsRequest. If not, it's better to use two separate
RPCs.
DescribeClusterMirrorsRequest should be used in the Describe Mirror flow,
right? Currently it's missing. Also, I assume the admin client refers to
the one in the kafka-cluster-mirrors tool. But it seems that sometimes it
also refers to the admin client on the broker? Perhaps we can use 'the
tool' instead to make it clear.
JR58. BumpLeaderEpochs: Why does it need to be issued to the broker,
instead of just to the controller?
JR59. Start Mirror Topics:
JR59.1 "During this state, the broker queries the source cluster for last
mirror epochs".
Could you describe which RPC is issued to find the LME?
JR59.2 When does it receive INVALID_REPLICATION_FACTOR? The request doesn't
contain a replication factor.
JR59.3 TOPIC_ALREADY_EXISTS: "Request topic ID is already used by a
different topic name"
Should we also return this error when the topic name is used as a regular
topic?
JR60. CreateClusterMirrorRequest: When will it receive an
INVALID_CLUSTER_MIRROR_STATES error not covered by
CLUSTER_MIRROR_ALREADY_EXISTS?
JR61. Since we have INVALID_CLUSTER_MIRROR_STATES, should we just remove
MIRROR_TOPIC_NOT_STOPPED, MIRROR_TOPIC_NOT_PAUSED and
CLUSTER_MIRROR_NOT_EMPTY? We can use the error message to describe the
specific state causing the error.
JR62. NOT_CONTROLLER/UNKNOWN_CLUSTER_MIRROR are included in some requests
(e.g. StopMirrorTopics), but not others (e.g. StartMirrorTopics). Could we
make them consistent?
JR63. StopMirrorTopics:
JR63.1 "The broker validates that all target topic partitions are in either
LOG_TRUNCATION or MIRRORING or FAILED state."
Is this accurate? LOG_TRUNCATION can't transition to STOP. PAUSED can.
JR63.2 Why does it receive MIRROR_TOPIC_ALREADY_PAUSED error since PAUSED
-> STOPPED is valid?
JR63.3 MIRROR_TOPIC_NOT_STOPPED: "The mirror topic is not in stopped state"
This is weird. The reason to issue StopMirrorTopics is that it's not in
stopped state.
JR64. ReadClusterMirrorLagsResponse.lag is redundant since it's derived
from SourceOffset and DestinationOffset.
JR65. Why do we need WriteMirrorStates.StoppedTopics? It seems that we
could just set the partition level state to STOPPED.
JR66. ReadMirrorStates/WriteMirrorStates: Why do they need to be sent to
the controller?
JR67. WriteMirrorStates: Do we need both LeaderEpoch and StateEpoch? It
seems both are used for fencing and fencing based on the latter is stronger.
JR68. Could we make the usage of topicId and topicName in the RPCs
consistent? Also, BumpLeaderEpochsRequest uses topicId, but
BumpLeaderEpochsReponse uses TopicName.
JR69. MIRROR_PID_RESET:
JR69.1 Could you specify the value of the control record's type?
JR69.2 How is the SourceClusterId field used?
JR70. MirrorPartitionState: type should be "data" not "coordinator-key",
right? Ditto for LastMirrorEpochs.
Jun
On Mon, Aug 10, 2026 at 5:52 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the comments.
> Answering them below.
>
> > JR36. Sounds good. Should we remove MirrorLeaderEpochExceededException?
>
> Sorry, it’s a confusing name. The MirrorLeaderEpochExceededException is
> used and thrown when the fetched mirror batch leader epoch > local leader
> epoch in destination cluster. And then we’ll stop mirroring and move the
> partition to EPOCH_FENSING state to wait for local leader epoch bump. I
> think since we don’t have the concept of `MirrorLeaderEpoch` anymore, the
> `MirrorLeaderEpochExceededException` should not cause confusion now. I’ll
> keep the name for the exception.
>
>
> > JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the
> mirror
> fetch request issued to the source cluster, independent of the value of
> mirror.support.unclean.leader.election? It's needed to detect divergence
> during failover. However, if it's also set when
> mirror.support.unclean.leader.election=false,
> an unclean leader election in the source cluster can halt the mirroring
> since we don’t do anything with the returned divergeEpoch in that mode
> and continue
> the next fetch.
>
> > Do we plan to always set lastFetchedEpoch in the mirror
> fetch request issued to the source cluster, independent of the value of
> Mirror.support.unclean.leader.election?
>
> Yes, we always set lastFetchedEpoch in the mirror fetch request.
>
> > However, if it's also set when
> mirror.support.unclean.leader.election=false,
> an unclean leader election in the source cluster can halt the mirroring
> since we don’t do anything with the returned divergeEpoch in that mode
> and continue the next fetch.
>
> Sorry I didn’t make it clear. When
> “mirror.support.unclean.leader.election=false” and a “divergeEpoch” is set
> in fetch response, the leader node will handle the divergeEpoch and then
> continue the next fetch as usual. The difference with
> “mirror.support.unclean.leader.election=true” is that “should we wait for
> ALL replicas caught up (complete truncation)”. Without the waiting, it’s
> possible the inconsistent log issue in KAFKA-18723 could happen between
> leader and follower nodes in the destination cluster because we don’t have
> “MirrorLeaderEpoch” concept now to reject higher leader epoch batches. That
> is, we won't halt the mirroring if "divergeEpoch" is set but
> “mirror.support.unclean.leader.election=false”.
>
>
> > JR55. The DescribeClusterMirrorsRequest still returns both the mirror
> partition state and the mirror partition leader state. The problem is that
> if a caller only needs one of the state, the other state becomes redundant.
> For example, when starting the mirror, to obtain LME, we only need the
> mirror partition state. Here is an alternative design: We expose the mirror
> partition state through ReadMirrorState. The client will use the
> FindCoordinatorRequest
> to find the right mirror partition coordinator to issue that request. We
> expose the mirror partition leader state through the ReadClusterMirrorLags
> request. The client can use the metadata request to find the partition
> leader to issue that request.
>
> > In the describing mirror flow, the admin client first issues the
> ListMirrorRequest to find all the mirrors and the topics being mirrored. It
> then issues the metadata request to find the partitions for those topics
> and issues a FindCoordinatorRequest to find the mirror partition state
> coordinators. Finally, it issues ReadMirrorState to the mirror partition
> state coordinator and ReadClusterMirrorLags to the partition leader.
>
> > In the starting mirror flow, the partition leader in the destination
> cluster issues a ListMirrorRequest to find the mirror name based on its
> cluster id. It then issues a FindCoordinatorRequest to find the mirror
> partition state coordinators. Finally, it issues ReadMirrorState to the
> mirror partition state coordinator to get the LME.
>
> > We can also improve the ListMirror request to accept a filter based on
> mirror name, source cluster ID and topic name.
>
>
> Answers:
>
> > The DescribeClusterMirrorsRequest still returns both the mirror
> partition state and the mirror partition leader state. The problem is that
> if a caller only needs one of the state, the other state becomes redundant.
> For example, when starting the mirror, to obtain LME, we only need the
> mirror partition state.
>
> Good point!
>
>
> > Here is an alternative design: We expose the mirror
> partition state through ReadMirrorState. The client will use the
> FindCoordinatorRequest
> to find the right mirror partition coordinator to issue that request. We
> expose the mirror partition leader state through the ReadClusterMirrorLags
> request. The client can use the metadata request to find the partition
> leader to issue that request.
>
>
> Sounds good. But we try not to expose
> ReadMirrorState/ReadClusterMirrorLags directly to the admin client because
> users might get confused about these 2 requests, and they are also both
> related to describeClusterMirror. I think we can add 2 additional fields to
> describeClusterMirrorRequest: “includeMirrorStates” and
> “includeMirrorLags”. So the admin client can include either one or both of
> them in the DescrbieClusterMirrorRequest based on the provided options. KIP
> is updated.
>
>
> > In the describing mirror flow, the admin client first issues the
> ListMirrorRequest to find all the mirrors and the topics being mirrored. It
> then issues the metadata request to find the partitions for those topics
> and issues a FindCoordinatorRequest to find the mirror partition state
> coordinators. Finally, it issues ReadMirrorState to the mirror partition
> state coordinator and ReadClusterMirrorLags to the partition leader.
>
> Makes sense. KIP is updated.
>
>
> > In the starting mirror flow, the partition leader in the destination
> cluster issues a ListMirrorRequest to find the mirror name based on its
> cluster id. It then issues a FindCoordinatorRequest to find the mirror
> partition state coordinators. Finally, it issues ReadMirrorState to the
> mirror partition state coordinator to get the LME.
>
> Make sense. So we can send DescribeClusterMirrorRequest with
> “includeMirrorStates” only.
>
> > We can also improve the ListMirror request to accept a filter based on
> mirror name, source cluster ID and topic name.
>
> Good suggestion, just like what ListGroupsRequest does.
> Added "SourceClusterIdFilter", "MirrorNameFilter",
> "DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP updated.
>
>
> > JR56. Could you list the ACL needed for each new RPC?
>
>
> Yes, it’s already listed in “Security Control” section.
>
>
> > JR57. In the command workflow, it sometimes says "The user sends" and
> some
> other times says "The admin client sends". It's better to use "The admin
> client" consistently in all those places.
>
> Agree. KIP updated.
>
>
> Thank you,
> Luke
>
> On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Luke,
>>
>> Thanks for the reply.
>>
>> JR36. Sounds good. Should we remove MirrorLeaderEpochExceededException?
>>
>> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the mirror
>> fetch request issued to the source cluster, independent of the value of
>> mirror.support.unclean.leader.election? It's needed to detect divergence
>> during failover. However, if it's also set when
>> mirror.support.unclean.leader.election=false,
>> an unclean leader election in the source cluster can halt the mirroring
>> since we don’t do anything with the returned divergeEpoch in that mode
>> and continue
>> the next fetch.
>>
>> JR55. The DescribeClusterMirrorsRequest still returns both the mirror
>> partition state and the mirror partition leader state. The problem is that
>> if a caller only needs one of the state, the other state becomes
>> redundant.
>> For example, when starting the mirror, to obtain LME, we only need the
>> mirror partition state. Here is an alternative design: We expose the
>> mirror
>> partition state through ReadMirrorState. The client will use the
>> FindCoordinatorRequest
>> to find the right mirror partition coordinator to issue that request. We
>> expose the mirror partition leader state through the ReadClusterMirrorLags
>> request. The client can use the metadata request to find the partition
>> leader to issue that request.
>>
>> In the describing mirror flow, the admin client first issues the
>> ListMirrorRequest to find all the mirrors and the topics being mirrored.
>> It
>> then issues the metadata request to find the partitions for those topics
>> and issues a FindCoordinatorRequest to find the mirror partition state
>> coordinators. Finally, it issues ReadMirrorState to the mirror partition
>> state coordinator and ReadClusterMirrorLags to the partition leader.
>>
>> In the starting mirror flow, the partition leader in the destination
>> cluster issues a ListMirrorRequest to find the mirror name based on its
>> cluster id. It then issues a FindCoordinatorRequest to find the mirror
>> partition state coordinators. Finally, it issues ReadMirrorState to the
>> mirror partition state coordinator to get the LME.
>>
>> We can also improve the ListMirror request to accept a filter based on
>> mirror name, source cluster ID and topic name.
>>
>> JR56. Could you list the ACL needed for each new RPC?
>>
>> JR57. In the command workflow, it sometimes says "The user sends" and some
>> other times says "The admin client sends". It's better to use "The admin
>> client" consistently in all those places.
>>
>> Jun
>>
>>
>>
>> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for your comments.
>> > Answering them below.
>> >
>> >
>> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
>> > designed for the unclean leader election support in Cluster Mirroring
>> > because the unclean leader election will break the assumption that the
>> > committed data won’t be truncated."
>> > If the source cluster has an unclean leader election, the destination
>> > cluster will detect the inconsistency and truncate the log at the
>> > destination leader. We can just
>> > extend mirror.support.unclean.leader.election to cover this case, i.e.,
>> we
>> > wait until all replicas' log end offsets are reduced to the target
>> offset
>> > before resuming mirroring. This avoids the inconsistency.
>> >
>> > If the destination cluster has an unclean leader election, it will
>> > re-fetch the same data from the source. There won't be any inconsistency
>> > either.
>> > So, I think we can avoid adding MirrorLeaderEpoch.
>> >
>> >
>> > Yes, that works, too. I agree this is better because the unclean leader
>> > election is rare, we don't want to occupy a field just for it.
>> >
>> > So to achieve this, we have to:
>> > 1. add one more state: “ULE_LOG_TRUNCATION”,
>> > 2. When the destination cluster leader completes a log truncation after
>> a
>> > fetch response due to divergeEpoch, we know it must be caused by unclean
>> > leader election in the source cluster because we only fetch committed
>> > records. If “mirror.support.unclean.leader.election=false”, we don’t do
>> > anything and continue the next fetch. If
>> > “mirror.support.unclean.leader.election=true”, we remove the mirror
>> fetcher
>> > and move the partition to “ULE_LOG_TRUNCATION” state.
>> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be caught
>> up
>> > into ISR and reduced to the expected LEO.
>> > 4. Move to “MIRRORING” state and create the mirror fetcher
>> >
>> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch API.
>> > KIP is updated.
>> >
>> >
>> > > JR51. Make sense. To truncate based on LME, we need to issue another
>> RPC
>> > to translate the LME to an offset. It would be useful to include that in
>> > the workflow description.
>> >
>> > About the additional RPC to translate the LME to the offset (i.e.
>> > OffsetForLeaderEpochRequest), actually we don’t rely on this to do the
>> > offset translation. We use the local leader epoch cache in the
>> destination
>> > cluster to translate the epoch to save the RPC call. This works because
>> of
>> > an invariant the LME truncation protocol maintains: for any leader
>> epoch ≤
>> > LME, that epoch's data belongs to exactly one cluster, either it
>> originated
>> > there, or it was mirrored. This holds even across repeated mirror ->
>> > reverse-mirror -> mirror cycles, since each direction's LOG_TRUNCATION
>> > phase re-establishes the invariant before that cluster starts mirroring.
>> > With this invariant, the replication protocol can find out the diverge
>> > point and truncate the log correctly when mirroring. For example:
>> >
>> > Cluster A (source) --> Cluster B (dest)
>> > foo-0 leader log foo-0 leader log
>> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> > Offset 2: A (LE: 0)
>> > Offset 3: A (LE: 1)
>> >
>> > Cluster A crashes while Cluster B mirrors to offset 1. Failover to B and
>> > store the LME=0 in cluster B. When cluster A recovers and starts the
>> > reverse mirroring, it’ll query the LME from B and get back LME: 0. Then
>> > cluster A gets the end offset for epoch 0 from leaderEpochCache, and
>> gets
>> > offset 2. So offset 3 is truncated. The log is like this now:
>> >
>> > Cluster A (dest) <-- Cluster B (source)
>> > foo-0 leader log foo-0 leader log
>> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
>> > Offset 3: B (LE: 10)
>> >
>> > When mirror starts, cluster A sends the fetch request with {
>> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the divergingEpoch check
>> in
>> > the cluster B queries the endOffset for leader epoch 0, and found the
>> end
>> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns fetch
>> > response with divergingEpoch info to cluster A for log truncation. In
>> the
>> > end, offset 2 in cluster A will be truncated and then log converges with
>> > cluster B.
>> >
>> >
>> >
>> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
>> > because it mixes querying the mirror partition state with querying the
>> > mirror partition leader. Another option is to use
>> ReadMirrorStatesRequest
>> > to query the mirror partition state and a new request to query the
>> mirror
>> > partition leader. If we want the adminClient to query the right
>> ClusterMirrorCoordinator
>> > directly, we can extend the FindCoordinatorRequest to support that.
>> >
>> >
>> > Good suggestion. Yes, we need to fan out to all brokers because we have
>> to
>> > collect info from mirror partition leaders and mirror coordinators. If
>> we
>> > create a new API to contact mirror partition leaders, it will make the
>> API
>> > cleaner and clearer. So we will do this:
>> >
>> > a. Remove SourceOffset, DestinationOffset, and Lag fields from
>> > DescribeClusterMirror request/response
>> >
>> > b. Add a new API: ReadClusterMirrorLags API to request for SourceOffset,
>> > DestinationOffset, and Lag info.
>> >
>> > The workflow for admin client describeClusterMirror will be like this:
>> > 1. adminClient sends describeClusterMirror to the leastLoadedNode broker
>> > 2. The received broker collects the current mirrored partitions, and
>> > groups them by coordinator node and by partition leader node.
>> > 3. The broker sends ReadMirrorState request to the coordinator nodes to
>> > get the mirror state related info
>> > 4. The broker sends ReadClusterMirrorLags request to the mirror
>> partition
>> > leader nodes to get the mirror lag, source offset, destination offset
>> > (which updates on every fetch response).
>> > 5. The broker merge all the response into describeClusterMirror response
>> > and send to admin client
>> >
>> > KIP is updated. Thanks for the suggestion.
>> >
>> >
>> > > JR55. For __mirror_state topic, could we describe whether it is
>> > compacted?
>> >
>> > Yes, it is a compacted topic. I made it much clearer in the
>> > “ClusterMirrorCoordinator” section.
>> >
>> >
>> > Thank you,
>> > Luke
>> >
>> >
>> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <de...@kafka.apache.org>
>> > wrote:
>> >
>> >> Hi, Luke,
>> >>
>> >> Thanks for the reply.
>> >>
>> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
>> >> designed for the unclean leader election support in Cluster Mirroring
>> >> because the unclean leader election will break the assumption that the
>> >> committed data won’t be truncated."
>> >> If the source cluster has an unclean leader election, the destination
>> >> cluster will detect the inconsistency and truncate the log at the
>> >> destination leader. We can just
>> >> extend mirror.support.unclean.leader.election to cover this case,
>> i.e., we
>> >> wait until all replicas' log end offsets are reduced to the target
>> offset
>> >> before resuming mirroring. This avoids the inconsistency.
>> >>
>> >> If the destination cluster has an unclean leader election, it will
>> >> re-fetch
>> >> the same data from the source. There won't be any inconsistency either.
>> >>
>> >> So, I think we can avoid adding MirrorLeaderEpoch.
>> >>
>> >> JR51. Make sense. To truncate based on LME, we need to issue another
>> RPC
>> >> to
>> >> translate the LME to an offset. It would be useful to include that in
>> the
>> >> workflow description.
>> >>
>> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
>> because
>> >> it mixes querying the mirror partition state with querying the mirror
>> >> partition leader. Another option is to use ReadMirrorStatesRequest to
>> >> query
>> >> the mirror partition state and a new request to query the mirror
>> partition
>> >> leader. If we want the adminClient to query the right
>> >> ClusterMirrorCoordinator
>> >> directly, we can extend the FindCoordinatorRequest to support that.
>> >>
>> >> JR55. For __mirror_state topic, could we describe whether it is
>> compacted?
>> >>
>> >> Jun
>> >>
>> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com> wrote:
>> >>
>> >> > Hi Jun,
>> >> >
>> >> > Thanks for the comments.
>> >> > Answering them below:
>> >> >
>> >> > > JR31.3 It makes sense for the new partition leader to query the
>> >> current
>> >> > mirror state from the local or remote MirrorMetadataManager that owns
>> >> the
>> >> > partition's state. However, the wording in the KIP is confusing.
>> >> > MirrorMetadataManager
>> >> > mirror partition state cache should be populated only by one way,
>> which
>> >> is
>> >> > replaying __mirror_state partitions this broker leads. By saying
>> "and by
>> >> > fetching state from remote coordinators via ReadMirrorStates RPCs.",
>> >> it's
>> >> > as if the mirror partition state cache on each broker stores the
>> state
>> >> for
>> >> > all mirror partitions.
>> >> >
>> >> > You’re right. KIP is updated to make it clear.
>> >> >
>> >> >
>> >> > > JR36. I understand the intention now, but I still don't think we
>> need
>> >> > MirrorLeaderEpoch. Let's
>> >> > assume the common case: no unclean leader election in the source
>> cluster
>> >> > during mirroring. Mirroring only fetches the committed data from the
>> >> > source, which never changes if obtained a second time. In your
>> example
>> >> > image, follower broker 1 in the destination cluster can never obtain
>> the
>> >> > record B(0) because it's never committed in the source and thus is
>> never
>> >> > exposed to the destination leader. It's possible for broker 1 to
>> first
>> >> get
>> >> > D(1) which is truncated later when broker 0 follows a new leader. But
>> >> the
>> >> > new leader will refetch the same D(1) from the source cluster. So,
>> it's
>> >> ok
>> >> > for broker 1 to keep D(1).
>> >> >
>> >> >
>> >> > Yes, you are correct. Because mirroring only fetches the committed
>> data
>> >> > from the source, the MirrorLeaderEpoch is unnecessary for normal
>> >> mirroring.
>> >> > Thanks for the clarification. So I think we can say, the
>> >> > “MirrorLeaderEpoch” is specifically designed for the unclean leader
>> >> > election support in Cluster Mirroring because the unclean leader
>> >> election
>> >> > will break the assumption that the committed data won’t be truncated.
>> >> >
>> >> > Given that, we could gate the MirrorLeaderEpoch field by only
>> setting it
>> >> > in the Fetch request when
>> "mirror.support.unclean.leader.election=true"
>> >> > (false by default), and skip it entirely for normal mirroring. What
>> do
>> >> you
>> >> > think?
>> >> >
>> >> >
>> >> > > JR51. Sounds good. Why do we choose to store the last mirror epoch
>> >> > instead
>> >> > of a last mirror offset?
>> >> >
>> >> > Yes, originally we stored the “last mirror offset” instead of “last
>> >> mirror
>> >> > epoch”. But later, we found it doesn’t work for unclean leader
>> election
>> >> > cases. (yes, `unclean leader election` again :) ) For example:
>> >> >
>> >> > Cluster A (source) --> Cluster B (dest)
>> >> > foo-0 leader log foo-0 leader log
>> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >> >
>> >> > Cluster B is mirroring from A. And A crashes and failover to B. B
>> stores
>> >> > the “last mirror offset” as 1.
>> >> > When cluster A recovers, somehow the unclean leader election happens,
>> >> log
>> >> > is empty in the new leader, and then some more leadership changes to
>> >> make
>> >> > the leader epoch become 10. Then it starts the reverse mirror from B.
>> >> The
>> >> > log looks like this:
>> >> >
>> >> > Cluster A (dest) <-- Cluster B (source)
>> >> > foo-0 leader log foo-0 leader log
>> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
>> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
>> >> > Offset 2: pid_res (LE:
>> 10)
>> >> >
>> >> > When A starts mirroring from B, log is truncated to last mirror
>> offset:
>> >> 1
>> >> > (no-op), and then fetch from cluster B. The 1st fetch request is like
>> >> this:
>> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2, … }.
>> The
>> >> > divergingEpoch check in the cluster B queries the endOffset for
>> leader
>> >> > epoch 10 (in Partition#readRecords), and found the end offset for LE
>> 10
>> >> is
>> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch validation
>> passes
>> >> and
>> >> > returns offset 2 to cluster A. In short, relying on “last mirror
>> offset”
>> >> > cannot resolve the log diverge issue in unclean leader election case.
>> >> >
>> >> > If instead, we truncate until “last mirror epoch”: 0, the offset 0
>> and 1
>> >> > in cluster A will be truncated as expected and then log converge with
>> >> > cluster B correctly in the end.
>> >> >
>> >> >
>> >> > Also, about JR53.2:
>> >> > > JR53.2 "2. The admin client fans out the request to all brokers and
>> >> > merges
>> >> > the responses client-side."
>> >> > This is not ideal. Not every broker hosts a mirror partition. So
>> sending
>> >> > the request to every broker is wasteful. It's probably better to send
>> >> the
>> >> > request to a single broker, which can then contact the right
>> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >
>> >> > While working on this, I found we need to add one more tagged field
>> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema because
>> >> otherwise
>> >> > the receiving broker doesn’t know if this is the request from client
>> >> and it
>> >> > needs to contact other brokers, or this is just a forwarded request.
>> The
>> >> > workflow is like this:
>> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest to a
>> >> > leastLoaded broker, which doesn’t contain the "Forwarded" field.
>> >> > 2. The broker receives the request and sends
>> >> DescribeClusterMirrorsRequest
>> >> > with "Forwarded=true" to the responsible brokers and waits for the
>> >> response
>> >> > async.
>> >> > 3. When all responses are received, merge the results and respond to
>> the
>> >> > client.
>> >> >
>> >> > Note: we cannot use ReadMirrorStatesRequest here because it doesn’t
>> >> > contain the lag info in the coordinator.
>> >> >
>> >> > The alternative solution is the adminClient sends
>> >> > DescribeClusterMirrorsRequest to the “responsible brokers only” (via
>> >> > describeTopics), instead of ALL brokers, and merge the results in the
>> >> > client side. This solution doesn’t need to add a new field in the
>> >> > DescribeClusterMirrorsRequest and still can avoid unnecessary
>> requests
>> >> > sent.
>> >> >
>> >> > My preference is to keep the merge logic on the admin client side,
>> which
>> >> > saves the overhead in the broker side.
>> >> > What do you think?
>> >> >
>> >> >
>> >> > Thank you,
>> >> > Luke
>> >> >
>> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <dev@kafka.apache.org
>> >
>> >> > wrote:
>> >> >
>> >> >> Hi, Luke,
>> >> >>
>> >> >> Thanks for the reply.
>> >> >>
>> >> >> JR31.3 It makes sense for the new partition leader to query the
>> current
>> >> >> mirror state from the local or remote MirrorMetadataManager that
>> owns
>> >> the
>> >> >> partition's state. However, the wording in the KIP is confusing.
>> >> >> MirrorMetadataManager
>> >> >> mirror partition state cache should be populated only by one way,
>> >> which is
>> >> >> replaying __mirror_state partitions this broker leads. By saying
>> "and
>> >> by
>> >> >> fetching state from remote coordinators via ReadMirrorStates RPCs.",
>> >> it's
>> >> >> as if the mirror partition state cache on each broker stores the
>> state
>> >> for
>> >> >> all mirror partitions.
>> >> >>
>> >> >> JR36. I understand the intention now, but I still don't think we
>> need
>> >> >> MirrorLeaderEpoch. Let's
>> >> >> assume the common case: no unclean leader election in the source
>> >> cluster
>> >> >> during mirroring. Mirroring only fetches the committed data from the
>> >> >> source, which never changes if obtained a second time. In your
>> example
>> >> >> image, follower broker 1 in the destination cluster can never obtain
>> >> the
>> >> >> record B(0) because it's never committed in the source and thus is
>> >> never
>> >> >> exposed to the destination leader. It's possible for broker 1 to
>> first
>> >> get
>> >> >> D(1) which is truncated later when broker 0 follows a new leader.
>> But
>> >> the
>> >> >> new leader will refetch the same D(1) from the source cluster. So,
>> >> it's ok
>> >> >> for broker 1 to keep D(1).
>> >> >>
>> >> >> JR51. Sounds good. Why do we choose to store the last mirror epoch
>> >> instead
>> >> >> of a last mirror offset?
>> >> >>
>> >> >> Jun
>> >> >>
>> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com> wrote:
>> >> >>
>> >> >> > Hi Jun,
>> >> >> >
>> >> >> > Thanks for the great comments again.
>> >> >> > Answering your question below.
>> >> >> >
>> >> >> > > JR31.3 "Regarding state caching: the mirror partition cache
>> lives
>> >> in
>> >> >> > MirrorMetadataManager. It is populated in two ways: by replaying
>> >> >> > __mirror_state partitions this broker leads, and by fetching state
>> >> >> > from remote coordinators via ReadMirrorStates RPCs."
>> >> >> > Hmm, a given broker only caches the mirror partition state for
>> >> >> partitions
>> >> >> > hashed into the __mirror_state partition hosted on this broker,
>> >> right?
>> >> >> Why
>> >> >> > does it need to fetch the state from other remote coordinators?
>> >> >> >
>> >> >> > > a given broker only caches the mirror partition state for
>> >> partitions
>> >> >> > hashed into the __mirror_state partition hosted on this broker,
>> >> right?
>> >> >> >
>> >> >> > Right.
>> >> >> >
>> >> >> > > Why does it need to fetch the state from other remote
>> coordinators?
>> >> >> >
>> >> >> > Because the data partition leader needs to know the current mirror
>> >> state
>> >> >> > of the partitions it leads. On a leadership change, the metadata
>> >> delta
>> >> >> is
>> >> >> > published to the MirrorMetadataManager, which queries the current
>> >> mirror
>> >> >> > state for its led partitions from the coordinator (local or
>> remote),
>> >> and
>> >> >> > processes tasks based on that state. This is the reason why it
>> >> fetches
>> >> >> the
>> >> >> > state from other remote coordinator. This part of cache will be
>> >> cleared
>> >> >> > after the leadership change.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > > JR36. I am still not quite sure why we need MirrorLeaderEpoch in
>> >> the
>> >> >> > Fetch
>> >> >> > request. In the current Fetch request, in addition to
>> >> >> CurrentLeaderEpoch,
>> >> >> > it has another field LastFetchedEpoch, which captures the last
>> leader
>> >> >> epoch
>> >> >> > in the data and is used to resolve the inconsistency between the
>> >> >> follower
>> >> >> > and the leader. Does it achieve the same purpose as
>> >> MirrorLeaderEpoch?
>> >> >> >
>> >> >> >
>> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to do the
>> >> validation
>> >> >> to
>> >> >> > avoid the issue in KAFKA-18723. If we reject any fetched batches
>> with
>> >> >> epoch
>> >> >> > higher than the LastFetchedEpoch, it means the follower can never
>> >> accept
>> >> >> > new appended batches with higher epoch in the leader node.
>> >> >> >
>> >> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes in the
>> >> fetch
>> >> >> > API:
>> >> >> > 1. The receiver (leader node) uses it to validate it matches the
>> >> leader
>> >> >> > epoch in the leader node
>> >> >> > 2. The sender (follower node) uses it to validate the epoch in the
>> >> >> fetched
>> >> >> > batches are all <= this value.
>> >> >> >
>> >> >> > For purpose (1), the CurrentLeaderEpoch is still serving for
>> >> followers
>> >> >> in
>> >> >> > the destination cluster. For purpose (2), my understanding of the
>> >> >> > validation for KAFKA-18723 is like: the follower knows the
>> >> >> > CurrentLeaderEpoch X is the highest epoch the leader node should
>> >> >> contain in
>> >> >> > the log batches when fetch request is sent. Any log batch beyond
>> this
>> >> >> epoch
>> >> >> > in the fetch response means the follower’s metadata is stale. The
>> >> >> batches
>> >> >> > need to be rejected and follower needs to update metadata for next
>> >> >> fetch.
>> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves
>> this
>> >> >> purpose
>> >> >> > to let the followers know the highest epoch in the leader node
>> log in
>> >> >> each
>> >> >> > fetch response. Once the info is stale in the follower side, it
>> >> updates
>> >> >> it
>> >> >> > from fetch response, and send next fetch request with correct
>> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is possible
>> like
>> >> >> > KAFKA-18723.
>> >> >> >
>> >> >> > I tried not to add new field in Fetch API, but this is the best
>> >> >> solution I
>> >> >> > can think of.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > > JR48. "The NumPartitions field is needed so the controller can
>> >> create
>> >> >> the
>> >> >> > destination topic if it does not already exist. When a topic is
>> being
>> >> >> > mirrored for the first time, the destination cluster has no
>> knowledge
>> >> >> > of the source topic's partition count, so the request must carry
>> it."
>> >> >> > Hmm, this seems inconsistent. When a mirror is started, the source
>> >> topic
>> >> >> > may not exist yet. When the source topic is created later, we can
>> >> start
>> >> >> > mirroring without requiring the user to provide the number of
>> >> >> partitions.
>> >> >> >
>> >> >> > > When the source topic is created later, we can start mirroring
>> >> without
>> >> >> > requiring the user to provide the number of partitions.
>> >> >> >
>> >> >> > This is because this happens in periodical sync up in the
>> destination
>> >> >> > cluster, and the numPartitions and other topic info in the source
>> >> >> cluster
>> >> >> > is already retrieved during the sync up process. Also, if users
>> are
>> >> >> > creating mirror topics via admin API, this info can also be
>> retrieved
>> >> >> > inside the admin client if not provided.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > > JR51. Start Mirror: I am wondering if we truly need the last
>> mirror
>> >> >> epoch
>> >> >> > for truncation when starting a mirror. When a mirror is stopped at
>> >> the
>> >> >> > source, we already bump up the leader epoch and write some
>> additional
>> >> >> > records for cleanup. It seems that we can just use the
>> DivergingEpoch
>> >> >> > returned in the initial mirror Fetch request to detect the
>> diverging
>> >> >> epoch
>> >> >> > and use that for truncation at the mirror destination.
>> >> >> >
>> >> >> > That’s a good question. I thought about this before but
>> >> unfortunately it
>> >> >> > doesn’t work. The reason we need the last mirror epoch record is
>> >> because
>> >> >> > the destination cluster might not completely sync with the source
>> >> >> cluster
>> >> >> > before STOPPED. And this un-mirrored data might confuse the
>> >> replication
>> >> >> > protocol when reverse mirror. For example,
>> >> >> >
>> >> >> > Cluster A (source) —> Cluster B (dest)
>> >> >> > foo-0 leader log foo-0 leader log
>> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> >> > Offset 2: from A (LE: 10)
>> >> >> >
>> >> >> > While cluster B mirrors until offset 1, the cluster A crashes and
>> >> >> failover
>> >> >> > to cluster B. B bumps leader epoch to 10, appends a
>> MIRROR_PID_RESET
>> >> >> with
>> >> >> > LE: 10. Then cluster A starts to reverse mirror from cluster B,
>> the
>> >> log
>> >> >> > will be like this:
>> >> >> >
>> >> >> > Cluster A (dest) <-- Cluster B (source)
>> >> >> > foo-0 leader log: foo-0 leader log:
>> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
>> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
>> >> >> >
>> >> >> > As you can see, cluster A (new destination cluster) doesn’t know
>> the
>> >> >> > offset 2 is not belonging to the cluster B, which causes
>> inconsistent
>> >> >> logs.
>> >> >> > So like what we described in the KIP, the last mirror epoch is the
>> >> >> > synchronization point between source and destination. Any data
>> beyond
>> >> >> the
>> >> >> > point is treated as unknown to the source cluster and should be
>> >> >> truncated.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > > JR52. Stop Mirror:
>> >> >> > > JR52.1 "3. If patterns are provided, the controller removes
>> >> matching
>> >> >> > entries from mirror.topics.include or adds them to
>> >> >> mirror.topics.exclude on
>> >> >> > the CLUSTER_MIRROR resource."
>> >> >> > Why is it 'or' and not 'and'?
>> >> >> >
>> >> >> > Yes, that should be AND. KIP updated. Thanks.
>> >> >> >
>> >> >> >
>> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
>> >> >> MirrorMetadataManager
>> >> >> > discovers new source topics matching the persisted include/exclude
>> >> >> patterns
>> >> >> > and repeats steps 3 through 7 for each."
>> >> >> > The broker will send a create topic request to the controller. How
>> >> does
>> >> >> the
>> >> >> > controller associate it with the mirror name to create a
>> >> >> > `MirrorTopicStateChangeRecord`?
>> >> >> >
>> >> >> > No, we don’t send a create topic request to the controller here.
>> >> >> Actually
>> >> >> > we send a StartMirrorTopics request to the controller, which
>> contains
>> >> >> the
>> >> >> > mirror name and topic metadata like NumPartitions, topicId… etc.
>> So
>> >> the
>> >> >> > controller can create the non-existed topic as explained in JR48,
>> and
>> >> >> > associate it to MirrorTopicStateChangeRecord.
>> >> >> >
>> >> >> >
>> >> >> > > JR53. Describe Mirrors:
>> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors request with
>> >> >> optional
>> >> >> > mirror names (empty means all mirrors)."
>> >> >> > In metadata request, we use a null arrary to represent all
>> topics. It
>> >> >> would
>> >> >> > be useful to be consistent here.
>> >> >> >
>> >> >> > OK, KIP updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR53.2 "2. The admin client fans out the request to all brokers
>> and
>> >> >> > merges
>> >> >> > the responses client-side."
>> >> >> > This is not ideal. Not every broker hosts a mirror partition. So
>> >> sending
>> >> >> > the request to every broker is wasteful. It's probably better to
>> send
>> >> >> the
>> >> >> > request to a single broker, which can then contact the right
>> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >
>> >> >> > Agree, KIP updated.
>> >> >> >
>> >> >> >
>> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
>> >> >> PAUSED,
>> >> >> > STOPPED the only validate states? The state table has other states
>> >> too.
>> >> >> > Ditto for MirrorPartitionState.
>> >> >> > The DesiredState field in MirrorTopicStateChangeRecord is to
>> define
>> >> the
>> >> >> > target state the user wants. And the desired state users can
>> assign
>> >> is
>> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as
>> >> >> STOPPING
>> >> >> > or LOG_TRUNCATION… etc because they are more like intermediate
>> >> states.
>> >> >> >
>> >> >> > About MirrorPartitionState record, it’ll record any state in the
>> >> state
>> >> >> > table that the current partition is. This is the real state for
>> each
>> >> >> > partition. It’s like in the kubernetes world, when we change the
>> >> replica
>> >> >> > field of the deployment yaml from 3 -> 5 (desired state), it won’t
>> >> >> increase
>> >> >> > to 5 immediately. The current replica (MirrorPartitionState
>> record)
>> >> is
>> >> >> > still 3, then 4 (maybe), and finally 5.
>> >> >> >
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Luke
>> >> >> >
>> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
>> dev@kafka.apache.org
>> >> >
>> >> >> > wrote:
>> >> >> >
>> >> >> >> Hi, Fede,
>> >> >> >>
>> >> >> >> Thanks for the reply. A few more comments.
>> >> >> >>
>> >> >> >> JR31.3 "Regarding state caching: the mirror partition cache
>> lives in
>> >> >> >> MirrorMetadataManager. It is populated in two ways: by replaying
>> >> >> >> __mirror_state partitions this broker leads, and by fetching
>> state
>> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
>> >> >> >> Hmm, a given broker only caches the mirror partition state for
>> >> >> partitions
>> >> >> >> hashed into the __mirror_state partition hosted on this broker,
>> >> right?
>> >> >> Why
>> >> >> >> does it need to fetch the state from other remote coordinators?
>> >> >> >>
>> >> >> >> JR36. I am still not quite sure why we need MirrorLeaderEpoch in
>> the
>> >> >> Fetch
>> >> >> >> request. In the current Fetch request, in addition to
>> >> >> CurrentLeaderEpoch,
>> >> >> >> it has another field LastFetchedEpoch, which captures the last
>> >> leader
>> >> >> >> epoch
>> >> >> >> in the data and is used to resolve the inconsistency between the
>> >> >> follower
>> >> >> >> and the leader. Does it achieve the same purpose as
>> >> MirrorLeaderEpoch?
>> >> >> >>
>> >> >> >> JR48. "The NumPartitions field is needed so the controller can
>> >> create
>> >> >> the
>> >> >> >> destination topic if it does not already exist. When a topic is
>> >> being
>> >> >> >> mirrored for the first time, the destination cluster has no
>> >> knowledge
>> >> >> >> of the source topic's partition count, so the request must carry
>> >> it."
>> >> >> >> Hmm, this seems inconsistent. When a mirror is started, the
>> source
>> >> >> topic
>> >> >> >> may not exist yet. When the source topic is created later, we can
>> >> start
>> >> >> >> mirroring without requiring the user to provide the number of
>> >> >> partitions.
>> >> >> >>
>> >> >> >> JR51. Start Mirror: I am wondering if we truly need the last
>> mirror
>> >> >> epoch
>> >> >> >> for truncation when starting a mirror. When a mirror is stopped
>> at
>> >> the
>> >> >> >> source, we already bump up the leader epoch and write some
>> >> additional
>> >> >> >> records for cleanup. It seems that we can just use the
>> >> DivergingEpoch
>> >> >> >> returned in the initial mirror Fetch request to detect the
>> diverging
>> >> >> epoch
>> >> >> >> and use that for truncation at the mirror destination.
>> >> >> >>
>> >> >> >> JR52. Stop Mirror:
>> >> >> >> JR52.1 "3. If patterns are provided, the controller removes
>> matching
>> >> >> >> entries from mirror.topics.include or adds them to
>> >> >> mirror.topics.exclude
>> >> >> >> on
>> >> >> >> the CLUSTER_MIRROR resource."
>> >> >> >> Why is it 'or' and not 'and'?
>> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
>> >> MirrorMetadataManager
>> >> >> >> discovers new source topics matching the persisted
>> include/exclude
>> >> >> >> patterns
>> >> >> >> and repeats steps 3 through 7 for each."
>> >> >> >> The broker will send a create topic request to the controller.
>> How
>> >> does
>> >> >> >> the
>> >> >> >> controller associate it with the mirror name to create a
>> >> >> >> `MirrorTopicStateChangeRecord`?
>> >> >> >>
>> >> >> >> JR53. Describe Mirrors:
>> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors request with
>> >> >> optional
>> >> >> >> mirror names (empty means all mirrors)."
>> >> >> >> In metadata request, we use a null arrary to represent all
>> topics.
>> >> It
>> >> >> >> would
>> >> >> >> be useful to be consistent here.
>> >> >> >> JR53.2 "2. The admin client fans out the request to all brokers
>> and
>> >> >> merges
>> >> >> >> the responses client-side."
>> >> >> >> This is not ideal. Not every broker hosts a mirror partition. So
>> >> >> sending
>> >> >> >> the request to every broker is wasteful. It's probably better to
>> >> send
>> >> >> the
>> >> >> >> request to a single broker, which can then contact the right
>> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >> >>
>> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
>> >> PAUSED,
>> >> >> >> STOPPED the only validate states? The state table has other
>> states
>> >> too.
>> >> >> >> Ditto for MirrorPartitionState.
>> >> >> >>
>> >> >> >> Jun
>> >> >> >>
>> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
>> >> fedevaleri@gmail.com>
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >> > Hi Andrew, yes, validation is called at the start of
>> createTopic.
>> >> If
>> >> >> >> > the topic ID is invalid, the request fails with
>> INVALID_REQUEST.
>> >> If
>> >> >> >> > the topic ID is already used by a different topic name, it
>> fails
>> >> with
>> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual topic
>> in
>> >> the
>> >> >> >> > batch fails; other topics in the same CreateTopics request
>> >> continue
>> >> >> >> > processing normally. Added this information to the KIP.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
>> >> >> >> schofielaj@apache.org>
>> >> >> >> > wrote:
>> >> >> >> > >
>> >> >> >> > > Hi Fede,
>> >> >> >> > > One more small question.
>> >> >> >> > >
>> >> >> >> > > AS37: If the controller fails validation for the
>> >> MirrorInfo.TopicId
>> >> >> >> > added to the CreateTopic request, what does it do? I suspect
>> that
>> >> it
>> >> >> >> fails
>> >> >> >> > the request with a particular error code.
>> >> >> >> > >
>> >> >> >> > > Thanks,
>> >> >> >> > > Andrew
>> >> >> >> > >
>> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> >> >> >> > > > Hi all,
>> >> >> >> > > >
>> >> >> >> > > > We updated the KIP to include Coordinator Runtime
>> >> configurations
>> >> >> and
>> >> >> >> > > > the coordinator state transition validation paragraph. The
>> >> latter
>> >> >> >> > > > covers the per-partition epoch fencing mechanism (leader
>> epoch
>> >> >> and
>> >> >> >> > > > state epoch) that complements the existing per-topic
>> >> StateOffset
>> >> >> >> > > > fencing for lifecycle operations.
>> >> >> >> > > >
>> >> >> >> > > > Thanks,
>> >> >> >> > > > Fede
>> >> >> >> > > >
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for the comments.
Answering them below.
> JR36. Sounds good. Should we remove MirrorLeaderEpochExceededException?
Sorry, it’s a confusing name. The MirrorLeaderEpochExceededException is
used and thrown when the fetched mirror batch leader epoch > local leader
epoch in destination cluster. And then we’ll stop mirroring and move the
partition to EPOCH_FENSING state to wait for local leader epoch bump. I
think since we don’t have the concept of `MirrorLeaderEpoch` anymore, the
`MirrorLeaderEpochExceededException` should not cause confusion now. I’ll
keep the name for the exception.
> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the mirror
fetch request issued to the source cluster, independent of the value of
mirror.support.unclean.leader.election? It's needed to detect divergence
during failover. However, if it's also set when
mirror.support.unclean.leader.election=false,
an unclean leader election in the source cluster can halt the mirroring
since we don’t do anything with the returned divergeEpoch in that mode
and continue
the next fetch.
> Do we plan to always set lastFetchedEpoch in the mirror
fetch request issued to the source cluster, independent of the value of
Mirror.support.unclean.leader.election?
Yes, we always set lastFetchedEpoch in the mirror fetch request.
> However, if it's also set when
mirror.support.unclean.leader.election=false,
an unclean leader election in the source cluster can halt the mirroring
since we don’t do anything with the returned divergeEpoch in that mode
and continue the next fetch.
Sorry I didn’t make it clear. When
“mirror.support.unclean.leader.election=false” and a “divergeEpoch” is set
in fetch response, the leader node will handle the divergeEpoch and then
continue the next fetch as usual. The difference with
“mirror.support.unclean.leader.election=true” is that “should we wait for
ALL replicas caught up (complete truncation)”. Without the waiting, it’s
possible the inconsistent log issue in KAFKA-18723 could happen between
leader and follower nodes in the destination cluster because we don’t have
“MirrorLeaderEpoch” concept now to reject higher leader epoch batches. That
is, we won't halt the mirroring if "divergeEpoch" is set but
“mirror.support.unclean.leader.election=false”.
> JR55. The DescribeClusterMirrorsRequest still returns both the mirror
partition state and the mirror partition leader state. The problem is that
if a caller only needs one of the state, the other state becomes redundant.
For example, when starting the mirror, to obtain LME, we only need the
mirror partition state. Here is an alternative design: We expose the mirror
partition state through ReadMirrorState. The client will use the
FindCoordinatorRequest
to find the right mirror partition coordinator to issue that request. We
expose the mirror partition leader state through the ReadClusterMirrorLags
request. The client can use the metadata request to find the partition
leader to issue that request.
> In the describing mirror flow, the admin client first issues the
ListMirrorRequest to find all the mirrors and the topics being mirrored. It
then issues the metadata request to find the partitions for those topics
and issues a FindCoordinatorRequest to find the mirror partition state
coordinators. Finally, it issues ReadMirrorState to the mirror partition
state coordinator and ReadClusterMirrorLags to the partition leader.
> In the starting mirror flow, the partition leader in the destination
cluster issues a ListMirrorRequest to find the mirror name based on its
cluster id. It then issues a FindCoordinatorRequest to find the mirror
partition state coordinators. Finally, it issues ReadMirrorState to the
mirror partition state coordinator to get the LME.
> We can also improve the ListMirror request to accept a filter based on
mirror name, source cluster ID and topic name.
Answers:
> The DescribeClusterMirrorsRequest still returns both the mirror
partition state and the mirror partition leader state. The problem is that
if a caller only needs one of the state, the other state becomes redundant.
For example, when starting the mirror, to obtain LME, we only need the
mirror partition state.
Good point!
> Here is an alternative design: We expose the mirror
partition state through ReadMirrorState. The client will use the
FindCoordinatorRequest
to find the right mirror partition coordinator to issue that request. We
expose the mirror partition leader state through the ReadClusterMirrorLags
request. The client can use the metadata request to find the partition
leader to issue that request.
Sounds good. But we try not to expose ReadMirrorState/ReadClusterMirrorLags
directly to the admin client because users might get confused about these 2
requests, and they are also both related to describeClusterMirror. I think
we can add 2 additional fields to describeClusterMirrorRequest:
“includeMirrorStates” and “includeMirrorLags”. So the admin client can
include either one or both of them in the DescrbieClusterMirrorRequest
based on the provided options. KIP is updated.
> In the describing mirror flow, the admin client first issues the
ListMirrorRequest to find all the mirrors and the topics being mirrored. It
then issues the metadata request to find the partitions for those topics
and issues a FindCoordinatorRequest to find the mirror partition state
coordinators. Finally, it issues ReadMirrorState to the mirror partition
state coordinator and ReadClusterMirrorLags to the partition leader.
Makes sense. KIP is updated.
> In the starting mirror flow, the partition leader in the destination
cluster issues a ListMirrorRequest to find the mirror name based on its
cluster id. It then issues a FindCoordinatorRequest to find the mirror
partition state coordinators. Finally, it issues ReadMirrorState to the
mirror partition state coordinator to get the LME.
Make sense. So we can send DescribeClusterMirrorRequest with
“includeMirrorStates” only.
> We can also improve the ListMirror request to accept a filter based on
mirror name, source cluster ID and topic name.
Good suggestion, just like what ListGroupsRequest does.
Added "SourceClusterIdFilter", "MirrorNameFilter",
"DesiredMirrorStateFilter" in the ListClusterMirrorsRequest. KIP updated.
> JR56. Could you list the ACL needed for each new RPC?
Yes, it’s already listed in “Security Control” section.
> JR57. In the command workflow, it sometimes says "The user sends" and some
other times says "The admin client sends". It's better to use "The admin
client" consistently in all those places.
Agree. KIP updated.
Thank you,
Luke
On Sat, Aug 8, 2026 at 9:34 AM Jun Rao via dev <de...@kafka.apache.org> wrote:
> Hi, Luke,
>
> Thanks for the reply.
>
> JR36. Sounds good. Should we remove MirrorLeaderEpochExceededException?
>
> JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the mirror
> fetch request issued to the source cluster, independent of the value of
> mirror.support.unclean.leader.election? It's needed to detect divergence
> during failover. However, if it's also set when
> mirror.support.unclean.leader.election=false,
> an unclean leader election in the source cluster can halt the mirroring
> since we don’t do anything with the returned divergeEpoch in that mode
> and continue
> the next fetch.
>
> JR55. The DescribeClusterMirrorsRequest still returns both the mirror
> partition state and the mirror partition leader state. The problem is that
> if a caller only needs one of the state, the other state becomes redundant.
> For example, when starting the mirror, to obtain LME, we only need the
> mirror partition state. Here is an alternative design: We expose the mirror
> partition state through ReadMirrorState. The client will use the
> FindCoordinatorRequest
> to find the right mirror partition coordinator to issue that request. We
> expose the mirror partition leader state through the ReadClusterMirrorLags
> request. The client can use the metadata request to find the partition
> leader to issue that request.
>
> In the describing mirror flow, the admin client first issues the
> ListMirrorRequest to find all the mirrors and the topics being mirrored. It
> then issues the metadata request to find the partitions for those topics
> and issues a FindCoordinatorRequest to find the mirror partition state
> coordinators. Finally, it issues ReadMirrorState to the mirror partition
> state coordinator and ReadClusterMirrorLags to the partition leader.
>
> In the starting mirror flow, the partition leader in the destination
> cluster issues a ListMirrorRequest to find the mirror name based on its
> cluster id. It then issues a FindCoordinatorRequest to find the mirror
> partition state coordinators. Finally, it issues ReadMirrorState to the
> mirror partition state coordinator to get the LME.
>
> We can also improve the ListMirror request to accept a filter based on
> mirror name, source cluster ID and topic name.
>
> JR56. Could you list the ACL needed for each new RPC?
>
> JR57. In the command workflow, it sometimes says "The user sends" and some
> other times says "The admin client sends". It's better to use "The admin
> client" consistently in all those places.
>
> Jun
>
>
>
> On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for your comments.
> > Answering them below.
> >
> >
> > > JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
> > designed for the unclean leader election support in Cluster Mirroring
> > because the unclean leader election will break the assumption that the
> > committed data won’t be truncated."
> > If the source cluster has an unclean leader election, the destination
> > cluster will detect the inconsistency and truncate the log at the
> > destination leader. We can just
> > extend mirror.support.unclean.leader.election to cover this case, i.e.,
> we
> > wait until all replicas' log end offsets are reduced to the target offset
> > before resuming mirroring. This avoids the inconsistency.
> >
> > If the destination cluster has an unclean leader election, it will
> > re-fetch the same data from the source. There won't be any inconsistency
> > either.
> > So, I think we can avoid adding MirrorLeaderEpoch.
> >
> >
> > Yes, that works, too. I agree this is better because the unclean leader
> > election is rare, we don't want to occupy a field just for it.
> >
> > So to achieve this, we have to:
> > 1. add one more state: “ULE_LOG_TRUNCATION”,
> > 2. When the destination cluster leader completes a log truncation after a
> > fetch response due to divergeEpoch, we know it must be caused by unclean
> > leader election in the source cluster because we only fetch committed
> > records. If “mirror.support.unclean.leader.election=false”, we don’t do
> > anything and continue the next fetch. If
> > “mirror.support.unclean.leader.election=true”, we remove the mirror
> fetcher
> > and move the partition to “ULE_LOG_TRUNCATION” state.
> > 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be caught
> up
> > into ISR and reduced to the expected LEO.
> > 4. Move to “MIRRORING” state and create the mirror fetcher
> >
> > So that we don’t need the “MirrorLeaderEpoch” field in Fetch API.
> > KIP is updated.
> >
> >
> > > JR51. Make sense. To truncate based on LME, we need to issue another
> RPC
> > to translate the LME to an offset. It would be useful to include that in
> > the workflow description.
> >
> > About the additional RPC to translate the LME to the offset (i.e.
> > OffsetForLeaderEpochRequest), actually we don’t rely on this to do the
> > offset translation. We use the local leader epoch cache in the
> destination
> > cluster to translate the epoch to save the RPC call. This works because
> of
> > an invariant the LME truncation protocol maintains: for any leader epoch
> ≤
> > LME, that epoch's data belongs to exactly one cluster, either it
> originated
> > there, or it was mirrored. This holds even across repeated mirror ->
> > reverse-mirror -> mirror cycles, since each direction's LOG_TRUNCATION
> > phase re-establishes the invariant before that cluster starts mirroring.
> > With this invariant, the replication protocol can find out the diverge
> > point and truncate the log correctly when mirroring. For example:
> >
> > Cluster A (source) --> Cluster B (dest)
> > foo-0 leader log foo-0 leader log
> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> > Offset 2: A (LE: 0)
> > Offset 3: A (LE: 1)
> >
> > Cluster A crashes while Cluster B mirrors to offset 1. Failover to B and
> > store the LME=0 in cluster B. When cluster A recovers and starts the
> > reverse mirroring, it’ll query the LME from B and get back LME: 0. Then
> > cluster A gets the end offset for epoch 0 from leaderEpochCache, and gets
> > offset 2. So offset 3 is truncated. The log is like this now:
> >
> > Cluster A (dest) <-- Cluster B (source)
> > foo-0 leader log foo-0 leader log
> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> > Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
> > Offset 3: B (LE: 10)
> >
> > When mirror starts, cluster A sends the fetch request with {
> > lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the divergingEpoch check
> in
> > the cluster B queries the endOffset for leader epoch 0, and found the end
> > offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns fetch
> > response with divergingEpoch info to cluster A for log truncation. In the
> > end, offset 2 in cluster A will be truncated and then log converges with
> > cluster B.
> >
> >
> >
> > > JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
> > because it mixes querying the mirror partition state with querying the
> > mirror partition leader. Another option is to use ReadMirrorStatesRequest
> > to query the mirror partition state and a new request to query the mirror
> > partition leader. If we want the adminClient to query the right
> ClusterMirrorCoordinator
> > directly, we can extend the FindCoordinatorRequest to support that.
> >
> >
> > Good suggestion. Yes, we need to fan out to all brokers because we have
> to
> > collect info from mirror partition leaders and mirror coordinators. If we
> > create a new API to contact mirror partition leaders, it will make the
> API
> > cleaner and clearer. So we will do this:
> >
> > a. Remove SourceOffset, DestinationOffset, and Lag fields from
> > DescribeClusterMirror request/response
> >
> > b. Add a new API: ReadClusterMirrorLags API to request for SourceOffset,
> > DestinationOffset, and Lag info.
> >
> > The workflow for admin client describeClusterMirror will be like this:
> > 1. adminClient sends describeClusterMirror to the leastLoadedNode broker
> > 2. The received broker collects the current mirrored partitions, and
> > groups them by coordinator node and by partition leader node.
> > 3. The broker sends ReadMirrorState request to the coordinator nodes to
> > get the mirror state related info
> > 4. The broker sends ReadClusterMirrorLags request to the mirror partition
> > leader nodes to get the mirror lag, source offset, destination offset
> > (which updates on every fetch response).
> > 5. The broker merge all the response into describeClusterMirror response
> > and send to admin client
> >
> > KIP is updated. Thanks for the suggestion.
> >
> >
> > > JR55. For __mirror_state topic, could we describe whether it is
> > compacted?
> >
> > Yes, it is a compacted topic. I made it much clearer in the
> > “ClusterMirrorCoordinator” section.
> >
> >
> > Thank you,
> > Luke
> >
> >
> > On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Luke,
> >>
> >> Thanks for the reply.
> >>
> >> JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
> >> designed for the unclean leader election support in Cluster Mirroring
> >> because the unclean leader election will break the assumption that the
> >> committed data won’t be truncated."
> >> If the source cluster has an unclean leader election, the destination
> >> cluster will detect the inconsistency and truncate the log at the
> >> destination leader. We can just
> >> extend mirror.support.unclean.leader.election to cover this case, i.e.,
> we
> >> wait until all replicas' log end offsets are reduced to the target
> offset
> >> before resuming mirroring. This avoids the inconsistency.
> >>
> >> If the destination cluster has an unclean leader election, it will
> >> re-fetch
> >> the same data from the source. There won't be any inconsistency either.
> >>
> >> So, I think we can avoid adding MirrorLeaderEpoch.
> >>
> >> JR51. Make sense. To truncate based on LME, we need to issue another RPC
> >> to
> >> translate the LME to an offset. It would be useful to include that in
> the
> >> workflow description.
> >>
> >> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
> because
> >> it mixes querying the mirror partition state with querying the mirror
> >> partition leader. Another option is to use ReadMirrorStatesRequest to
> >> query
> >> the mirror partition state and a new request to query the mirror
> partition
> >> leader. If we want the adminClient to query the right
> >> ClusterMirrorCoordinator
> >> directly, we can extend the FindCoordinatorRequest to support that.
> >>
> >> JR55. For __mirror_state topic, could we describe whether it is
> compacted?
> >>
> >> Jun
> >>
> >> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thanks for the comments.
> >> > Answering them below:
> >> >
> >> > > JR31.3 It makes sense for the new partition leader to query the
> >> current
> >> > mirror state from the local or remote MirrorMetadataManager that owns
> >> the
> >> > partition's state. However, the wording in the KIP is confusing.
> >> > MirrorMetadataManager
> >> > mirror partition state cache should be populated only by one way,
> which
> >> is
> >> > replaying __mirror_state partitions this broker leads. By saying "and
> by
> >> > fetching state from remote coordinators via ReadMirrorStates RPCs.",
> >> it's
> >> > as if the mirror partition state cache on each broker stores the state
> >> for
> >> > all mirror partitions.
> >> >
> >> > You’re right. KIP is updated to make it clear.
> >> >
> >> >
> >> > > JR36. I understand the intention now, but I still don't think we
> need
> >> > MirrorLeaderEpoch. Let's
> >> > assume the common case: no unclean leader election in the source
> cluster
> >> > during mirroring. Mirroring only fetches the committed data from the
> >> > source, which never changes if obtained a second time. In your example
> >> > image, follower broker 1 in the destination cluster can never obtain
> the
> >> > record B(0) because it's never committed in the source and thus is
> never
> >> > exposed to the destination leader. It's possible for broker 1 to first
> >> get
> >> > D(1) which is truncated later when broker 0 follows a new leader. But
> >> the
> >> > new leader will refetch the same D(1) from the source cluster. So,
> it's
> >> ok
> >> > for broker 1 to keep D(1).
> >> >
> >> >
> >> > Yes, you are correct. Because mirroring only fetches the committed
> data
> >> > from the source, the MirrorLeaderEpoch is unnecessary for normal
> >> mirroring.
> >> > Thanks for the clarification. So I think we can say, the
> >> > “MirrorLeaderEpoch” is specifically designed for the unclean leader
> >> > election support in Cluster Mirroring because the unclean leader
> >> election
> >> > will break the assumption that the committed data won’t be truncated.
> >> >
> >> > Given that, we could gate the MirrorLeaderEpoch field by only setting
> it
> >> > in the Fetch request when
> "mirror.support.unclean.leader.election=true"
> >> > (false by default), and skip it entirely for normal mirroring. What do
> >> you
> >> > think?
> >> >
> >> >
> >> > > JR51. Sounds good. Why do we choose to store the last mirror epoch
> >> > instead
> >> > of a last mirror offset?
> >> >
> >> > Yes, originally we stored the “last mirror offset” instead of “last
> >> mirror
> >> > epoch”. But later, we found it doesn’t work for unclean leader
> election
> >> > cases. (yes, `unclean leader election` again :) ) For example:
> >> >
> >> > Cluster A (source) --> Cluster B (dest)
> >> > foo-0 leader log foo-0 leader log
> >> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> >> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >> >
> >> > Cluster B is mirroring from A. And A crashes and failover to B. B
> stores
> >> > the “last mirror offset” as 1.
> >> > When cluster A recovers, somehow the unclean leader election happens,
> >> log
> >> > is empty in the new leader, and then some more leadership changes to
> >> make
> >> > the leader epoch become 10. Then it starts the reverse mirror from B.
> >> The
> >> > log looks like this:
> >> >
> >> > Cluster A (dest) <-- Cluster B (source)
> >> > foo-0 leader log foo-0 leader log
> >> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> >> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> >> > Offset 2: pid_res (LE: 10)
> >> >
> >> > When A starts mirroring from B, log is truncated to last mirror
> offset:
> >> 1
> >> > (no-op), and then fetch from cluster B. The 1st fetch request is like
> >> this:
> >> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2, … }.
> The
> >> > divergingEpoch check in the cluster B queries the endOffset for leader
> >> > epoch 10 (in Partition#readRecords), and found the end offset for LE
> 10
> >> is
> >> > 2 in B, which is >= fetchOffset 2, so divergingEpoch validation passes
> >> and
> >> > returns offset 2 to cluster A. In short, relying on “last mirror
> offset”
> >> > cannot resolve the log diverge issue in unclean leader election case.
> >> >
> >> > If instead, we truncate until “last mirror epoch”: 0, the offset 0
> and 1
> >> > in cluster A will be truncated as expected and then log converge with
> >> > cluster B correctly in the end.
> >> >
> >> >
> >> > Also, about JR53.2:
> >> > > JR53.2 "2. The admin client fans out the request to all brokers and
> >> > merges
> >> > the responses client-side."
> >> > This is not ideal. Not every broker hosts a mirror partition. So
> sending
> >> > the request to every broker is wasteful. It's probably better to send
> >> the
> >> > request to a single broker, which can then contact the right
> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >
> >> > While working on this, I found we need to add one more tagged field
> >> > "Forwarded" to the DescribeClusterMirrorsRequest schema because
> >> otherwise
> >> > the receiving broker doesn’t know if this is the request from client
> >> and it
> >> > needs to contact other brokers, or this is just a forwarded request.
> The
> >> > workflow is like this:
> >> > 1. the adminClient sends the DescribeClusterMirrorsRequest to a
> >> > leastLoaded broker, which doesn’t contain the "Forwarded" field.
> >> > 2. The broker receives the request and sends
> >> DescribeClusterMirrorsRequest
> >> > with "Forwarded=true" to the responsible brokers and waits for the
> >> response
> >> > async.
> >> > 3. When all responses are received, merge the results and respond to
> the
> >> > client.
> >> >
> >> > Note: we cannot use ReadMirrorStatesRequest here because it doesn’t
> >> > contain the lag info in the coordinator.
> >> >
> >> > The alternative solution is the adminClient sends
> >> > DescribeClusterMirrorsRequest to the “responsible brokers only” (via
> >> > describeTopics), instead of ALL brokers, and merge the results in the
> >> > client side. This solution doesn’t need to add a new field in the
> >> > DescribeClusterMirrorsRequest and still can avoid unnecessary requests
> >> > sent.
> >> >
> >> > My preference is to keep the merge logic on the admin client side,
> which
> >> > saves the overhead in the broker side.
> >> > What do you think?
> >> >
> >> >
> >> > Thank you,
> >> > Luke
> >> >
> >> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <de...@kafka.apache.org>
> >> > wrote:
> >> >
> >> >> Hi, Luke,
> >> >>
> >> >> Thanks for the reply.
> >> >>
> >> >> JR31.3 It makes sense for the new partition leader to query the
> current
> >> >> mirror state from the local or remote MirrorMetadataManager that owns
> >> the
> >> >> partition's state. However, the wording in the KIP is confusing.
> >> >> MirrorMetadataManager
> >> >> mirror partition state cache should be populated only by one way,
> >> which is
> >> >> replaying __mirror_state partitions this broker leads. By saying "and
> >> by
> >> >> fetching state from remote coordinators via ReadMirrorStates RPCs.",
> >> it's
> >> >> as if the mirror partition state cache on each broker stores the
> state
> >> for
> >> >> all mirror partitions.
> >> >>
> >> >> JR36. I understand the intention now, but I still don't think we need
> >> >> MirrorLeaderEpoch. Let's
> >> >> assume the common case: no unclean leader election in the source
> >> cluster
> >> >> during mirroring. Mirroring only fetches the committed data from the
> >> >> source, which never changes if obtained a second time. In your
> example
> >> >> image, follower broker 1 in the destination cluster can never obtain
> >> the
> >> >> record B(0) because it's never committed in the source and thus is
> >> never
> >> >> exposed to the destination leader. It's possible for broker 1 to
> first
> >> get
> >> >> D(1) which is truncated later when broker 0 follows a new leader. But
> >> the
> >> >> new leader will refetch the same D(1) from the source cluster. So,
> >> it's ok
> >> >> for broker 1 to keep D(1).
> >> >>
> >> >> JR51. Sounds good. Why do we choose to store the last mirror epoch
> >> instead
> >> >> of a last mirror offset?
> >> >>
> >> >> Jun
> >> >>
> >> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com> wrote:
> >> >>
> >> >> > Hi Jun,
> >> >> >
> >> >> > Thanks for the great comments again.
> >> >> > Answering your question below.
> >> >> >
> >> >> > > JR31.3 "Regarding state caching: the mirror partition cache lives
> >> in
> >> >> > MirrorMetadataManager. It is populated in two ways: by replaying
> >> >> > __mirror_state partitions this broker leads, and by fetching state
> >> >> > from remote coordinators via ReadMirrorStates RPCs."
> >> >> > Hmm, a given broker only caches the mirror partition state for
> >> >> partitions
> >> >> > hashed into the __mirror_state partition hosted on this broker,
> >> right?
> >> >> Why
> >> >> > does it need to fetch the state from other remote coordinators?
> >> >> >
> >> >> > > a given broker only caches the mirror partition state for
> >> partitions
> >> >> > hashed into the __mirror_state partition hosted on this broker,
> >> right?
> >> >> >
> >> >> > Right.
> >> >> >
> >> >> > > Why does it need to fetch the state from other remote
> coordinators?
> >> >> >
> >> >> > Because the data partition leader needs to know the current mirror
> >> state
> >> >> > of the partitions it leads. On a leadership change, the metadata
> >> delta
> >> >> is
> >> >> > published to the MirrorMetadataManager, which queries the current
> >> mirror
> >> >> > state for its led partitions from the coordinator (local or
> remote),
> >> and
> >> >> > processes tasks based on that state. This is the reason why it
> >> fetches
> >> >> the
> >> >> > state from other remote coordinator. This part of cache will be
> >> cleared
> >> >> > after the leadership change.
> >> >> >
> >> >> >
> >> >> >
> >> >> > > JR36. I am still not quite sure why we need MirrorLeaderEpoch in
> >> the
> >> >> > Fetch
> >> >> > request. In the current Fetch request, in addition to
> >> >> CurrentLeaderEpoch,
> >> >> > it has another field LastFetchedEpoch, which captures the last
> leader
> >> >> epoch
> >> >> > in the data and is used to resolve the inconsistency between the
> >> >> follower
> >> >> > and the leader. Does it achieve the same purpose as
> >> MirrorLeaderEpoch?
> >> >> >
> >> >> >
> >> >> > Unfortunately, the LastFetchedEpoch cannot be used to do the
> >> validation
> >> >> to
> >> >> > avoid the issue in KAFKA-18723. If we reject any fetched batches
> with
> >> >> epoch
> >> >> > higher than the LastFetchedEpoch, it means the follower can never
> >> accept
> >> >> > new appended batches with higher epoch in the leader node.
> >> >> >
> >> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes in the
> >> fetch
> >> >> > API:
> >> >> > 1. The receiver (leader node) uses it to validate it matches the
> >> leader
> >> >> > epoch in the leader node
> >> >> > 2. The sender (follower node) uses it to validate the epoch in the
> >> >> fetched
> >> >> > batches are all <= this value.
> >> >> >
> >> >> > For purpose (1), the CurrentLeaderEpoch is still serving for
> >> followers
> >> >> in
> >> >> > the destination cluster. For purpose (2), my understanding of the
> >> >> > validation for KAFKA-18723 is like: the follower knows the
> >> >> > CurrentLeaderEpoch X is the highest epoch the leader node should
> >> >> contain in
> >> >> > the log batches when fetch request is sent. Any log batch beyond
> this
> >> >> epoch
> >> >> > in the fetch response means the follower’s metadata is stale. The
> >> >> batches
> >> >> > need to be rejected and follower needs to update metadata for next
> >> >> fetch.
> >> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves this
> >> >> purpose
> >> >> > to let the followers know the highest epoch in the leader node log
> in
> >> >> each
> >> >> > fetch response. Once the info is stale in the follower side, it
> >> updates
> >> >> it
> >> >> > from fetch response, and send next fetch request with correct
> >> >> > MirrorLeaderEpoch to make sure no inconsistent data is possible
> like
> >> >> > KAFKA-18723.
> >> >> >
> >> >> > I tried not to add new field in Fetch API, but this is the best
> >> >> solution I
> >> >> > can think of.
> >> >> >
> >> >> >
> >> >> >
> >> >> > > JR48. "The NumPartitions field is needed so the controller can
> >> create
> >> >> the
> >> >> > destination topic if it does not already exist. When a topic is
> being
> >> >> > mirrored for the first time, the destination cluster has no
> knowledge
> >> >> > of the source topic's partition count, so the request must carry
> it."
> >> >> > Hmm, this seems inconsistent. When a mirror is started, the source
> >> topic
> >> >> > may not exist yet. When the source topic is created later, we can
> >> start
> >> >> > mirroring without requiring the user to provide the number of
> >> >> partitions.
> >> >> >
> >> >> > > When the source topic is created later, we can start mirroring
> >> without
> >> >> > requiring the user to provide the number of partitions.
> >> >> >
> >> >> > This is because this happens in periodical sync up in the
> destination
> >> >> > cluster, and the numPartitions and other topic info in the source
> >> >> cluster
> >> >> > is already retrieved during the sync up process. Also, if users are
> >> >> > creating mirror topics via admin API, this info can also be
> retrieved
> >> >> > inside the admin client if not provided.
> >> >> >
> >> >> >
> >> >> >
> >> >> > > JR51. Start Mirror: I am wondering if we truly need the last
> mirror
> >> >> epoch
> >> >> > for truncation when starting a mirror. When a mirror is stopped at
> >> the
> >> >> > source, we already bump up the leader epoch and write some
> additional
> >> >> > records for cleanup. It seems that we can just use the
> DivergingEpoch
> >> >> > returned in the initial mirror Fetch request to detect the
> diverging
> >> >> epoch
> >> >> > and use that for truncation at the mirror destination.
> >> >> >
> >> >> > That’s a good question. I thought about this before but
> >> unfortunately it
> >> >> > doesn’t work. The reason we need the last mirror epoch record is
> >> because
> >> >> > the destination cluster might not completely sync with the source
> >> >> cluster
> >> >> > before STOPPED. And this un-mirrored data might confuse the
> >> replication
> >> >> > protocol when reverse mirror. For example,
> >> >> >
> >> >> > Cluster A (source) —> Cluster B (dest)
> >> >> > foo-0 leader log foo-0 leader log
> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> > Offset 2: from A (LE: 10)
> >> >> >
> >> >> > While cluster B mirrors until offset 1, the cluster A crashes and
> >> >> failover
> >> >> > to cluster B. B bumps leader epoch to 10, appends a
> MIRROR_PID_RESET
> >> >> with
> >> >> > LE: 10. Then cluster A starts to reverse mirror from cluster B, the
> >> log
> >> >> > will be like this:
> >> >> >
> >> >> > Cluster A (dest) <-- Cluster B (source)
> >> >> > foo-0 leader log: foo-0 leader log:
> >> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
> >> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
> >> >> >
> >> >> > As you can see, cluster A (new destination cluster) doesn’t know
> the
> >> >> > offset 2 is not belonging to the cluster B, which causes
> inconsistent
> >> >> logs.
> >> >> > So like what we described in the KIP, the last mirror epoch is the
> >> >> > synchronization point between source and destination. Any data
> beyond
> >> >> the
> >> >> > point is treated as unknown to the source cluster and should be
> >> >> truncated.
> >> >> >
> >> >> >
> >> >> >
> >> >> > > JR52. Stop Mirror:
> >> >> > > JR52.1 "3. If patterns are provided, the controller removes
> >> matching
> >> >> > entries from mirror.topics.include or adds them to
> >> >> mirror.topics.exclude on
> >> >> > the CLUSTER_MIRROR resource."
> >> >> > Why is it 'or' and not 'and'?
> >> >> >
> >> >> > Yes, that should be AND. KIP updated. Thanks.
> >> >> >
> >> >> >
> >> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
> >> >> MirrorMetadataManager
> >> >> > discovers new source topics matching the persisted include/exclude
> >> >> patterns
> >> >> > and repeats steps 3 through 7 for each."
> >> >> > The broker will send a create topic request to the controller. How
> >> does
> >> >> the
> >> >> > controller associate it with the mirror name to create a
> >> >> > `MirrorTopicStateChangeRecord`?
> >> >> >
> >> >> > No, we don’t send a create topic request to the controller here.
> >> >> Actually
> >> >> > we send a StartMirrorTopics request to the controller, which
> contains
> >> >> the
> >> >> > mirror name and topic metadata like NumPartitions, topicId… etc. So
> >> the
> >> >> > controller can create the non-existed topic as explained in JR48,
> and
> >> >> > associate it to MirrorTopicStateChangeRecord.
> >> >> >
> >> >> >
> >> >> > > JR53. Describe Mirrors:
> >> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors request with
> >> >> optional
> >> >> > mirror names (empty means all mirrors)."
> >> >> > In metadata request, we use a null arrary to represent all topics.
> It
> >> >> would
> >> >> > be useful to be consistent here.
> >> >> >
> >> >> > OK, KIP updated.
> >> >> >
> >> >> >
> >> >> > > JR53.2 "2. The admin client fans out the request to all brokers
> and
> >> >> > merges
> >> >> > the responses client-side."
> >> >> > This is not ideal. Not every broker hosts a mirror partition. So
> >> sending
> >> >> > the request to every broker is wasteful. It's probably better to
> send
> >> >> the
> >> >> > request to a single broker, which can then contact the right
> >> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >
> >> >> > Agree, KIP updated.
> >> >> >
> >> >> >
> >> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
> >> >> PAUSED,
> >> >> > STOPPED the only validate states? The state table has other states
> >> too.
> >> >> > Ditto for MirrorPartitionState.
> >> >> > The DesiredState field in MirrorTopicStateChangeRecord is to define
> >> the
> >> >> > target state the user wants. And the desired state users can assign
> >> is
> >> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as
> >> >> STOPPING
> >> >> > or LOG_TRUNCATION… etc because they are more like intermediate
> >> states.
> >> >> >
> >> >> > About MirrorPartitionState record, it’ll record any state in the
> >> state
> >> >> > table that the current partition is. This is the real state for
> each
> >> >> > partition. It’s like in the kubernetes world, when we change the
> >> replica
> >> >> > field of the deployment yaml from 3 -> 5 (desired state), it won’t
> >> >> increase
> >> >> > to 5 immediately. The current replica (MirrorPartitionState record)
> >> is
> >> >> > still 3, then 4 (maybe), and finally 5.
> >> >> >
> >> >> >
> >> >> > Thank you,
> >> >> > Luke
> >> >> >
> >> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <
> dev@kafka.apache.org
> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> Hi, Fede,
> >> >> >>
> >> >> >> Thanks for the reply. A few more comments.
> >> >> >>
> >> >> >> JR31.3 "Regarding state caching: the mirror partition cache lives
> in
> >> >> >> MirrorMetadataManager. It is populated in two ways: by replaying
> >> >> >> __mirror_state partitions this broker leads, and by fetching state
> >> >> >> from remote coordinators via ReadMirrorStates RPCs."
> >> >> >> Hmm, a given broker only caches the mirror partition state for
> >> >> partitions
> >> >> >> hashed into the __mirror_state partition hosted on this broker,
> >> right?
> >> >> Why
> >> >> >> does it need to fetch the state from other remote coordinators?
> >> >> >>
> >> >> >> JR36. I am still not quite sure why we need MirrorLeaderEpoch in
> the
> >> >> Fetch
> >> >> >> request. In the current Fetch request, in addition to
> >> >> CurrentLeaderEpoch,
> >> >> >> it has another field LastFetchedEpoch, which captures the last
> >> leader
> >> >> >> epoch
> >> >> >> in the data and is used to resolve the inconsistency between the
> >> >> follower
> >> >> >> and the leader. Does it achieve the same purpose as
> >> MirrorLeaderEpoch?
> >> >> >>
> >> >> >> JR48. "The NumPartitions field is needed so the controller can
> >> create
> >> >> the
> >> >> >> destination topic if it does not already exist. When a topic is
> >> being
> >> >> >> mirrored for the first time, the destination cluster has no
> >> knowledge
> >> >> >> of the source topic's partition count, so the request must carry
> >> it."
> >> >> >> Hmm, this seems inconsistent. When a mirror is started, the source
> >> >> topic
> >> >> >> may not exist yet. When the source topic is created later, we can
> >> start
> >> >> >> mirroring without requiring the user to provide the number of
> >> >> partitions.
> >> >> >>
> >> >> >> JR51. Start Mirror: I am wondering if we truly need the last
> mirror
> >> >> epoch
> >> >> >> for truncation when starting a mirror. When a mirror is stopped at
> >> the
> >> >> >> source, we already bump up the leader epoch and write some
> >> additional
> >> >> >> records for cleanup. It seems that we can just use the
> >> DivergingEpoch
> >> >> >> returned in the initial mirror Fetch request to detect the
> diverging
> >> >> epoch
> >> >> >> and use that for truncation at the mirror destination.
> >> >> >>
> >> >> >> JR52. Stop Mirror:
> >> >> >> JR52.1 "3. If patterns are provided, the controller removes
> matching
> >> >> >> entries from mirror.topics.include or adds them to
> >> >> mirror.topics.exclude
> >> >> >> on
> >> >> >> the CLUSTER_MIRROR resource."
> >> >> >> Why is it 'or' and not 'and'?
> >> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
> >> MirrorMetadataManager
> >> >> >> discovers new source topics matching the persisted include/exclude
> >> >> >> patterns
> >> >> >> and repeats steps 3 through 7 for each."
> >> >> >> The broker will send a create topic request to the controller. How
> >> does
> >> >> >> the
> >> >> >> controller associate it with the mirror name to create a
> >> >> >> `MirrorTopicStateChangeRecord`?
> >> >> >>
> >> >> >> JR53. Describe Mirrors:
> >> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors request with
> >> >> optional
> >> >> >> mirror names (empty means all mirrors)."
> >> >> >> In metadata request, we use a null arrary to represent all topics.
> >> It
> >> >> >> would
> >> >> >> be useful to be consistent here.
> >> >> >> JR53.2 "2. The admin client fans out the request to all brokers
> and
> >> >> merges
> >> >> >> the responses client-side."
> >> >> >> This is not ideal. Not every broker hosts a mirror partition. So
> >> >> sending
> >> >> >> the request to every broker is wasteful. It's probably better to
> >> send
> >> >> the
> >> >> >> request to a single broker, which can then contact the right
> >> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >> >>
> >> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
> >> PAUSED,
> >> >> >> STOPPED the only validate states? The state table has other states
> >> too.
> >> >> >> Ditto for MirrorPartitionState.
> >> >> >>
> >> >> >> Jun
> >> >> >>
> >> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
> >> fedevaleri@gmail.com>
> >> >> >> wrote:
> >> >> >>
> >> >> >> > Hi Andrew, yes, validation is called at the start of
> createTopic.
> >> If
> >> >> >> > the topic ID is invalid, the request fails with INVALID_REQUEST.
> >> If
> >> >> >> > the topic ID is already used by a different topic name, it fails
> >> with
> >> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual topic
> in
> >> the
> >> >> >> > batch fails; other topics in the same CreateTopics request
> >> continue
> >> >> >> > processing normally. Added this information to the KIP.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
> >> >> >> schofielaj@apache.org>
> >> >> >> > wrote:
> >> >> >> > >
> >> >> >> > > Hi Fede,
> >> >> >> > > One more small question.
> >> >> >> > >
> >> >> >> > > AS37: If the controller fails validation for the
> >> MirrorInfo.TopicId
> >> >> >> > added to the CreateTopic request, what does it do? I suspect
> that
> >> it
> >> >> >> fails
> >> >> >> > the request with a particular error code.
> >> >> >> > >
> >> >> >> > > Thanks,
> >> >> >> > > Andrew
> >> >> >> > >
> >> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> >> >> >> > > > Hi all,
> >> >> >> > > >
> >> >> >> > > > We updated the KIP to include Coordinator Runtime
> >> configurations
> >> >> and
> >> >> >> > > > the coordinator state transition validation paragraph. The
> >> latter
> >> >> >> > > > covers the per-partition epoch fencing mechanism (leader
> epoch
> >> >> and
> >> >> >> > > > state epoch) that complements the existing per-topic
> >> StateOffset
> >> >> >> > > > fencing for lifecycle operations.
> >> >> >> > > >
> >> >> >> > > > Thanks,
> >> >> >> > > > Fede
> >> >> >> > > >
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply.
JR36. Sounds good. Should we remove MirrorLeaderEpochExceededException?
JR51. Sounds good. Do we plan to always set lastFetchedEpoch in the mirror
fetch request issued to the source cluster, independent of the value of
mirror.support.unclean.leader.election? It's needed to detect divergence
during failover. However, if it's also set when
mirror.support.unclean.leader.election=false,
an unclean leader election in the source cluster can halt the mirroring
since we don’t do anything with the returned divergeEpoch in that mode
and continue
the next fetch.
JR55. The DescribeClusterMirrorsRequest still returns both the mirror
partition state and the mirror partition leader state. The problem is that
if a caller only needs one of the state, the other state becomes redundant.
For example, when starting the mirror, to obtain LME, we only need the
mirror partition state. Here is an alternative design: We expose the mirror
partition state through ReadMirrorState. The client will use the
FindCoordinatorRequest
to find the right mirror partition coordinator to issue that request. We
expose the mirror partition leader state through the ReadClusterMirrorLags
request. The client can use the metadata request to find the partition
leader to issue that request.
In the describing mirror flow, the admin client first issues the
ListMirrorRequest to find all the mirrors and the topics being mirrored. It
then issues the metadata request to find the partitions for those topics
and issues a FindCoordinatorRequest to find the mirror partition state
coordinators. Finally, it issues ReadMirrorState to the mirror partition
state coordinator and ReadClusterMirrorLags to the partition leader.
In the starting mirror flow, the partition leader in the destination
cluster issues a ListMirrorRequest to find the mirror name based on its
cluster id. It then issues a FindCoordinatorRequest to find the mirror
partition state coordinators. Finally, it issues ReadMirrorState to the
mirror partition state coordinator to get the LME.
We can also improve the ListMirror request to accept a filter based on
mirror name, source cluster ID and topic name.
JR56. Could you list the ACL needed for each new RPC?
JR57. In the command workflow, it sometimes says "The user sends" and some
other times says "The admin client sends". It's better to use "The admin
client" consistently in all those places.
Jun
On Thu, Aug 6, 2026 at 5:13 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for your comments.
> Answering them below.
>
>
> > JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
> designed for the unclean leader election support in Cluster Mirroring
> because the unclean leader election will break the assumption that the
> committed data won’t be truncated."
> If the source cluster has an unclean leader election, the destination
> cluster will detect the inconsistency and truncate the log at the
> destination leader. We can just
> extend mirror.support.unclean.leader.election to cover this case, i.e., we
> wait until all replicas' log end offsets are reduced to the target offset
> before resuming mirroring. This avoids the inconsistency.
>
> If the destination cluster has an unclean leader election, it will
> re-fetch the same data from the source. There won't be any inconsistency
> either.
> So, I think we can avoid adding MirrorLeaderEpoch.
>
>
> Yes, that works, too. I agree this is better because the unclean leader
> election is rare, we don't want to occupy a field just for it.
>
> So to achieve this, we have to:
> 1. add one more state: “ULE_LOG_TRUNCATION”,
> 2. When the destination cluster leader completes a log truncation after a
> fetch response due to divergeEpoch, we know it must be caused by unclean
> leader election in the source cluster because we only fetch committed
> records. If “mirror.support.unclean.leader.election=false”, we don’t do
> anything and continue the next fetch. If
> “mirror.support.unclean.leader.election=true”, we remove the mirror fetcher
> and move the partition to “ULE_LOG_TRUNCATION” state.
> 3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be caught up
> into ISR and reduced to the expected LEO.
> 4. Move to “MIRRORING” state and create the mirror fetcher
>
> So that we don’t need the “MirrorLeaderEpoch” field in Fetch API.
> KIP is updated.
>
>
> > JR51. Make sense. To truncate based on LME, we need to issue another RPC
> to translate the LME to an offset. It would be useful to include that in
> the workflow description.
>
> About the additional RPC to translate the LME to the offset (i.e.
> OffsetForLeaderEpochRequest), actually we don’t rely on this to do the
> offset translation. We use the local leader epoch cache in the destination
> cluster to translate the epoch to save the RPC call. This works because of
> an invariant the LME truncation protocol maintains: for any leader epoch ≤
> LME, that epoch's data belongs to exactly one cluster, either it originated
> there, or it was mirrored. This holds even across repeated mirror ->
> reverse-mirror -> mirror cycles, since each direction's LOG_TRUNCATION
> phase re-establishes the invariant before that cluster starts mirroring.
> With this invariant, the replication protocol can find out the diverge
> point and truncate the log correctly when mirroring. For example:
>
> Cluster A (source) --> Cluster B (dest)
> foo-0 leader log foo-0 leader log
> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> Offset 2: A (LE: 0)
> Offset 3: A (LE: 1)
>
> Cluster A crashes while Cluster B mirrors to offset 1. Failover to B and
> store the LME=0 in cluster B. When cluster A recovers and starts the
> reverse mirroring, it’ll query the LME from B and get back LME: 0. Then
> cluster A gets the end offset for epoch 0 from leaderEpochCache, and gets
> offset 2. So offset 3 is truncated. The log is like this now:
>
> Cluster A (dest) <-- Cluster B (source)
> foo-0 leader log foo-0 leader log
> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
> Offset 3: B (LE: 10)
>
> When mirror starts, cluster A sends the fetch request with {
> lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the divergingEpoch check in
> the cluster B queries the endOffset for leader epoch 0, and found the end
> offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns fetch
> response with divergingEpoch info to cluster A for log truncation. In the
> end, offset 2 in cluster A will be truncated and then log converges with
> cluster B.
>
>
>
> > JR53.2 The current DescribeClusterMirrorsRequest is a bit strange
> because it mixes querying the mirror partition state with querying the
> mirror partition leader. Another option is to use ReadMirrorStatesRequest
> to query the mirror partition state and a new request to query the mirror
> partition leader. If we want the adminClient to query the right ClusterMirrorCoordinator
> directly, we can extend the FindCoordinatorRequest to support that.
>
>
> Good suggestion. Yes, we need to fan out to all brokers because we have to
> collect info from mirror partition leaders and mirror coordinators. If we
> create a new API to contact mirror partition leaders, it will make the API
> cleaner and clearer. So we will do this:
>
> a. Remove SourceOffset, DestinationOffset, and Lag fields from
> DescribeClusterMirror request/response
>
> b. Add a new API: ReadClusterMirrorLags API to request for SourceOffset,
> DestinationOffset, and Lag info.
>
> The workflow for admin client describeClusterMirror will be like this:
> 1. adminClient sends describeClusterMirror to the leastLoadedNode broker
> 2. The received broker collects the current mirrored partitions, and
> groups them by coordinator node and by partition leader node.
> 3. The broker sends ReadMirrorState request to the coordinator nodes to
> get the mirror state related info
> 4. The broker sends ReadClusterMirrorLags request to the mirror partition
> leader nodes to get the mirror lag, source offset, destination offset
> (which updates on every fetch response).
> 5. The broker merge all the response into describeClusterMirror response
> and send to admin client
>
> KIP is updated. Thanks for the suggestion.
>
>
> > JR55. For __mirror_state topic, could we describe whether it is
> compacted?
>
> Yes, it is a compacted topic. I made it much clearer in the
> “ClusterMirrorCoordinator” section.
>
>
> Thank you,
> Luke
>
>
> On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Luke,
>>
>> Thanks for the reply.
>>
>> JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
>> designed for the unclean leader election support in Cluster Mirroring
>> because the unclean leader election will break the assumption that the
>> committed data won’t be truncated."
>> If the source cluster has an unclean leader election, the destination
>> cluster will detect the inconsistency and truncate the log at the
>> destination leader. We can just
>> extend mirror.support.unclean.leader.election to cover this case, i.e., we
>> wait until all replicas' log end offsets are reduced to the target offset
>> before resuming mirroring. This avoids the inconsistency.
>>
>> If the destination cluster has an unclean leader election, it will
>> re-fetch
>> the same data from the source. There won't be any inconsistency either.
>>
>> So, I think we can avoid adding MirrorLeaderEpoch.
>>
>> JR51. Make sense. To truncate based on LME, we need to issue another RPC
>> to
>> translate the LME to an offset. It would be useful to include that in the
>> workflow description.
>>
>> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange because
>> it mixes querying the mirror partition state with querying the mirror
>> partition leader. Another option is to use ReadMirrorStatesRequest to
>> query
>> the mirror partition state and a new request to query the mirror partition
>> leader. If we want the adminClient to query the right
>> ClusterMirrorCoordinator
>> directly, we can extend the FindCoordinatorRequest to support that.
>>
>> JR55. For __mirror_state topic, could we describe whether it is compacted?
>>
>> Jun
>>
>> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for the comments.
>> > Answering them below:
>> >
>> > > JR31.3 It makes sense for the new partition leader to query the
>> current
>> > mirror state from the local or remote MirrorMetadataManager that owns
>> the
>> > partition's state. However, the wording in the KIP is confusing.
>> > MirrorMetadataManager
>> > mirror partition state cache should be populated only by one way, which
>> is
>> > replaying __mirror_state partitions this broker leads. By saying "and by
>> > fetching state from remote coordinators via ReadMirrorStates RPCs.",
>> it's
>> > as if the mirror partition state cache on each broker stores the state
>> for
>> > all mirror partitions.
>> >
>> > You’re right. KIP is updated to make it clear.
>> >
>> >
>> > > JR36. I understand the intention now, but I still don't think we need
>> > MirrorLeaderEpoch. Let's
>> > assume the common case: no unclean leader election in the source cluster
>> > during mirroring. Mirroring only fetches the committed data from the
>> > source, which never changes if obtained a second time. In your example
>> > image, follower broker 1 in the destination cluster can never obtain the
>> > record B(0) because it's never committed in the source and thus is never
>> > exposed to the destination leader. It's possible for broker 1 to first
>> get
>> > D(1) which is truncated later when broker 0 follows a new leader. But
>> the
>> > new leader will refetch the same D(1) from the source cluster. So, it's
>> ok
>> > for broker 1 to keep D(1).
>> >
>> >
>> > Yes, you are correct. Because mirroring only fetches the committed data
>> > from the source, the MirrorLeaderEpoch is unnecessary for normal
>> mirroring.
>> > Thanks for the clarification. So I think we can say, the
>> > “MirrorLeaderEpoch” is specifically designed for the unclean leader
>> > election support in Cluster Mirroring because the unclean leader
>> election
>> > will break the assumption that the committed data won’t be truncated.
>> >
>> > Given that, we could gate the MirrorLeaderEpoch field by only setting it
>> > in the Fetch request when "mirror.support.unclean.leader.election=true"
>> > (false by default), and skip it entirely for normal mirroring. What do
>> you
>> > think?
>> >
>> >
>> > > JR51. Sounds good. Why do we choose to store the last mirror epoch
>> > instead
>> > of a last mirror offset?
>> >
>> > Yes, originally we stored the “last mirror offset” instead of “last
>> mirror
>> > epoch”. But later, we found it doesn’t work for unclean leader election
>> > cases. (yes, `unclean leader election` again :) ) For example:
>> >
>> > Cluster A (source) --> Cluster B (dest)
>> > foo-0 leader log foo-0 leader log
>> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
>> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>> >
>> > Cluster B is mirroring from A. And A crashes and failover to B. B stores
>> > the “last mirror offset” as 1.
>> > When cluster A recovers, somehow the unclean leader election happens,
>> log
>> > is empty in the new leader, and then some more leadership changes to
>> make
>> > the leader epoch become 10. Then it starts the reverse mirror from B.
>> The
>> > log looks like this:
>> >
>> > Cluster A (dest) <-- Cluster B (source)
>> > foo-0 leader log foo-0 leader log
>> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
>> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
>> > Offset 2: pid_res (LE: 10)
>> >
>> > When A starts mirroring from B, log is truncated to last mirror offset:
>> 1
>> > (no-op), and then fetch from cluster B. The 1st fetch request is like
>> this:
>> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2, … }. The
>> > divergingEpoch check in the cluster B queries the endOffset for leader
>> > epoch 10 (in Partition#readRecords), and found the end offset for LE 10
>> is
>> > 2 in B, which is >= fetchOffset 2, so divergingEpoch validation passes
>> and
>> > returns offset 2 to cluster A. In short, relying on “last mirror offset”
>> > cannot resolve the log diverge issue in unclean leader election case.
>> >
>> > If instead, we truncate until “last mirror epoch”: 0, the offset 0 and 1
>> > in cluster A will be truncated as expected and then log converge with
>> > cluster B correctly in the end.
>> >
>> >
>> > Also, about JR53.2:
>> > > JR53.2 "2. The admin client fans out the request to all brokers and
>> > merges
>> > the responses client-side."
>> > This is not ideal. Not every broker hosts a mirror partition. So sending
>> > the request to every broker is wasteful. It's probably better to send
>> the
>> > request to a single broker, which can then contact the right
>> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >
>> > While working on this, I found we need to add one more tagged field
>> > "Forwarded" to the DescribeClusterMirrorsRequest schema because
>> otherwise
>> > the receiving broker doesn’t know if this is the request from client
>> and it
>> > needs to contact other brokers, or this is just a forwarded request. The
>> > workflow is like this:
>> > 1. the adminClient sends the DescribeClusterMirrorsRequest to a
>> > leastLoaded broker, which doesn’t contain the "Forwarded" field.
>> > 2. The broker receives the request and sends
>> DescribeClusterMirrorsRequest
>> > with "Forwarded=true" to the responsible brokers and waits for the
>> response
>> > async.
>> > 3. When all responses are received, merge the results and respond to the
>> > client.
>> >
>> > Note: we cannot use ReadMirrorStatesRequest here because it doesn’t
>> > contain the lag info in the coordinator.
>> >
>> > The alternative solution is the adminClient sends
>> > DescribeClusterMirrorsRequest to the “responsible brokers only” (via
>> > describeTopics), instead of ALL brokers, and merge the results in the
>> > client side. This solution doesn’t need to add a new field in the
>> > DescribeClusterMirrorsRequest and still can avoid unnecessary requests
>> > sent.
>> >
>> > My preference is to keep the merge logic on the admin client side, which
>> > saves the overhead in the broker side.
>> > What do you think?
>> >
>> >
>> > Thank you,
>> > Luke
>> >
>> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <de...@kafka.apache.org>
>> > wrote:
>> >
>> >> Hi, Luke,
>> >>
>> >> Thanks for the reply.
>> >>
>> >> JR31.3 It makes sense for the new partition leader to query the current
>> >> mirror state from the local or remote MirrorMetadataManager that owns
>> the
>> >> partition's state. However, the wording in the KIP is confusing.
>> >> MirrorMetadataManager
>> >> mirror partition state cache should be populated only by one way,
>> which is
>> >> replaying __mirror_state partitions this broker leads. By saying "and
>> by
>> >> fetching state from remote coordinators via ReadMirrorStates RPCs.",
>> it's
>> >> as if the mirror partition state cache on each broker stores the state
>> for
>> >> all mirror partitions.
>> >>
>> >> JR36. I understand the intention now, but I still don't think we need
>> >> MirrorLeaderEpoch. Let's
>> >> assume the common case: no unclean leader election in the source
>> cluster
>> >> during mirroring. Mirroring only fetches the committed data from the
>> >> source, which never changes if obtained a second time. In your example
>> >> image, follower broker 1 in the destination cluster can never obtain
>> the
>> >> record B(0) because it's never committed in the source and thus is
>> never
>> >> exposed to the destination leader. It's possible for broker 1 to first
>> get
>> >> D(1) which is truncated later when broker 0 follows a new leader. But
>> the
>> >> new leader will refetch the same D(1) from the source cluster. So,
>> it's ok
>> >> for broker 1 to keep D(1).
>> >>
>> >> JR51. Sounds good. Why do we choose to store the last mirror epoch
>> instead
>> >> of a last mirror offset?
>> >>
>> >> Jun
>> >>
>> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com> wrote:
>> >>
>> >> > Hi Jun,
>> >> >
>> >> > Thanks for the great comments again.
>> >> > Answering your question below.
>> >> >
>> >> > > JR31.3 "Regarding state caching: the mirror partition cache lives
>> in
>> >> > MirrorMetadataManager. It is populated in two ways: by replaying
>> >> > __mirror_state partitions this broker leads, and by fetching state
>> >> > from remote coordinators via ReadMirrorStates RPCs."
>> >> > Hmm, a given broker only caches the mirror partition state for
>> >> partitions
>> >> > hashed into the __mirror_state partition hosted on this broker,
>> right?
>> >> Why
>> >> > does it need to fetch the state from other remote coordinators?
>> >> >
>> >> > > a given broker only caches the mirror partition state for
>> partitions
>> >> > hashed into the __mirror_state partition hosted on this broker,
>> right?
>> >> >
>> >> > Right.
>> >> >
>> >> > > Why does it need to fetch the state from other remote coordinators?
>> >> >
>> >> > Because the data partition leader needs to know the current mirror
>> state
>> >> > of the partitions it leads. On a leadership change, the metadata
>> delta
>> >> is
>> >> > published to the MirrorMetadataManager, which queries the current
>> mirror
>> >> > state for its led partitions from the coordinator (local or remote),
>> and
>> >> > processes tasks based on that state. This is the reason why it
>> fetches
>> >> the
>> >> > state from other remote coordinator. This part of cache will be
>> cleared
>> >> > after the leadership change.
>> >> >
>> >> >
>> >> >
>> >> > > JR36. I am still not quite sure why we need MirrorLeaderEpoch in
>> the
>> >> > Fetch
>> >> > request. In the current Fetch request, in addition to
>> >> CurrentLeaderEpoch,
>> >> > it has another field LastFetchedEpoch, which captures the last leader
>> >> epoch
>> >> > in the data and is used to resolve the inconsistency between the
>> >> follower
>> >> > and the leader. Does it achieve the same purpose as
>> MirrorLeaderEpoch?
>> >> >
>> >> >
>> >> > Unfortunately, the LastFetchedEpoch cannot be used to do the
>> validation
>> >> to
>> >> > avoid the issue in KAFKA-18723. If we reject any fetched batches with
>> >> epoch
>> >> > higher than the LastFetchedEpoch, it means the follower can never
>> accept
>> >> > new appended batches with higher epoch in the leader node.
>> >> >
>> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes in the
>> fetch
>> >> > API:
>> >> > 1. The receiver (leader node) uses it to validate it matches the
>> leader
>> >> > epoch in the leader node
>> >> > 2. The sender (follower node) uses it to validate the epoch in the
>> >> fetched
>> >> > batches are all <= this value.
>> >> >
>> >> > For purpose (1), the CurrentLeaderEpoch is still serving for
>> followers
>> >> in
>> >> > the destination cluster. For purpose (2), my understanding of the
>> >> > validation for KAFKA-18723 is like: the follower knows the
>> >> > CurrentLeaderEpoch X is the highest epoch the leader node should
>> >> contain in
>> >> > the log batches when fetch request is sent. Any log batch beyond this
>> >> epoch
>> >> > in the fetch response means the follower’s metadata is stale. The
>> >> batches
>> >> > need to be rejected and follower needs to update metadata for next
>> >> fetch.
>> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves this
>> >> purpose
>> >> > to let the followers know the highest epoch in the leader node log in
>> >> each
>> >> > fetch response. Once the info is stale in the follower side, it
>> updates
>> >> it
>> >> > from fetch response, and send next fetch request with correct
>> >> > MirrorLeaderEpoch to make sure no inconsistent data is possible like
>> >> > KAFKA-18723.
>> >> >
>> >> > I tried not to add new field in Fetch API, but this is the best
>> >> solution I
>> >> > can think of.
>> >> >
>> >> >
>> >> >
>> >> > > JR48. "The NumPartitions field is needed so the controller can
>> create
>> >> the
>> >> > destination topic if it does not already exist. When a topic is being
>> >> > mirrored for the first time, the destination cluster has no knowledge
>> >> > of the source topic's partition count, so the request must carry it."
>> >> > Hmm, this seems inconsistent. When a mirror is started, the source
>> topic
>> >> > may not exist yet. When the source topic is created later, we can
>> start
>> >> > mirroring without requiring the user to provide the number of
>> >> partitions.
>> >> >
>> >> > > When the source topic is created later, we can start mirroring
>> without
>> >> > requiring the user to provide the number of partitions.
>> >> >
>> >> > This is because this happens in periodical sync up in the destination
>> >> > cluster, and the numPartitions and other topic info in the source
>> >> cluster
>> >> > is already retrieved during the sync up process. Also, if users are
>> >> > creating mirror topics via admin API, this info can also be retrieved
>> >> > inside the admin client if not provided.
>> >> >
>> >> >
>> >> >
>> >> > > JR51. Start Mirror: I am wondering if we truly need the last mirror
>> >> epoch
>> >> > for truncation when starting a mirror. When a mirror is stopped at
>> the
>> >> > source, we already bump up the leader epoch and write some additional
>> >> > records for cleanup. It seems that we can just use the DivergingEpoch
>> >> > returned in the initial mirror Fetch request to detect the diverging
>> >> epoch
>> >> > and use that for truncation at the mirror destination.
>> >> >
>> >> > That’s a good question. I thought about this before but
>> unfortunately it
>> >> > doesn’t work. The reason we need the last mirror epoch record is
>> because
>> >> > the destination cluster might not completely sync with the source
>> >> cluster
>> >> > before STOPPED. And this un-mirrored data might confuse the
>> replication
>> >> > protocol when reverse mirror. For example,
>> >> >
>> >> > Cluster A (source) —> Cluster B (dest)
>> >> > foo-0 leader log foo-0 leader log
>> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> > Offset 2: from A (LE: 10)
>> >> >
>> >> > While cluster B mirrors until offset 1, the cluster A crashes and
>> >> failover
>> >> > to cluster B. B bumps leader epoch to 10, appends a MIRROR_PID_RESET
>> >> with
>> >> > LE: 10. Then cluster A starts to reverse mirror from cluster B, the
>> log
>> >> > will be like this:
>> >> >
>> >> > Cluster A (dest) <-- Cluster B (source)
>> >> > foo-0 leader log: foo-0 leader log:
>> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
>> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
>> >> >
>> >> > As you can see, cluster A (new destination cluster) doesn’t know the
>> >> > offset 2 is not belonging to the cluster B, which causes inconsistent
>> >> logs.
>> >> > So like what we described in the KIP, the last mirror epoch is the
>> >> > synchronization point between source and destination. Any data beyond
>> >> the
>> >> > point is treated as unknown to the source cluster and should be
>> >> truncated.
>> >> >
>> >> >
>> >> >
>> >> > > JR52. Stop Mirror:
>> >> > > JR52.1 "3. If patterns are provided, the controller removes
>> matching
>> >> > entries from mirror.topics.include or adds them to
>> >> mirror.topics.exclude on
>> >> > the CLUSTER_MIRROR resource."
>> >> > Why is it 'or' and not 'and'?
>> >> >
>> >> > Yes, that should be AND. KIP updated. Thanks.
>> >> >
>> >> >
>> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
>> >> MirrorMetadataManager
>> >> > discovers new source topics matching the persisted include/exclude
>> >> patterns
>> >> > and repeats steps 3 through 7 for each."
>> >> > The broker will send a create topic request to the controller. How
>> does
>> >> the
>> >> > controller associate it with the mirror name to create a
>> >> > `MirrorTopicStateChangeRecord`?
>> >> >
>> >> > No, we don’t send a create topic request to the controller here.
>> >> Actually
>> >> > we send a StartMirrorTopics request to the controller, which contains
>> >> the
>> >> > mirror name and topic metadata like NumPartitions, topicId… etc. So
>> the
>> >> > controller can create the non-existed topic as explained in JR48, and
>> >> > associate it to MirrorTopicStateChangeRecord.
>> >> >
>> >> >
>> >> > > JR53. Describe Mirrors:
>> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors request with
>> >> optional
>> >> > mirror names (empty means all mirrors)."
>> >> > In metadata request, we use a null arrary to represent all topics. It
>> >> would
>> >> > be useful to be consistent here.
>> >> >
>> >> > OK, KIP updated.
>> >> >
>> >> >
>> >> > > JR53.2 "2. The admin client fans out the request to all brokers and
>> >> > merges
>> >> > the responses client-side."
>> >> > This is not ideal. Not every broker hosts a mirror partition. So
>> sending
>> >> > the request to every broker is wasteful. It's probably better to send
>> >> the
>> >> > request to a single broker, which can then contact the right
>> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >
>> >> > Agree, KIP updated.
>> >> >
>> >> >
>> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
>> >> PAUSED,
>> >> > STOPPED the only validate states? The state table has other states
>> too.
>> >> > Ditto for MirrorPartitionState.
>> >> > The DesiredState field in MirrorTopicStateChangeRecord is to define
>> the
>> >> > target state the user wants. And the desired state users can assign
>> is
>> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as
>> >> STOPPING
>> >> > or LOG_TRUNCATION… etc because they are more like intermediate
>> states.
>> >> >
>> >> > About MirrorPartitionState record, it’ll record any state in the
>> state
>> >> > table that the current partition is. This is the real state for each
>> >> > partition. It’s like in the kubernetes world, when we change the
>> replica
>> >> > field of the deployment yaml from 3 -> 5 (desired state), it won’t
>> >> increase
>> >> > to 5 immediately. The current replica (MirrorPartitionState record)
>> is
>> >> > still 3, then 4 (maybe), and finally 5.
>> >> >
>> >> >
>> >> > Thank you,
>> >> > Luke
>> >> >
>> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <dev@kafka.apache.org
>> >
>> >> > wrote:
>> >> >
>> >> >> Hi, Fede,
>> >> >>
>> >> >> Thanks for the reply. A few more comments.
>> >> >>
>> >> >> JR31.3 "Regarding state caching: the mirror partition cache lives in
>> >> >> MirrorMetadataManager. It is populated in two ways: by replaying
>> >> >> __mirror_state partitions this broker leads, and by fetching state
>> >> >> from remote coordinators via ReadMirrorStates RPCs."
>> >> >> Hmm, a given broker only caches the mirror partition state for
>> >> partitions
>> >> >> hashed into the __mirror_state partition hosted on this broker,
>> right?
>> >> Why
>> >> >> does it need to fetch the state from other remote coordinators?
>> >> >>
>> >> >> JR36. I am still not quite sure why we need MirrorLeaderEpoch in the
>> >> Fetch
>> >> >> request. In the current Fetch request, in addition to
>> >> CurrentLeaderEpoch,
>> >> >> it has another field LastFetchedEpoch, which captures the last
>> leader
>> >> >> epoch
>> >> >> in the data and is used to resolve the inconsistency between the
>> >> follower
>> >> >> and the leader. Does it achieve the same purpose as
>> MirrorLeaderEpoch?
>> >> >>
>> >> >> JR48. "The NumPartitions field is needed so the controller can
>> create
>> >> the
>> >> >> destination topic if it does not already exist. When a topic is
>> being
>> >> >> mirrored for the first time, the destination cluster has no
>> knowledge
>> >> >> of the source topic's partition count, so the request must carry
>> it."
>> >> >> Hmm, this seems inconsistent. When a mirror is started, the source
>> >> topic
>> >> >> may not exist yet. When the source topic is created later, we can
>> start
>> >> >> mirroring without requiring the user to provide the number of
>> >> partitions.
>> >> >>
>> >> >> JR51. Start Mirror: I am wondering if we truly need the last mirror
>> >> epoch
>> >> >> for truncation when starting a mirror. When a mirror is stopped at
>> the
>> >> >> source, we already bump up the leader epoch and write some
>> additional
>> >> >> records for cleanup. It seems that we can just use the
>> DivergingEpoch
>> >> >> returned in the initial mirror Fetch request to detect the diverging
>> >> epoch
>> >> >> and use that for truncation at the mirror destination.
>> >> >>
>> >> >> JR52. Stop Mirror:
>> >> >> JR52.1 "3. If patterns are provided, the controller removes matching
>> >> >> entries from mirror.topics.include or adds them to
>> >> mirror.topics.exclude
>> >> >> on
>> >> >> the CLUSTER_MIRROR resource."
>> >> >> Why is it 'or' and not 'and'?
>> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
>> MirrorMetadataManager
>> >> >> discovers new source topics matching the persisted include/exclude
>> >> >> patterns
>> >> >> and repeats steps 3 through 7 for each."
>> >> >> The broker will send a create topic request to the controller. How
>> does
>> >> >> the
>> >> >> controller associate it with the mirror name to create a
>> >> >> `MirrorTopicStateChangeRecord`?
>> >> >>
>> >> >> JR53. Describe Mirrors:
>> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors request with
>> >> optional
>> >> >> mirror names (empty means all mirrors)."
>> >> >> In metadata request, we use a null arrary to represent all topics.
>> It
>> >> >> would
>> >> >> be useful to be consistent here.
>> >> >> JR53.2 "2. The admin client fans out the request to all brokers and
>> >> merges
>> >> >> the responses client-side."
>> >> >> This is not ideal. Not every broker hosts a mirror partition. So
>> >> sending
>> >> >> the request to every broker is wasteful. It's probably better to
>> send
>> >> the
>> >> >> request to a single broker, which can then contact the right
>> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
>> >> >>
>> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
>> PAUSED,
>> >> >> STOPPED the only validate states? The state table has other states
>> too.
>> >> >> Ditto for MirrorPartitionState.
>> >> >>
>> >> >> Jun
>> >> >>
>> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
>> fedevaleri@gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >> > Hi Andrew, yes, validation is called at the start of createTopic.
>> If
>> >> >> > the topic ID is invalid, the request fails with INVALID_REQUEST.
>> If
>> >> >> > the topic ID is already used by a different topic name, it fails
>> with
>> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual topic in
>> the
>> >> >> > batch fails; other topics in the same CreateTopics request
>> continue
>> >> >> > processing normally. Added this information to the KIP.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
>> >> >> schofielaj@apache.org>
>> >> >> > wrote:
>> >> >> > >
>> >> >> > > Hi Fede,
>> >> >> > > One more small question.
>> >> >> > >
>> >> >> > > AS37: If the controller fails validation for the
>> MirrorInfo.TopicId
>> >> >> > added to the CreateTopic request, what does it do? I suspect that
>> it
>> >> >> fails
>> >> >> > the request with a particular error code.
>> >> >> > >
>> >> >> > > Thanks,
>> >> >> > > Andrew
>> >> >> > >
>> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> >> >> > > > Hi all,
>> >> >> > > >
>> >> >> > > > We updated the KIP to include Coordinator Runtime
>> configurations
>> >> and
>> >> >> > > > the coordinator state transition validation paragraph. The
>> latter
>> >> >> > > > covers the per-partition epoch fencing mechanism (leader epoch
>> >> and
>> >> >> > > > state epoch) that complements the existing per-topic
>> StateOffset
>> >> >> > > > fencing for lifecycle operations.
>> >> >> > > >
>> >> >> > > > Thanks,
>> >> >> > > > Fede
>> >> >> > > >
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for your comments.
Answering them below.
> JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
designed for the unclean leader election support in Cluster Mirroring
because the unclean leader election will break the assumption that the
committed data won’t be truncated."
If the source cluster has an unclean leader election, the destination
cluster will detect the inconsistency and truncate the log at the
destination leader. We can just
extend mirror.support.unclean.leader.election to cover this case, i.e., we
wait until all replicas' log end offsets are reduced to the target offset
before resuming mirroring. This avoids the inconsistency.
If the destination cluster has an unclean leader election, it will re-fetch
the same data from the source. There won't be any inconsistency either.
So, I think we can avoid adding MirrorLeaderEpoch.
Yes, that works, too. I agree this is better because the unclean leader
election is rare, we don't want to occupy a field just for it.
So to achieve this, we have to:
1. add one more state: “ULE_LOG_TRUNCATION”,
2. When the destination cluster leader completes a log truncation after a
fetch response due to divergeEpoch, we know it must be caused by unclean
leader election in the source cluster because we only fetch committed
records. If “mirror.support.unclean.leader.election=false”, we don’t do
anything and continue the next fetch. If
“mirror.support.unclean.leader.election=true”, we remove the mirror fetcher
and move the partition to “ULE_LOG_TRUNCATION” state.
3. In “ULE_LOG_TRUNCATION” state, we wait for ALL replicas to be caught up
into ISR and reduced to the expected LEO.
4. Move to “MIRRORING” state and create the mirror fetcher
So that we don’t need the “MirrorLeaderEpoch” field in Fetch API.
KIP is updated.
> JR51. Make sense. To truncate based on LME, we need to issue another RPC
to translate the LME to an offset. It would be useful to include that in
the workflow description.
About the additional RPC to translate the LME to the offset (i.e.
OffsetForLeaderEpochRequest), actually we don’t rely on this to do the
offset translation. We use the local leader epoch cache in the destination
cluster to translate the epoch to save the RPC call. This works because of
an invariant the LME truncation protocol maintains: for any leader epoch ≤
LME, that epoch's data belongs to exactly one cluster, either it originated
there, or it was mirrored. This holds even across repeated mirror ->
reverse-mirror -> mirror cycles, since each direction's LOG_TRUNCATION
phase re-establishes the invariant before that cluster starts mirroring.
With this invariant, the replication protocol can find out the diverge
point and truncate the log correctly when mirroring. For example:
Cluster A (source) --> Cluster B (dest)
foo-0 leader log foo-0 leader log
Offset 0: A (LE: 0) Offset 0: A (LE: 0)
Offset 1: A (LE: 0) Offset 1: A (LE: 0)
Offset 2: A (LE: 0)
Offset 3: A (LE: 1)
Cluster A crashes while Cluster B mirrors to offset 1. Failover to B and
store the LME=0 in cluster B. When cluster A recovers and starts the
reverse mirroring, it’ll query the LME from B and get back LME: 0. Then
cluster A gets the end offset for epoch 0 from leaderEpochCache, and gets
offset 2. So offset 3 is truncated. The log is like this now:
Cluster A (dest) <-- Cluster B (source)
foo-0 leader log foo-0 leader log
Offset 0: A (LE: 0) Offset 0: A (LE: 0)
Offset 1: A (LE: 0) Offset 1: A (LE: 0)
Offset 2: A (LE: 0) Offset 2: pid_res (LE: 10)
Offset 3: B (LE: 10)
When mirror starts, cluster A sends the fetch request with {
lastFetchedEpoch: 0, fetchOffset: 3, … } to B, the divergingEpoch check in
the cluster B queries the endOffset for leader epoch 0, and found the end
offset for LE 0 is 1 in B, which is < fetchOffset 3, so returns fetch
response with divergingEpoch info to cluster A for log truncation. In the
end, offset 2 in cluster A will be truncated and then log converges with
cluster B.
> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange because
it mixes querying the mirror partition state with querying the mirror
partition leader. Another option is to use ReadMirrorStatesRequest to query
the mirror partition state and a new request to query the mirror partition
leader. If we want the adminClient to query the right ClusterMirrorCoordinator
directly, we can extend the FindCoordinatorRequest to support that.
Good suggestion. Yes, we need to fan out to all brokers because we have to
collect info from mirror partition leaders and mirror coordinators. If we
create a new API to contact mirror partition leaders, it will make the API
cleaner and clearer. So we will do this:
a. Remove SourceOffset, DestinationOffset, and Lag fields from
DescribeClusterMirror request/response
b. Add a new API: ReadClusterMirrorLags API to request for SourceOffset,
DestinationOffset, and Lag info.
The workflow for admin client describeClusterMirror will be like this:
1. adminClient sends describeClusterMirror to the leastLoadedNode broker
2. The received broker collects the current mirrored partitions, and groups
them by coordinator node and by partition leader node.
3. The broker sends ReadMirrorState request to the coordinator nodes to get
the mirror state related info
4. The broker sends ReadClusterMirrorLags request to the mirror partition
leader nodes to get the mirror lag, source offset, destination offset
(which updates on every fetch response).
5. The broker merge all the response into describeClusterMirror response
and send to admin client
KIP is updated. Thanks for the suggestion.
> JR55. For __mirror_state topic, could we describe whether it is compacted?
Yes, it is a compacted topic. I made it much clearer in the
“ClusterMirrorCoordinator” section.
Thank you,
Luke
On Thu, Aug 6, 2026 at 6:47 AM Jun Rao via dev <de...@kafka.apache.org> wrote:
> Hi, Luke,
>
> Thanks for the reply.
>
> JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
> designed for the unclean leader election support in Cluster Mirroring
> because the unclean leader election will break the assumption that the
> committed data won’t be truncated."
> If the source cluster has an unclean leader election, the destination
> cluster will detect the inconsistency and truncate the log at the
> destination leader. We can just
> extend mirror.support.unclean.leader.election to cover this case, i.e., we
> wait until all replicas' log end offsets are reduced to the target offset
> before resuming mirroring. This avoids the inconsistency.
>
> If the destination cluster has an unclean leader election, it will re-fetch
> the same data from the source. There won't be any inconsistency either.
>
> So, I think we can avoid adding MirrorLeaderEpoch.
>
> JR51. Make sense. To truncate based on LME, we need to issue another RPC to
> translate the LME to an offset. It would be useful to include that in the
> workflow description.
>
> JR53.2 The current DescribeClusterMirrorsRequest is a bit strange because
> it mixes querying the mirror partition state with querying the mirror
> partition leader. Another option is to use ReadMirrorStatesRequest to query
> the mirror partition state and a new request to query the mirror partition
> leader. If we want the adminClient to query the right
> ClusterMirrorCoordinator
> directly, we can extend the FindCoordinatorRequest to support that.
>
> JR55. For __mirror_state topic, could we describe whether it is compacted?
>
> Jun
>
> On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the comments.
> > Answering them below:
> >
> > > JR31.3 It makes sense for the new partition leader to query the current
> > mirror state from the local or remote MirrorMetadataManager that owns the
> > partition's state. However, the wording in the KIP is confusing.
> > MirrorMetadataManager
> > mirror partition state cache should be populated only by one way, which
> is
> > replaying __mirror_state partitions this broker leads. By saying "and by
> > fetching state from remote coordinators via ReadMirrorStates RPCs.", it's
> > as if the mirror partition state cache on each broker stores the state
> for
> > all mirror partitions.
> >
> > You’re right. KIP is updated to make it clear.
> >
> >
> > > JR36. I understand the intention now, but I still don't think we need
> > MirrorLeaderEpoch. Let's
> > assume the common case: no unclean leader election in the source cluster
> > during mirroring. Mirroring only fetches the committed data from the
> > source, which never changes if obtained a second time. In your example
> > image, follower broker 1 in the destination cluster can never obtain the
> > record B(0) because it's never committed in the source and thus is never
> > exposed to the destination leader. It's possible for broker 1 to first
> get
> > D(1) which is truncated later when broker 0 follows a new leader. But the
> > new leader will refetch the same D(1) from the source cluster. So, it's
> ok
> > for broker 1 to keep D(1).
> >
> >
> > Yes, you are correct. Because mirroring only fetches the committed data
> > from the source, the MirrorLeaderEpoch is unnecessary for normal
> mirroring.
> > Thanks for the clarification. So I think we can say, the
> > “MirrorLeaderEpoch” is specifically designed for the unclean leader
> > election support in Cluster Mirroring because the unclean leader election
> > will break the assumption that the committed data won’t be truncated.
> >
> > Given that, we could gate the MirrorLeaderEpoch field by only setting it
> > in the Fetch request when "mirror.support.unclean.leader.election=true"
> > (false by default), and skip it entirely for normal mirroring. What do
> you
> > think?
> >
> >
> > > JR51. Sounds good. Why do we choose to store the last mirror epoch
> > instead
> > of a last mirror offset?
> >
> > Yes, originally we stored the “last mirror offset” instead of “last
> mirror
> > epoch”. But later, we found it doesn’t work for unclean leader election
> > cases. (yes, `unclean leader election` again :) ) For example:
> >
> > Cluster A (source) --> Cluster B (dest)
> > foo-0 leader log foo-0 leader log
> > Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> > Offset 1: A (LE: 0) Offset 1: A (LE: 0)
> >
> > Cluster B is mirroring from A. And A crashes and failover to B. B stores
> > the “last mirror offset” as 1.
> > When cluster A recovers, somehow the unclean leader election happens, log
> > is empty in the new leader, and then some more leadership changes to make
> > the leader epoch become 10. Then it starts the reverse mirror from B. The
> > log looks like this:
> >
> > Cluster A (dest) <-- Cluster B (source)
> > foo-0 leader log foo-0 leader log
> > Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> > Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> > Offset 2: pid_res (LE: 10)
> >
> > When A starts mirroring from B, log is truncated to last mirror offset: 1
> > (no-op), and then fetch from cluster B. The 1st fetch request is like
> this:
> > { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2, … }. The
> > divergingEpoch check in the cluster B queries the endOffset for leader
> > epoch 10 (in Partition#readRecords), and found the end offset for LE 10
> is
> > 2 in B, which is >= fetchOffset 2, so divergingEpoch validation passes
> and
> > returns offset 2 to cluster A. In short, relying on “last mirror offset”
> > cannot resolve the log diverge issue in unclean leader election case.
> >
> > If instead, we truncate until “last mirror epoch”: 0, the offset 0 and 1
> > in cluster A will be truncated as expected and then log converge with
> > cluster B correctly in the end.
> >
> >
> > Also, about JR53.2:
> > > JR53.2 "2. The admin client fans out the request to all brokers and
> > merges
> > the responses client-side."
> > This is not ideal. Not every broker hosts a mirror partition. So sending
> > the request to every broker is wasteful. It's probably better to send the
> > request to a single broker, which can then contact the right
> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >
> > While working on this, I found we need to add one more tagged field
> > "Forwarded" to the DescribeClusterMirrorsRequest schema because otherwise
> > the receiving broker doesn’t know if this is the request from client and
> it
> > needs to contact other brokers, or this is just a forwarded request. The
> > workflow is like this:
> > 1. the adminClient sends the DescribeClusterMirrorsRequest to a
> > leastLoaded broker, which doesn’t contain the "Forwarded" field.
> > 2. The broker receives the request and sends
> DescribeClusterMirrorsRequest
> > with "Forwarded=true" to the responsible brokers and waits for the
> response
> > async.
> > 3. When all responses are received, merge the results and respond to the
> > client.
> >
> > Note: we cannot use ReadMirrorStatesRequest here because it doesn’t
> > contain the lag info in the coordinator.
> >
> > The alternative solution is the adminClient sends
> > DescribeClusterMirrorsRequest to the “responsible brokers only” (via
> > describeTopics), instead of ALL brokers, and merge the results in the
> > client side. This solution doesn’t need to add a new field in the
> > DescribeClusterMirrorsRequest and still can avoid unnecessary requests
> > sent.
> >
> > My preference is to keep the merge logic on the admin client side, which
> > saves the overhead in the broker side.
> > What do you think?
> >
> >
> > Thank you,
> > Luke
> >
> > On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Luke,
> >>
> >> Thanks for the reply.
> >>
> >> JR31.3 It makes sense for the new partition leader to query the current
> >> mirror state from the local or remote MirrorMetadataManager that owns
> the
> >> partition's state. However, the wording in the KIP is confusing.
> >> MirrorMetadataManager
> >> mirror partition state cache should be populated only by one way, which
> is
> >> replaying __mirror_state partitions this broker leads. By saying "and by
> >> fetching state from remote coordinators via ReadMirrorStates RPCs.",
> it's
> >> as if the mirror partition state cache on each broker stores the state
> for
> >> all mirror partitions.
> >>
> >> JR36. I understand the intention now, but I still don't think we need
> >> MirrorLeaderEpoch. Let's
> >> assume the common case: no unclean leader election in the source cluster
> >> during mirroring. Mirroring only fetches the committed data from the
> >> source, which never changes if obtained a second time. In your example
> >> image, follower broker 1 in the destination cluster can never obtain the
> >> record B(0) because it's never committed in the source and thus is never
> >> exposed to the destination leader. It's possible for broker 1 to first
> get
> >> D(1) which is truncated later when broker 0 follows a new leader. But
> the
> >> new leader will refetch the same D(1) from the source cluster. So, it's
> ok
> >> for broker 1 to keep D(1).
> >>
> >> JR51. Sounds good. Why do we choose to store the last mirror epoch
> instead
> >> of a last mirror offset?
> >>
> >> Jun
> >>
> >> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thanks for the great comments again.
> >> > Answering your question below.
> >> >
> >> > > JR31.3 "Regarding state caching: the mirror partition cache lives in
> >> > MirrorMetadataManager. It is populated in two ways: by replaying
> >> > __mirror_state partitions this broker leads, and by fetching state
> >> > from remote coordinators via ReadMirrorStates RPCs."
> >> > Hmm, a given broker only caches the mirror partition state for
> >> partitions
> >> > hashed into the __mirror_state partition hosted on this broker, right?
> >> Why
> >> > does it need to fetch the state from other remote coordinators?
> >> >
> >> > > a given broker only caches the mirror partition state for partitions
> >> > hashed into the __mirror_state partition hosted on this broker, right?
> >> >
> >> > Right.
> >> >
> >> > > Why does it need to fetch the state from other remote coordinators?
> >> >
> >> > Because the data partition leader needs to know the current mirror
> state
> >> > of the partitions it leads. On a leadership change, the metadata delta
> >> is
> >> > published to the MirrorMetadataManager, which queries the current
> mirror
> >> > state for its led partitions from the coordinator (local or remote),
> and
> >> > processes tasks based on that state. This is the reason why it fetches
> >> the
> >> > state from other remote coordinator. This part of cache will be
> cleared
> >> > after the leadership change.
> >> >
> >> >
> >> >
> >> > > JR36. I am still not quite sure why we need MirrorLeaderEpoch in the
> >> > Fetch
> >> > request. In the current Fetch request, in addition to
> >> CurrentLeaderEpoch,
> >> > it has another field LastFetchedEpoch, which captures the last leader
> >> epoch
> >> > in the data and is used to resolve the inconsistency between the
> >> follower
> >> > and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
> >> >
> >> >
> >> > Unfortunately, the LastFetchedEpoch cannot be used to do the
> validation
> >> to
> >> > avoid the issue in KAFKA-18723. If we reject any fetched batches with
> >> epoch
> >> > higher than the LastFetchedEpoch, it means the follower can never
> accept
> >> > new appended batches with higher epoch in the leader node.
> >> >
> >> > Currently, the CurrentLeaderEpoch field serves 2 purposes in the fetch
> >> > API:
> >> > 1. The receiver (leader node) uses it to validate it matches the
> leader
> >> > epoch in the leader node
> >> > 2. The sender (follower node) uses it to validate the epoch in the
> >> fetched
> >> > batches are all <= this value.
> >> >
> >> > For purpose (1), the CurrentLeaderEpoch is still serving for followers
> >> in
> >> > the destination cluster. For purpose (2), my understanding of the
> >> > validation for KAFKA-18723 is like: the follower knows the
> >> > CurrentLeaderEpoch X is the highest epoch the leader node should
> >> contain in
> >> > the log batches when fetch request is sent. Any log batch beyond this
> >> epoch
> >> > in the fetch response means the follower’s metadata is stale. The
> >> batches
> >> > need to be rejected and follower needs to update metadata for next
> >> fetch.
> >> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves this
> >> purpose
> >> > to let the followers know the highest epoch in the leader node log in
> >> each
> >> > fetch response. Once the info is stale in the follower side, it
> updates
> >> it
> >> > from fetch response, and send next fetch request with correct
> >> > MirrorLeaderEpoch to make sure no inconsistent data is possible like
> >> > KAFKA-18723.
> >> >
> >> > I tried not to add new field in Fetch API, but this is the best
> >> solution I
> >> > can think of.
> >> >
> >> >
> >> >
> >> > > JR48. "The NumPartitions field is needed so the controller can
> create
> >> the
> >> > destination topic if it does not already exist. When a topic is being
> >> > mirrored for the first time, the destination cluster has no knowledge
> >> > of the source topic's partition count, so the request must carry it."
> >> > Hmm, this seems inconsistent. When a mirror is started, the source
> topic
> >> > may not exist yet. When the source topic is created later, we can
> start
> >> > mirroring without requiring the user to provide the number of
> >> partitions.
> >> >
> >> > > When the source topic is created later, we can start mirroring
> without
> >> > requiring the user to provide the number of partitions.
> >> >
> >> > This is because this happens in periodical sync up in the destination
> >> > cluster, and the numPartitions and other topic info in the source
> >> cluster
> >> > is already retrieved during the sync up process. Also, if users are
> >> > creating mirror topics via admin API, this info can also be retrieved
> >> > inside the admin client if not provided.
> >> >
> >> >
> >> >
> >> > > JR51. Start Mirror: I am wondering if we truly need the last mirror
> >> epoch
> >> > for truncation when starting a mirror. When a mirror is stopped at the
> >> > source, we already bump up the leader epoch and write some additional
> >> > records for cleanup. It seems that we can just use the DivergingEpoch
> >> > returned in the initial mirror Fetch request to detect the diverging
> >> epoch
> >> > and use that for truncation at the mirror destination.
> >> >
> >> > That’s a good question. I thought about this before but unfortunately
> it
> >> > doesn’t work. The reason we need the last mirror epoch record is
> because
> >> > the destination cluster might not completely sync with the source
> >> cluster
> >> > before STOPPED. And this un-mirrored data might confuse the
> replication
> >> > protocol when reverse mirror. For example,
> >> >
> >> > Cluster A (source) —> Cluster B (dest)
> >> > foo-0 leader log foo-0 leader log
> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> > Offset 2: from A (LE: 10)
> >> >
> >> > While cluster B mirrors until offset 1, the cluster A crashes and
> >> failover
> >> > to cluster B. B bumps leader epoch to 10, appends a MIRROR_PID_RESET
> >> with
> >> > LE: 10. Then cluster A starts to reverse mirror from cluster B, the
> log
> >> > will be like this:
> >> >
> >> > Cluster A (dest) <-- Cluster B (source)
> >> > foo-0 leader log: foo-0 leader log:
> >> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> >> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> >> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
> >> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
> >> >
> >> > As you can see, cluster A (new destination cluster) doesn’t know the
> >> > offset 2 is not belonging to the cluster B, which causes inconsistent
> >> logs.
> >> > So like what we described in the KIP, the last mirror epoch is the
> >> > synchronization point between source and destination. Any data beyond
> >> the
> >> > point is treated as unknown to the source cluster and should be
> >> truncated.
> >> >
> >> >
> >> >
> >> > > JR52. Stop Mirror:
> >> > > JR52.1 "3. If patterns are provided, the controller removes matching
> >> > entries from mirror.topics.include or adds them to
> >> mirror.topics.exclude on
> >> > the CLUSTER_MIRROR resource."
> >> > Why is it 'or' and not 'and'?
> >> >
> >> > Yes, that should be AND. KIP updated. Thanks.
> >> >
> >> >
> >> > > JR52.2 "7. On subsequent metadata refresh cycles,
> >> MirrorMetadataManager
> >> > discovers new source topics matching the persisted include/exclude
> >> patterns
> >> > and repeats steps 3 through 7 for each."
> >> > The broker will send a create topic request to the controller. How
> does
> >> the
> >> > controller associate it with the mirror name to create a
> >> > `MirrorTopicStateChangeRecord`?
> >> >
> >> > No, we don’t send a create topic request to the controller here.
> >> Actually
> >> > we send a StartMirrorTopics request to the controller, which contains
> >> the
> >> > mirror name and topic metadata like NumPartitions, topicId… etc. So
> the
> >> > controller can create the non-existed topic as explained in JR48, and
> >> > associate it to MirrorTopicStateChangeRecord.
> >> >
> >> >
> >> > > JR53. Describe Mirrors:
> >> > > JR53.1 "1. The user sends a DescribeClusterMirrors request with
> >> optional
> >> > mirror names (empty means all mirrors)."
> >> > In metadata request, we use a null arrary to represent all topics. It
> >> would
> >> > be useful to be consistent here.
> >> >
> >> > OK, KIP updated.
> >> >
> >> >
> >> > > JR53.2 "2. The admin client fans out the request to all brokers and
> >> > merges
> >> > the responses client-side."
> >> > This is not ideal. Not every broker hosts a mirror partition. So
> sending
> >> > the request to every broker is wasteful. It's probably better to send
> >> the
> >> > request to a single broker, which can then contact the right
> >> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >
> >> > Agree, KIP updated.
> >> >
> >> >
> >> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
> >> PAUSED,
> >> > STOPPED the only validate states? The state table has other states
> too.
> >> > Ditto for MirrorPartitionState.
> >> > The DesiredState field in MirrorTopicStateChangeRecord is to define
> the
> >> > target state the user wants. And the desired state users can assign is
> >> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as
> >> STOPPING
> >> > or LOG_TRUNCATION… etc because they are more like intermediate states.
> >> >
> >> > About MirrorPartitionState record, it’ll record any state in the state
> >> > table that the current partition is. This is the real state for each
> >> > partition. It’s like in the kubernetes world, when we change the
> replica
> >> > field of the deployment yaml from 3 -> 5 (desired state), it won’t
> >> increase
> >> > to 5 immediately. The current replica (MirrorPartitionState record) is
> >> > still 3, then 4 (maybe), and finally 5.
> >> >
> >> >
> >> > Thank you,
> >> > Luke
> >> >
> >> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <de...@kafka.apache.org>
> >> > wrote:
> >> >
> >> >> Hi, Fede,
> >> >>
> >> >> Thanks for the reply. A few more comments.
> >> >>
> >> >> JR31.3 "Regarding state caching: the mirror partition cache lives in
> >> >> MirrorMetadataManager. It is populated in two ways: by replaying
> >> >> __mirror_state partitions this broker leads, and by fetching state
> >> >> from remote coordinators via ReadMirrorStates RPCs."
> >> >> Hmm, a given broker only caches the mirror partition state for
> >> partitions
> >> >> hashed into the __mirror_state partition hosted on this broker,
> right?
> >> Why
> >> >> does it need to fetch the state from other remote coordinators?
> >> >>
> >> >> JR36. I am still not quite sure why we need MirrorLeaderEpoch in the
> >> Fetch
> >> >> request. In the current Fetch request, in addition to
> >> CurrentLeaderEpoch,
> >> >> it has another field LastFetchedEpoch, which captures the last leader
> >> >> epoch
> >> >> in the data and is used to resolve the inconsistency between the
> >> follower
> >> >> and the leader. Does it achieve the same purpose as
> MirrorLeaderEpoch?
> >> >>
> >> >> JR48. "The NumPartitions field is needed so the controller can create
> >> the
> >> >> destination topic if it does not already exist. When a topic is being
> >> >> mirrored for the first time, the destination cluster has no knowledge
> >> >> of the source topic's partition count, so the request must carry it."
> >> >> Hmm, this seems inconsistent. When a mirror is started, the source
> >> topic
> >> >> may not exist yet. When the source topic is created later, we can
> start
> >> >> mirroring without requiring the user to provide the number of
> >> partitions.
> >> >>
> >> >> JR51. Start Mirror: I am wondering if we truly need the last mirror
> >> epoch
> >> >> for truncation when starting a mirror. When a mirror is stopped at
> the
> >> >> source, we already bump up the leader epoch and write some additional
> >> >> records for cleanup. It seems that we can just use the DivergingEpoch
> >> >> returned in the initial mirror Fetch request to detect the diverging
> >> epoch
> >> >> and use that for truncation at the mirror destination.
> >> >>
> >> >> JR52. Stop Mirror:
> >> >> JR52.1 "3. If patterns are provided, the controller removes matching
> >> >> entries from mirror.topics.include or adds them to
> >> mirror.topics.exclude
> >> >> on
> >> >> the CLUSTER_MIRROR resource."
> >> >> Why is it 'or' and not 'and'?
> >> >> JR52.2 "7. On subsequent metadata refresh cycles,
> MirrorMetadataManager
> >> >> discovers new source topics matching the persisted include/exclude
> >> >> patterns
> >> >> and repeats steps 3 through 7 for each."
> >> >> The broker will send a create topic request to the controller. How
> does
> >> >> the
> >> >> controller associate it with the mirror name to create a
> >> >> `MirrorTopicStateChangeRecord`?
> >> >>
> >> >> JR53. Describe Mirrors:
> >> >> JR53.1 "1. The user sends a DescribeClusterMirrors request with
> >> optional
> >> >> mirror names (empty means all mirrors)."
> >> >> In metadata request, we use a null arrary to represent all topics. It
> >> >> would
> >> >> be useful to be consistent here.
> >> >> JR53.2 "2. The admin client fans out the request to all brokers and
> >> merges
> >> >> the responses client-side."
> >> >> This is not ideal. Not every broker hosts a mirror partition. So
> >> sending
> >> >> the request to every broker is wasteful. It's probably better to send
> >> the
> >> >> request to a single broker, which can then contact the right
> >> >> MirrorFetcherManager and ClusterMirrorCoordinator.
> >> >>
> >> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
> PAUSED,
> >> >> STOPPED the only validate states? The state table has other states
> too.
> >> >> Ditto for MirrorPartitionState.
> >> >>
> >> >> Jun
> >> >>
> >> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <
> fedevaleri@gmail.com>
> >> >> wrote:
> >> >>
> >> >> > Hi Andrew, yes, validation is called at the start of createTopic.
> If
> >> >> > the topic ID is invalid, the request fails with INVALID_REQUEST. If
> >> >> > the topic ID is already used by a different topic name, it fails
> with
> >> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual topic in
> the
> >> >> > batch fails; other topics in the same CreateTopics request continue
> >> >> > processing normally. Added this information to the KIP.
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
> >> >> schofielaj@apache.org>
> >> >> > wrote:
> >> >> > >
> >> >> > > Hi Fede,
> >> >> > > One more small question.
> >> >> > >
> >> >> > > AS37: If the controller fails validation for the
> MirrorInfo.TopicId
> >> >> > added to the CreateTopic request, what does it do? I suspect that
> it
> >> >> fails
> >> >> > the request with a particular error code.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Andrew
> >> >> > >
> >> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> >> >> > > > Hi all,
> >> >> > > >
> >> >> > > > We updated the KIP to include Coordinator Runtime
> configurations
> >> and
> >> >> > > > the coordinator state transition validation paragraph. The
> latter
> >> >> > > > covers the per-partition epoch fencing mechanism (leader epoch
> >> and
> >> >> > > > state epoch) that complements the existing per-topic
> StateOffset
> >> >> > > > fencing for lifecycle operations.
> >> >> > > >
> >> >> > > > Thanks,
> >> >> > > > Fede
> >> >> > > >
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply.
JR36. "So I think we can say, the “MirrorLeaderEpoch” is specifically
designed for the unclean leader election support in Cluster Mirroring
because the unclean leader election will break the assumption that the
committed data won’t be truncated."
If the source cluster has an unclean leader election, the destination
cluster will detect the inconsistency and truncate the log at the
destination leader. We can just
extend mirror.support.unclean.leader.election to cover this case, i.e., we
wait until all replicas' log end offsets are reduced to the target offset
before resuming mirroring. This avoids the inconsistency.
If the destination cluster has an unclean leader election, it will re-fetch
the same data from the source. There won't be any inconsistency either.
So, I think we can avoid adding MirrorLeaderEpoch.
JR51. Make sense. To truncate based on LME, we need to issue another RPC to
translate the LME to an offset. It would be useful to include that in the
workflow description.
JR53.2 The current DescribeClusterMirrorsRequest is a bit strange because
it mixes querying the mirror partition state with querying the mirror
partition leader. Another option is to use ReadMirrorStatesRequest to query
the mirror partition state and a new request to query the mirror partition
leader. If we want the adminClient to query the right ClusterMirrorCoordinator
directly, we can extend the FindCoordinatorRequest to support that.
JR55. For __mirror_state topic, could we describe whether it is compacted?
Jun
On Wed, Aug 5, 2026 at 4:51 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the comments.
> Answering them below:
>
> > JR31.3 It makes sense for the new partition leader to query the current
> mirror state from the local or remote MirrorMetadataManager that owns the
> partition's state. However, the wording in the KIP is confusing.
> MirrorMetadataManager
> mirror partition state cache should be populated only by one way, which is
> replaying __mirror_state partitions this broker leads. By saying "and by
> fetching state from remote coordinators via ReadMirrorStates RPCs.", it's
> as if the mirror partition state cache on each broker stores the state for
> all mirror partitions.
>
> You’re right. KIP is updated to make it clear.
>
>
> > JR36. I understand the intention now, but I still don't think we need
> MirrorLeaderEpoch. Let's
> assume the common case: no unclean leader election in the source cluster
> during mirroring. Mirroring only fetches the committed data from the
> source, which never changes if obtained a second time. In your example
> image, follower broker 1 in the destination cluster can never obtain the
> record B(0) because it's never committed in the source and thus is never
> exposed to the destination leader. It's possible for broker 1 to first get
> D(1) which is truncated later when broker 0 follows a new leader. But the
> new leader will refetch the same D(1) from the source cluster. So, it's ok
> for broker 1 to keep D(1).
>
>
> Yes, you are correct. Because mirroring only fetches the committed data
> from the source, the MirrorLeaderEpoch is unnecessary for normal mirroring.
> Thanks for the clarification. So I think we can say, the
> “MirrorLeaderEpoch” is specifically designed for the unclean leader
> election support in Cluster Mirroring because the unclean leader election
> will break the assumption that the committed data won’t be truncated.
>
> Given that, we could gate the MirrorLeaderEpoch field by only setting it
> in the Fetch request when "mirror.support.unclean.leader.election=true"
> (false by default), and skip it entirely for normal mirroring. What do you
> think?
>
>
> > JR51. Sounds good. Why do we choose to store the last mirror epoch
> instead
> of a last mirror offset?
>
> Yes, originally we stored the “last mirror offset” instead of “last mirror
> epoch”. But later, we found it doesn’t work for unclean leader election
> cases. (yes, `unclean leader election` again :) ) For example:
>
> Cluster A (source) --> Cluster B (dest)
> foo-0 leader log foo-0 leader log
> Offset 0: A (LE: 0) Offset 0: A (LE: 0)
> Offset 1: A (LE: 0) Offset 1: A (LE: 0)
>
> Cluster B is mirroring from A. And A crashes and failover to B. B stores
> the “last mirror offset” as 1.
> When cluster A recovers, somehow the unclean leader election happens, log
> is empty in the new leader, and then some more leadership changes to make
> the leader epoch become 10. Then it starts the reverse mirror from B. The
> log looks like this:
>
> Cluster A (dest) <-- Cluster B (source)
> foo-0 leader log foo-0 leader log
> Offset 0: B (LE: 10) Offset 0: A (LE: 0)
> Offset 1: B (LE: 10) Offset 1: A (LE: 0)
> Offset 2: pid_res (LE: 10)
>
> When A starts mirroring from B, log is truncated to last mirror offset: 1
> (no-op), and then fetch from cluster B. The 1st fetch request is like this:
> { currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2, … }. The
> divergingEpoch check in the cluster B queries the endOffset for leader
> epoch 10 (in Partition#readRecords), and found the end offset for LE 10 is
> 2 in B, which is >= fetchOffset 2, so divergingEpoch validation passes and
> returns offset 2 to cluster A. In short, relying on “last mirror offset”
> cannot resolve the log diverge issue in unclean leader election case.
>
> If instead, we truncate until “last mirror epoch”: 0, the offset 0 and 1
> in cluster A will be truncated as expected and then log converge with
> cluster B correctly in the end.
>
>
> Also, about JR53.2:
> > JR53.2 "2. The admin client fans out the request to all brokers and
> merges
> the responses client-side."
> This is not ideal. Not every broker hosts a mirror partition. So sending
> the request to every broker is wasteful. It's probably better to send the
> request to a single broker, which can then contact the right
> MirrorFetcherManager and ClusterMirrorCoordinator.
>
> While working on this, I found we need to add one more tagged field
> "Forwarded" to the DescribeClusterMirrorsRequest schema because otherwise
> the receiving broker doesn’t know if this is the request from client and it
> needs to contact other brokers, or this is just a forwarded request. The
> workflow is like this:
> 1. the adminClient sends the DescribeClusterMirrorsRequest to a
> leastLoaded broker, which doesn’t contain the "Forwarded" field.
> 2. The broker receives the request and sends DescribeClusterMirrorsRequest
> with "Forwarded=true" to the responsible brokers and waits for the response
> async.
> 3. When all responses are received, merge the results and respond to the
> client.
>
> Note: we cannot use ReadMirrorStatesRequest here because it doesn’t
> contain the lag info in the coordinator.
>
> The alternative solution is the adminClient sends
> DescribeClusterMirrorsRequest to the “responsible brokers only” (via
> describeTopics), instead of ALL brokers, and merge the results in the
> client side. This solution doesn’t need to add a new field in the
> DescribeClusterMirrorsRequest and still can avoid unnecessary requests
> sent.
>
> My preference is to keep the merge logic on the admin client side, which
> saves the overhead in the broker side.
> What do you think?
>
>
> Thank you,
> Luke
>
> On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Luke,
>>
>> Thanks for the reply.
>>
>> JR31.3 It makes sense for the new partition leader to query the current
>> mirror state from the local or remote MirrorMetadataManager that owns the
>> partition's state. However, the wording in the KIP is confusing.
>> MirrorMetadataManager
>> mirror partition state cache should be populated only by one way, which is
>> replaying __mirror_state partitions this broker leads. By saying "and by
>> fetching state from remote coordinators via ReadMirrorStates RPCs.", it's
>> as if the mirror partition state cache on each broker stores the state for
>> all mirror partitions.
>>
>> JR36. I understand the intention now, but I still don't think we need
>> MirrorLeaderEpoch. Let's
>> assume the common case: no unclean leader election in the source cluster
>> during mirroring. Mirroring only fetches the committed data from the
>> source, which never changes if obtained a second time. In your example
>> image, follower broker 1 in the destination cluster can never obtain the
>> record B(0) because it's never committed in the source and thus is never
>> exposed to the destination leader. It's possible for broker 1 to first get
>> D(1) which is truncated later when broker 0 follows a new leader. But the
>> new leader will refetch the same D(1) from the source cluster. So, it's ok
>> for broker 1 to keep D(1).
>>
>> JR51. Sounds good. Why do we choose to store the last mirror epoch instead
>> of a last mirror offset?
>>
>> Jun
>>
>> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thanks for the great comments again.
>> > Answering your question below.
>> >
>> > > JR31.3 "Regarding state caching: the mirror partition cache lives in
>> > MirrorMetadataManager. It is populated in two ways: by replaying
>> > __mirror_state partitions this broker leads, and by fetching state
>> > from remote coordinators via ReadMirrorStates RPCs."
>> > Hmm, a given broker only caches the mirror partition state for
>> partitions
>> > hashed into the __mirror_state partition hosted on this broker, right?
>> Why
>> > does it need to fetch the state from other remote coordinators?
>> >
>> > > a given broker only caches the mirror partition state for partitions
>> > hashed into the __mirror_state partition hosted on this broker, right?
>> >
>> > Right.
>> >
>> > > Why does it need to fetch the state from other remote coordinators?
>> >
>> > Because the data partition leader needs to know the current mirror state
>> > of the partitions it leads. On a leadership change, the metadata delta
>> is
>> > published to the MirrorMetadataManager, which queries the current mirror
>> > state for its led partitions from the coordinator (local or remote), and
>> > processes tasks based on that state. This is the reason why it fetches
>> the
>> > state from other remote coordinator. This part of cache will be cleared
>> > after the leadership change.
>> >
>> >
>> >
>> > > JR36. I am still not quite sure why we need MirrorLeaderEpoch in the
>> > Fetch
>> > request. In the current Fetch request, in addition to
>> CurrentLeaderEpoch,
>> > it has another field LastFetchedEpoch, which captures the last leader
>> epoch
>> > in the data and is used to resolve the inconsistency between the
>> follower
>> > and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
>> >
>> >
>> > Unfortunately, the LastFetchedEpoch cannot be used to do the validation
>> to
>> > avoid the issue in KAFKA-18723. If we reject any fetched batches with
>> epoch
>> > higher than the LastFetchedEpoch, it means the follower can never accept
>> > new appended batches with higher epoch in the leader node.
>> >
>> > Currently, the CurrentLeaderEpoch field serves 2 purposes in the fetch
>> > API:
>> > 1. The receiver (leader node) uses it to validate it matches the leader
>> > epoch in the leader node
>> > 2. The sender (follower node) uses it to validate the epoch in the
>> fetched
>> > batches are all <= this value.
>> >
>> > For purpose (1), the CurrentLeaderEpoch is still serving for followers
>> in
>> > the destination cluster. For purpose (2), my understanding of the
>> > validation for KAFKA-18723 is like: the follower knows the
>> > CurrentLeaderEpoch X is the highest epoch the leader node should
>> contain in
>> > the log batches when fetch request is sent. Any log batch beyond this
>> epoch
>> > in the fetch response means the follower’s metadata is stale. The
>> batches
>> > need to be rejected and follower needs to update metadata for next
>> fetch.
>> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves this
>> purpose
>> > to let the followers know the highest epoch in the leader node log in
>> each
>> > fetch response. Once the info is stale in the follower side, it updates
>> it
>> > from fetch response, and send next fetch request with correct
>> > MirrorLeaderEpoch to make sure no inconsistent data is possible like
>> > KAFKA-18723.
>> >
>> > I tried not to add new field in Fetch API, but this is the best
>> solution I
>> > can think of.
>> >
>> >
>> >
>> > > JR48. "The NumPartitions field is needed so the controller can create
>> the
>> > destination topic if it does not already exist. When a topic is being
>> > mirrored for the first time, the destination cluster has no knowledge
>> > of the source topic's partition count, so the request must carry it."
>> > Hmm, this seems inconsistent. When a mirror is started, the source topic
>> > may not exist yet. When the source topic is created later, we can start
>> > mirroring without requiring the user to provide the number of
>> partitions.
>> >
>> > > When the source topic is created later, we can start mirroring without
>> > requiring the user to provide the number of partitions.
>> >
>> > This is because this happens in periodical sync up in the destination
>> > cluster, and the numPartitions and other topic info in the source
>> cluster
>> > is already retrieved during the sync up process. Also, if users are
>> > creating mirror topics via admin API, this info can also be retrieved
>> > inside the admin client if not provided.
>> >
>> >
>> >
>> > > JR51. Start Mirror: I am wondering if we truly need the last mirror
>> epoch
>> > for truncation when starting a mirror. When a mirror is stopped at the
>> > source, we already bump up the leader epoch and write some additional
>> > records for cleanup. It seems that we can just use the DivergingEpoch
>> > returned in the initial mirror Fetch request to detect the diverging
>> epoch
>> > and use that for truncation at the mirror destination.
>> >
>> > That’s a good question. I thought about this before but unfortunately it
>> > doesn’t work. The reason we need the last mirror epoch record is because
>> > the destination cluster might not completely sync with the source
>> cluster
>> > before STOPPED. And this un-mirrored data might confuse the replication
>> > protocol when reverse mirror. For example,
>> >
>> > Cluster A (source) —> Cluster B (dest)
>> > foo-0 leader log foo-0 leader log
>> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> > Offset 2: from A (LE: 10)
>> >
>> > While cluster B mirrors until offset 1, the cluster A crashes and
>> failover
>> > to cluster B. B bumps leader epoch to 10, appends a MIRROR_PID_RESET
>> with
>> > LE: 10. Then cluster A starts to reverse mirror from cluster B, the log
>> > will be like this:
>> >
>> > Cluster A (dest) <-- Cluster B (source)
>> > foo-0 leader log: foo-0 leader log:
>> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
>> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
>> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
>> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
>> >
>> > As you can see, cluster A (new destination cluster) doesn’t know the
>> > offset 2 is not belonging to the cluster B, which causes inconsistent
>> logs.
>> > So like what we described in the KIP, the last mirror epoch is the
>> > synchronization point between source and destination. Any data beyond
>> the
>> > point is treated as unknown to the source cluster and should be
>> truncated.
>> >
>> >
>> >
>> > > JR52. Stop Mirror:
>> > > JR52.1 "3. If patterns are provided, the controller removes matching
>> > entries from mirror.topics.include or adds them to
>> mirror.topics.exclude on
>> > the CLUSTER_MIRROR resource."
>> > Why is it 'or' and not 'and'?
>> >
>> > Yes, that should be AND. KIP updated. Thanks.
>> >
>> >
>> > > JR52.2 "7. On subsequent metadata refresh cycles,
>> MirrorMetadataManager
>> > discovers new source topics matching the persisted include/exclude
>> patterns
>> > and repeats steps 3 through 7 for each."
>> > The broker will send a create topic request to the controller. How does
>> the
>> > controller associate it with the mirror name to create a
>> > `MirrorTopicStateChangeRecord`?
>> >
>> > No, we don’t send a create topic request to the controller here.
>> Actually
>> > we send a StartMirrorTopics request to the controller, which contains
>> the
>> > mirror name and topic metadata like NumPartitions, topicId… etc. So the
>> > controller can create the non-existed topic as explained in JR48, and
>> > associate it to MirrorTopicStateChangeRecord.
>> >
>> >
>> > > JR53. Describe Mirrors:
>> > > JR53.1 "1. The user sends a DescribeClusterMirrors request with
>> optional
>> > mirror names (empty means all mirrors)."
>> > In metadata request, we use a null arrary to represent all topics. It
>> would
>> > be useful to be consistent here.
>> >
>> > OK, KIP updated.
>> >
>> >
>> > > JR53.2 "2. The admin client fans out the request to all brokers and
>> > merges
>> > the responses client-side."
>> > This is not ideal. Not every broker hosts a mirror partition. So sending
>> > the request to every broker is wasteful. It's probably better to send
>> the
>> > request to a single broker, which can then contact the right
>> > MirrorFetcherManager and ClusterMirrorCoordinator.
>> >
>> > Agree, KIP updated.
>> >
>> >
>> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING,
>> PAUSED,
>> > STOPPED the only validate states? The state table has other states too.
>> > Ditto for MirrorPartitionState.
>> > The DesiredState field in MirrorTopicStateChangeRecord is to define the
>> > target state the user wants. And the desired state users can assign is
>> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as
>> STOPPING
>> > or LOG_TRUNCATION… etc because they are more like intermediate states.
>> >
>> > About MirrorPartitionState record, it’ll record any state in the state
>> > table that the current partition is. This is the real state for each
>> > partition. It’s like in the kubernetes world, when we change the replica
>> > field of the deployment yaml from 3 -> 5 (desired state), it won’t
>> increase
>> > to 5 immediately. The current replica (MirrorPartitionState record) is
>> > still 3, then 4 (maybe), and finally 5.
>> >
>> >
>> > Thank you,
>> > Luke
>> >
>> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <de...@kafka.apache.org>
>> > wrote:
>> >
>> >> Hi, Fede,
>> >>
>> >> Thanks for the reply. A few more comments.
>> >>
>> >> JR31.3 "Regarding state caching: the mirror partition cache lives in
>> >> MirrorMetadataManager. It is populated in two ways: by replaying
>> >> __mirror_state partitions this broker leads, and by fetching state
>> >> from remote coordinators via ReadMirrorStates RPCs."
>> >> Hmm, a given broker only caches the mirror partition state for
>> partitions
>> >> hashed into the __mirror_state partition hosted on this broker, right?
>> Why
>> >> does it need to fetch the state from other remote coordinators?
>> >>
>> >> JR36. I am still not quite sure why we need MirrorLeaderEpoch in the
>> Fetch
>> >> request. In the current Fetch request, in addition to
>> CurrentLeaderEpoch,
>> >> it has another field LastFetchedEpoch, which captures the last leader
>> >> epoch
>> >> in the data and is used to resolve the inconsistency between the
>> follower
>> >> and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
>> >>
>> >> JR48. "The NumPartitions field is needed so the controller can create
>> the
>> >> destination topic if it does not already exist. When a topic is being
>> >> mirrored for the first time, the destination cluster has no knowledge
>> >> of the source topic's partition count, so the request must carry it."
>> >> Hmm, this seems inconsistent. When a mirror is started, the source
>> topic
>> >> may not exist yet. When the source topic is created later, we can start
>> >> mirroring without requiring the user to provide the number of
>> partitions.
>> >>
>> >> JR51. Start Mirror: I am wondering if we truly need the last mirror
>> epoch
>> >> for truncation when starting a mirror. When a mirror is stopped at the
>> >> source, we already bump up the leader epoch and write some additional
>> >> records for cleanup. It seems that we can just use the DivergingEpoch
>> >> returned in the initial mirror Fetch request to detect the diverging
>> epoch
>> >> and use that for truncation at the mirror destination.
>> >>
>> >> JR52. Stop Mirror:
>> >> JR52.1 "3. If patterns are provided, the controller removes matching
>> >> entries from mirror.topics.include or adds them to
>> mirror.topics.exclude
>> >> on
>> >> the CLUSTER_MIRROR resource."
>> >> Why is it 'or' and not 'and'?
>> >> JR52.2 "7. On subsequent metadata refresh cycles, MirrorMetadataManager
>> >> discovers new source topics matching the persisted include/exclude
>> >> patterns
>> >> and repeats steps 3 through 7 for each."
>> >> The broker will send a create topic request to the controller. How does
>> >> the
>> >> controller associate it with the mirror name to create a
>> >> `MirrorTopicStateChangeRecord`?
>> >>
>> >> JR53. Describe Mirrors:
>> >> JR53.1 "1. The user sends a DescribeClusterMirrors request with
>> optional
>> >> mirror names (empty means all mirrors)."
>> >> In metadata request, we use a null arrary to represent all topics. It
>> >> would
>> >> be useful to be consistent here.
>> >> JR53.2 "2. The admin client fans out the request to all brokers and
>> merges
>> >> the responses client-side."
>> >> This is not ideal. Not every broker hosts a mirror partition. So
>> sending
>> >> the request to every broker is wasteful. It's probably better to send
>> the
>> >> request to a single broker, which can then contact the right
>> >> MirrorFetcherManager and ClusterMirrorCoordinator.
>> >>
>> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING, PAUSED,
>> >> STOPPED the only validate states? The state table has other states too.
>> >> Ditto for MirrorPartitionState.
>> >>
>> >> Jun
>> >>
>> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <fe...@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi Andrew, yes, validation is called at the start of createTopic. If
>> >> > the topic ID is invalid, the request fails with INVALID_REQUEST. If
>> >> > the topic ID is already used by a different topic name, it fails with
>> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual topic in the
>> >> > batch fails; other topics in the same CreateTopics request continue
>> >> > processing normally. Added this information to the KIP.
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
>> >> schofielaj@apache.org>
>> >> > wrote:
>> >> > >
>> >> > > Hi Fede,
>> >> > > One more small question.
>> >> > >
>> >> > > AS37: If the controller fails validation for the MirrorInfo.TopicId
>> >> > added to the CreateTopic request, what does it do? I suspect that it
>> >> fails
>> >> > the request with a particular error code.
>> >> > >
>> >> > > Thanks,
>> >> > > Andrew
>> >> > >
>> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> >> > > > Hi all,
>> >> > > >
>> >> > > > We updated the KIP to include Coordinator Runtime configurations
>> and
>> >> > > > the coordinator state transition validation paragraph. The latter
>> >> > > > covers the per-partition epoch fencing mechanism (leader epoch
>> and
>> >> > > > state epoch) that complements the existing per-topic StateOffset
>> >> > > > fencing for lifecycle operations.
>> >> > > >
>> >> > > > Thanks,
>> >> > > > Fede
>> >> > > >
>> >> >
>> >>
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for the comments.
Answering them below:
> JR31.3 It makes sense for the new partition leader to query the current
mirror state from the local or remote MirrorMetadataManager that owns the
partition's state. However, the wording in the KIP is confusing.
MirrorMetadataManager
mirror partition state cache should be populated only by one way, which is
replaying __mirror_state partitions this broker leads. By saying "and by
fetching state from remote coordinators via ReadMirrorStates RPCs.", it's
as if the mirror partition state cache on each broker stores the state for
all mirror partitions.
You’re right. KIP is updated to make it clear.
> JR36. I understand the intention now, but I still don't think we need
MirrorLeaderEpoch. Let's
assume the common case: no unclean leader election in the source cluster
during mirroring. Mirroring only fetches the committed data from the
source, which never changes if obtained a second time. In your example
image, follower broker 1 in the destination cluster can never obtain the
record B(0) because it's never committed in the source and thus is never
exposed to the destination leader. It's possible for broker 1 to first get
D(1) which is truncated later when broker 0 follows a new leader. But the
new leader will refetch the same D(1) from the source cluster. So, it's ok
for broker 1 to keep D(1).
Yes, you are correct. Because mirroring only fetches the committed data
from the source, the MirrorLeaderEpoch is unnecessary for normal mirroring.
Thanks for the clarification. So I think we can say, the
“MirrorLeaderEpoch” is specifically designed for the unclean leader
election support in Cluster Mirroring because the unclean leader election
will break the assumption that the committed data won’t be truncated.
Given that, we could gate the MirrorLeaderEpoch field by only setting it in
the Fetch request when "mirror.support.unclean.leader.election=true" (false
by default), and skip it entirely for normal mirroring. What do you think?
> JR51. Sounds good. Why do we choose to store the last mirror epoch instead
of a last mirror offset?
Yes, originally we stored the “last mirror offset” instead of “last mirror
epoch”. But later, we found it doesn’t work for unclean leader election
cases. (yes, `unclean leader election` again :) ) For example:
Cluster A (source) --> Cluster B (dest)
foo-0 leader log foo-0 leader log
Offset 0: A (LE: 0) Offset 0: A (LE: 0)
Offset 1: A (LE: 0) Offset 1: A (LE: 0)
Cluster B is mirroring from A. And A crashes and failover to B. B stores
the “last mirror offset” as 1.
When cluster A recovers, somehow the unclean leader election happens, log
is empty in the new leader, and then some more leadership changes to make
the leader epoch become 10. Then it starts the reverse mirror from B. The
log looks like this:
Cluster A (dest) <-- Cluster B (source)
foo-0 leader log foo-0 leader log
Offset 0: B (LE: 10) Offset 0: A (LE: 0)
Offset 1: B (LE: 10) Offset 1: A (LE: 0)
Offset 2: pid_res (LE: 10)
When A starts mirroring from B, log is truncated to last mirror offset: 1
(no-op), and then fetch from cluster B. The 1st fetch request is like this:
{ currentLeaderEpoch: 10, lastFetchedEpoch: 10, fetchOffset: 2, … }. The
divergingEpoch check in the cluster B queries the endOffset for leader
epoch 10 (in Partition#readRecords), and found the end offset for LE 10 is
2 in B, which is >= fetchOffset 2, so divergingEpoch validation passes and
returns offset 2 to cluster A. In short, relying on “last mirror offset”
cannot resolve the log diverge issue in unclean leader election case.
If instead, we truncate until “last mirror epoch”: 0, the offset 0 and 1 in
cluster A will be truncated as expected and then log converge with cluster
B correctly in the end.
Also, about JR53.2:
> JR53.2 "2. The admin client fans out the request to all brokers and merges
the responses client-side."
This is not ideal. Not every broker hosts a mirror partition. So sending
the request to every broker is wasteful. It's probably better to send the
request to a single broker, which can then contact the right
MirrorFetcherManager and ClusterMirrorCoordinator.
While working on this, I found we need to add one more tagged field
"Forwarded" to the DescribeClusterMirrorsRequest schema because otherwise
the receiving broker doesn’t know if this is the request from client and it
needs to contact other brokers, or this is just a forwarded request. The
workflow is like this:
1. the adminClient sends the DescribeClusterMirrorsRequest to a leastLoaded
broker, which doesn’t contain the "Forwarded" field.
2. The broker receives the request and sends DescribeClusterMirrorsRequest
with "Forwarded=true" to the responsible brokers and waits for the response
async.
3. When all responses are received, merge the results and respond to the
client.
Note: we cannot use ReadMirrorStatesRequest here because it doesn’t contain
the lag info in the coordinator.
The alternative solution is the adminClient sends
DescribeClusterMirrorsRequest to the “responsible brokers only” (via
describeTopics), instead of ALL brokers, and merge the results in the
client side. This solution doesn’t need to add a new field in the
DescribeClusterMirrorsRequest and still can avoid unnecessary requests
sent.
My preference is to keep the merge logic on the admin client side, which
saves the overhead in the broker side.
What do you think?
Thank you,
Luke
On Wed, Aug 5, 2026 at 3:19 AM Jun Rao via dev <de...@kafka.apache.org> wrote:
> Hi, Luke,
>
> Thanks for the reply.
>
> JR31.3 It makes sense for the new partition leader to query the current
> mirror state from the local or remote MirrorMetadataManager that owns the
> partition's state. However, the wording in the KIP is confusing.
> MirrorMetadataManager
> mirror partition state cache should be populated only by one way, which is
> replaying __mirror_state partitions this broker leads. By saying "and by
> fetching state from remote coordinators via ReadMirrorStates RPCs.", it's
> as if the mirror partition state cache on each broker stores the state for
> all mirror partitions.
>
> JR36. I understand the intention now, but I still don't think we need
> MirrorLeaderEpoch. Let's
> assume the common case: no unclean leader election in the source cluster
> during mirroring. Mirroring only fetches the committed data from the
> source, which never changes if obtained a second time. In your example
> image, follower broker 1 in the destination cluster can never obtain the
> record B(0) because it's never committed in the source and thus is never
> exposed to the destination leader. It's possible for broker 1 to first get
> D(1) which is truncated later when broker 0 follows a new leader. But the
> new leader will refetch the same D(1) from the source cluster. So, it's ok
> for broker 1 to keep D(1).
>
> JR51. Sounds good. Why do we choose to store the last mirror epoch instead
> of a last mirror offset?
>
> Jun
>
> On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the great comments again.
> > Answering your question below.
> >
> > > JR31.3 "Regarding state caching: the mirror partition cache lives in
> > MirrorMetadataManager. It is populated in two ways: by replaying
> > __mirror_state partitions this broker leads, and by fetching state
> > from remote coordinators via ReadMirrorStates RPCs."
> > Hmm, a given broker only caches the mirror partition state for partitions
> > hashed into the __mirror_state partition hosted on this broker, right?
> Why
> > does it need to fetch the state from other remote coordinators?
> >
> > > a given broker only caches the mirror partition state for partitions
> > hashed into the __mirror_state partition hosted on this broker, right?
> >
> > Right.
> >
> > > Why does it need to fetch the state from other remote coordinators?
> >
> > Because the data partition leader needs to know the current mirror state
> > of the partitions it leads. On a leadership change, the metadata delta is
> > published to the MirrorMetadataManager, which queries the current mirror
> > state for its led partitions from the coordinator (local or remote), and
> > processes tasks based on that state. This is the reason why it fetches
> the
> > state from other remote coordinator. This part of cache will be cleared
> > after the leadership change.
> >
> >
> >
> > > JR36. I am still not quite sure why we need MirrorLeaderEpoch in the
> > Fetch
> > request. In the current Fetch request, in addition to CurrentLeaderEpoch,
> > it has another field LastFetchedEpoch, which captures the last leader
> epoch
> > in the data and is used to resolve the inconsistency between the follower
> > and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
> >
> >
> > Unfortunately, the LastFetchedEpoch cannot be used to do the validation
> to
> > avoid the issue in KAFKA-18723. If we reject any fetched batches with
> epoch
> > higher than the LastFetchedEpoch, it means the follower can never accept
> > new appended batches with higher epoch in the leader node.
> >
> > Currently, the CurrentLeaderEpoch field serves 2 purposes in the fetch
> > API:
> > 1. The receiver (leader node) uses it to validate it matches the leader
> > epoch in the leader node
> > 2. The sender (follower node) uses it to validate the epoch in the
> fetched
> > batches are all <= this value.
> >
> > For purpose (1), the CurrentLeaderEpoch is still serving for followers in
> > the destination cluster. For purpose (2), my understanding of the
> > validation for KAFKA-18723 is like: the follower knows the
> > CurrentLeaderEpoch X is the highest epoch the leader node should contain
> in
> > the log batches when fetch request is sent. Any log batch beyond this
> epoch
> > in the fetch response means the follower’s metadata is stale. The batches
> > need to be rejected and follower needs to update metadata for next fetch.
> > In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves this
> purpose
> > to let the followers know the highest epoch in the leader node log in
> each
> > fetch response. Once the info is stale in the follower side, it updates
> it
> > from fetch response, and send next fetch request with correct
> > MirrorLeaderEpoch to make sure no inconsistent data is possible like
> > KAFKA-18723.
> >
> > I tried not to add new field in Fetch API, but this is the best solution
> I
> > can think of.
> >
> >
> >
> > > JR48. "The NumPartitions field is needed so the controller can create
> the
> > destination topic if it does not already exist. When a topic is being
> > mirrored for the first time, the destination cluster has no knowledge
> > of the source topic's partition count, so the request must carry it."
> > Hmm, this seems inconsistent. When a mirror is started, the source topic
> > may not exist yet. When the source topic is created later, we can start
> > mirroring without requiring the user to provide the number of partitions.
> >
> > > When the source topic is created later, we can start mirroring without
> > requiring the user to provide the number of partitions.
> >
> > This is because this happens in periodical sync up in the destination
> > cluster, and the numPartitions and other topic info in the source cluster
> > is already retrieved during the sync up process. Also, if users are
> > creating mirror topics via admin API, this info can also be retrieved
> > inside the admin client if not provided.
> >
> >
> >
> > > JR51. Start Mirror: I am wondering if we truly need the last mirror
> epoch
> > for truncation when starting a mirror. When a mirror is stopped at the
> > source, we already bump up the leader epoch and write some additional
> > records for cleanup. It seems that we can just use the DivergingEpoch
> > returned in the initial mirror Fetch request to detect the diverging
> epoch
> > and use that for truncation at the mirror destination.
> >
> > That’s a good question. I thought about this before but unfortunately it
> > doesn’t work. The reason we need the last mirror epoch record is because
> > the destination cluster might not completely sync with the source cluster
> > before STOPPED. And this un-mirrored data might confuse the replication
> > protocol when reverse mirror. For example,
> >
> > Cluster A (source) —> Cluster B (dest)
> > foo-0 leader log foo-0 leader log
> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> > Offset 2: from A (LE: 10)
> >
> > While cluster B mirrors until offset 1, the cluster A crashes and
> failover
> > to cluster B. B bumps leader epoch to 10, appends a MIRROR_PID_RESET with
> > LE: 10. Then cluster A starts to reverse mirror from cluster B, the log
> > will be like this:
> >
> > Cluster A (dest) <-- Cluster B (source)
> > foo-0 leader log: foo-0 leader log:
> > Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> > Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> > Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
> > Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
> >
> > As you can see, cluster A (new destination cluster) doesn’t know the
> > offset 2 is not belonging to the cluster B, which causes inconsistent
> logs.
> > So like what we described in the KIP, the last mirror epoch is the
> > synchronization point between source and destination. Any data beyond the
> > point is treated as unknown to the source cluster and should be
> truncated.
> >
> >
> >
> > > JR52. Stop Mirror:
> > > JR52.1 "3. If patterns are provided, the controller removes matching
> > entries from mirror.topics.include or adds them to mirror.topics.exclude
> on
> > the CLUSTER_MIRROR resource."
> > Why is it 'or' and not 'and'?
> >
> > Yes, that should be AND. KIP updated. Thanks.
> >
> >
> > > JR52.2 "7. On subsequent metadata refresh cycles, MirrorMetadataManager
> > discovers new source topics matching the persisted include/exclude
> patterns
> > and repeats steps 3 through 7 for each."
> > The broker will send a create topic request to the controller. How does
> the
> > controller associate it with the mirror name to create a
> > `MirrorTopicStateChangeRecord`?
> >
> > No, we don’t send a create topic request to the controller here. Actually
> > we send a StartMirrorTopics request to the controller, which contains the
> > mirror name and topic metadata like NumPartitions, topicId… etc. So the
> > controller can create the non-existed topic as explained in JR48, and
> > associate it to MirrorTopicStateChangeRecord.
> >
> >
> > > JR53. Describe Mirrors:
> > > JR53.1 "1. The user sends a DescribeClusterMirrors request with
> optional
> > mirror names (empty means all mirrors)."
> > In metadata request, we use a null arrary to represent all topics. It
> would
> > be useful to be consistent here.
> >
> > OK, KIP updated.
> >
> >
> > > JR53.2 "2. The admin client fans out the request to all brokers and
> > merges
> > the responses client-side."
> > This is not ideal. Not every broker hosts a mirror partition. So sending
> > the request to every broker is wasteful. It's probably better to send the
> > request to a single broker, which can then contact the right
> > MirrorFetcherManager and ClusterMirrorCoordinator.
> >
> > Agree, KIP updated.
> >
> >
> > > JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING, PAUSED,
> > STOPPED the only validate states? The state table has other states too.
> > Ditto for MirrorPartitionState.
> > The DesiredState field in MirrorTopicStateChangeRecord is to define the
> > target state the user wants. And the desired state users can assign is
> > MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as STOPPING
> > or LOG_TRUNCATION… etc because they are more like intermediate states.
> >
> > About MirrorPartitionState record, it’ll record any state in the state
> > table that the current partition is. This is the real state for each
> > partition. It’s like in the kubernetes world, when we change the replica
> > field of the deployment yaml from 3 -> 5 (desired state), it won’t
> increase
> > to 5 immediately. The current replica (MirrorPartitionState record) is
> > still 3, then 4 (maybe), and finally 5.
> >
> >
> > Thank you,
> > Luke
> >
> > On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> >> Hi, Fede,
> >>
> >> Thanks for the reply. A few more comments.
> >>
> >> JR31.3 "Regarding state caching: the mirror partition cache lives in
> >> MirrorMetadataManager. It is populated in two ways: by replaying
> >> __mirror_state partitions this broker leads, and by fetching state
> >> from remote coordinators via ReadMirrorStates RPCs."
> >> Hmm, a given broker only caches the mirror partition state for
> partitions
> >> hashed into the __mirror_state partition hosted on this broker, right?
> Why
> >> does it need to fetch the state from other remote coordinators?
> >>
> >> JR36. I am still not quite sure why we need MirrorLeaderEpoch in the
> Fetch
> >> request. In the current Fetch request, in addition to
> CurrentLeaderEpoch,
> >> it has another field LastFetchedEpoch, which captures the last leader
> >> epoch
> >> in the data and is used to resolve the inconsistency between the
> follower
> >> and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
> >>
> >> JR48. "The NumPartitions field is needed so the controller can create
> the
> >> destination topic if it does not already exist. When a topic is being
> >> mirrored for the first time, the destination cluster has no knowledge
> >> of the source topic's partition count, so the request must carry it."
> >> Hmm, this seems inconsistent. When a mirror is started, the source topic
> >> may not exist yet. When the source topic is created later, we can start
> >> mirroring without requiring the user to provide the number of
> partitions.
> >>
> >> JR51. Start Mirror: I am wondering if we truly need the last mirror
> epoch
> >> for truncation when starting a mirror. When a mirror is stopped at the
> >> source, we already bump up the leader epoch and write some additional
> >> records for cleanup. It seems that we can just use the DivergingEpoch
> >> returned in the initial mirror Fetch request to detect the diverging
> epoch
> >> and use that for truncation at the mirror destination.
> >>
> >> JR52. Stop Mirror:
> >> JR52.1 "3. If patterns are provided, the controller removes matching
> >> entries from mirror.topics.include or adds them to mirror.topics.exclude
> >> on
> >> the CLUSTER_MIRROR resource."
> >> Why is it 'or' and not 'and'?
> >> JR52.2 "7. On subsequent metadata refresh cycles, MirrorMetadataManager
> >> discovers new source topics matching the persisted include/exclude
> >> patterns
> >> and repeats steps 3 through 7 for each."
> >> The broker will send a create topic request to the controller. How does
> >> the
> >> controller associate it with the mirror name to create a
> >> `MirrorTopicStateChangeRecord`?
> >>
> >> JR53. Describe Mirrors:
> >> JR53.1 "1. The user sends a DescribeClusterMirrors request with optional
> >> mirror names (empty means all mirrors)."
> >> In metadata request, we use a null arrary to represent all topics. It
> >> would
> >> be useful to be consistent here.
> >> JR53.2 "2. The admin client fans out the request to all brokers and
> merges
> >> the responses client-side."
> >> This is not ideal. Not every broker hosts a mirror partition. So sending
> >> the request to every broker is wasteful. It's probably better to send
> the
> >> request to a single broker, which can then contact the right
> >> MirrorFetcherManager and ClusterMirrorCoordinator.
> >>
> >> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING, PAUSED,
> >> STOPPED the only validate states? The state table has other states too.
> >> Ditto for MirrorPartitionState.
> >>
> >> Jun
> >>
> >> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <fe...@gmail.com>
> >> wrote:
> >>
> >> > Hi Andrew, yes, validation is called at the start of createTopic. If
> >> > the topic ID is invalid, the request fails with INVALID_REQUEST. If
> >> > the topic ID is already used by a different topic name, it fails with
> >> > TOPIC_ALREADY_EXISTS. In both cases, only the individual topic in the
> >> > batch fails; other topics in the same CreateTopics request continue
> >> > processing normally. Added this information to the KIP.
> >> >
> >> >
> >> >
> >> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
> >> schofielaj@apache.org>
> >> > wrote:
> >> > >
> >> > > Hi Fede,
> >> > > One more small question.
> >> > >
> >> > > AS37: If the controller fails validation for the MirrorInfo.TopicId
> >> > added to the CreateTopic request, what does it do? I suspect that it
> >> fails
> >> > the request with a particular error code.
> >> > >
> >> > > Thanks,
> >> > > Andrew
> >> > >
> >> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> >> > > > Hi all,
> >> > > >
> >> > > > We updated the KIP to include Coordinator Runtime configurations
> and
> >> > > > the coordinator state transition validation paragraph. The latter
> >> > > > covers the per-partition epoch fencing mechanism (leader epoch and
> >> > > > state epoch) that complements the existing per-topic StateOffset
> >> > > > fencing for lifecycle operations.
> >> > > >
> >> > > > Thanks,
> >> > > > Fede
> >> > > >
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Luke,
Thanks for the reply.
JR31.3 It makes sense for the new partition leader to query the current
mirror state from the local or remote MirrorMetadataManager that owns the
partition's state. However, the wording in the KIP is confusing.
MirrorMetadataManager
mirror partition state cache should be populated only by one way, which is
replaying __mirror_state partitions this broker leads. By saying "and by
fetching state from remote coordinators via ReadMirrorStates RPCs.", it's
as if the mirror partition state cache on each broker stores the state for
all mirror partitions.
JR36. I understand the intention now, but I still don't think we need
MirrorLeaderEpoch. Let's
assume the common case: no unclean leader election in the source cluster
during mirroring. Mirroring only fetches the committed data from the
source, which never changes if obtained a second time. In your example
image, follower broker 1 in the destination cluster can never obtain the
record B(0) because it's never committed in the source and thus is never
exposed to the destination leader. It's possible for broker 1 to first get
D(1) which is truncated later when broker 0 follows a new leader. But the
new leader will refetch the same D(1) from the source cluster. So, it's ok
for broker 1 to keep D(1).
JR51. Sounds good. Why do we choose to store the last mirror epoch instead
of a last mirror offset?
Jun
On Tue, Aug 4, 2026 at 5:26 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Jun,
>
> Thanks for the great comments again.
> Answering your question below.
>
> > JR31.3 "Regarding state caching: the mirror partition cache lives in
> MirrorMetadataManager. It is populated in two ways: by replaying
> __mirror_state partitions this broker leads, and by fetching state
> from remote coordinators via ReadMirrorStates RPCs."
> Hmm, a given broker only caches the mirror partition state for partitions
> hashed into the __mirror_state partition hosted on this broker, right? Why
> does it need to fetch the state from other remote coordinators?
>
> > a given broker only caches the mirror partition state for partitions
> hashed into the __mirror_state partition hosted on this broker, right?
>
> Right.
>
> > Why does it need to fetch the state from other remote coordinators?
>
> Because the data partition leader needs to know the current mirror state
> of the partitions it leads. On a leadership change, the metadata delta is
> published to the MirrorMetadataManager, which queries the current mirror
> state for its led partitions from the coordinator (local or remote), and
> processes tasks based on that state. This is the reason why it fetches the
> state from other remote coordinator. This part of cache will be cleared
> after the leadership change.
>
>
>
> > JR36. I am still not quite sure why we need MirrorLeaderEpoch in the
> Fetch
> request. In the current Fetch request, in addition to CurrentLeaderEpoch,
> it has another field LastFetchedEpoch, which captures the last leader epoch
> in the data and is used to resolve the inconsistency between the follower
> and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
>
>
> Unfortunately, the LastFetchedEpoch cannot be used to do the validation to
> avoid the issue in KAFKA-18723. If we reject any fetched batches with epoch
> higher than the LastFetchedEpoch, it means the follower can never accept
> new appended batches with higher epoch in the leader node.
>
> Currently, the CurrentLeaderEpoch field serves 2 purposes in the fetch
> API:
> 1. The receiver (leader node) uses it to validate it matches the leader
> epoch in the leader node
> 2. The sender (follower node) uses it to validate the epoch in the fetched
> batches are all <= this value.
>
> For purpose (1), the CurrentLeaderEpoch is still serving for followers in
> the destination cluster. For purpose (2), my understanding of the
> validation for KAFKA-18723 is like: the follower knows the
> CurrentLeaderEpoch X is the highest epoch the leader node should contain in
> the log batches when fetch request is sent. Any log batch beyond this epoch
> in the fetch response means the follower’s metadata is stale. The batches
> need to be rejected and follower needs to update metadata for next fetch.
> In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves this purpose
> to let the followers know the highest epoch in the leader node log in each
> fetch response. Once the info is stale in the follower side, it updates it
> from fetch response, and send next fetch request with correct
> MirrorLeaderEpoch to make sure no inconsistent data is possible like
> KAFKA-18723.
>
> I tried not to add new field in Fetch API, but this is the best solution I
> can think of.
>
>
>
> > JR48. "The NumPartitions field is needed so the controller can create the
> destination topic if it does not already exist. When a topic is being
> mirrored for the first time, the destination cluster has no knowledge
> of the source topic's partition count, so the request must carry it."
> Hmm, this seems inconsistent. When a mirror is started, the source topic
> may not exist yet. When the source topic is created later, we can start
> mirroring without requiring the user to provide the number of partitions.
>
> > When the source topic is created later, we can start mirroring without
> requiring the user to provide the number of partitions.
>
> This is because this happens in periodical sync up in the destination
> cluster, and the numPartitions and other topic info in the source cluster
> is already retrieved during the sync up process. Also, if users are
> creating mirror topics via admin API, this info can also be retrieved
> inside the admin client if not provided.
>
>
>
> > JR51. Start Mirror: I am wondering if we truly need the last mirror epoch
> for truncation when starting a mirror. When a mirror is stopped at the
> source, we already bump up the leader epoch and write some additional
> records for cleanup. It seems that we can just use the DivergingEpoch
> returned in the initial mirror Fetch request to detect the diverging epoch
> and use that for truncation at the mirror destination.
>
> That’s a good question. I thought about this before but unfortunately it
> doesn’t work. The reason we need the last mirror epoch record is because
> the destination cluster might not completely sync with the source cluster
> before STOPPED. And this un-mirrored data might confuse the replication
> protocol when reverse mirror. For example,
>
> Cluster A (source) —> Cluster B (dest)
> foo-0 leader log foo-0 leader log
> Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> Offset 2: from A (LE: 10)
>
> While cluster B mirrors until offset 1, the cluster A crashes and failover
> to cluster B. B bumps leader epoch to 10, appends a MIRROR_PID_RESET with
> LE: 10. Then cluster A starts to reverse mirror from cluster B, the log
> will be like this:
>
> Cluster A (dest) <-- Cluster B (source)
> foo-0 leader log: foo-0 leader log:
> Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
> Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
> Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
> Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
>
> As you can see, cluster A (new destination cluster) doesn’t know the
> offset 2 is not belonging to the cluster B, which causes inconsistent logs.
> So like what we described in the KIP, the last mirror epoch is the
> synchronization point between source and destination. Any data beyond the
> point is treated as unknown to the source cluster and should be truncated.
>
>
>
> > JR52. Stop Mirror:
> > JR52.1 "3. If patterns are provided, the controller removes matching
> entries from mirror.topics.include or adds them to mirror.topics.exclude on
> the CLUSTER_MIRROR resource."
> Why is it 'or' and not 'and'?
>
> Yes, that should be AND. KIP updated. Thanks.
>
>
> > JR52.2 "7. On subsequent metadata refresh cycles, MirrorMetadataManager
> discovers new source topics matching the persisted include/exclude patterns
> and repeats steps 3 through 7 for each."
> The broker will send a create topic request to the controller. How does the
> controller associate it with the mirror name to create a
> `MirrorTopicStateChangeRecord`?
>
> No, we don’t send a create topic request to the controller here. Actually
> we send a StartMirrorTopics request to the controller, which contains the
> mirror name and topic metadata like NumPartitions, topicId… etc. So the
> controller can create the non-existed topic as explained in JR48, and
> associate it to MirrorTopicStateChangeRecord.
>
>
> > JR53. Describe Mirrors:
> > JR53.1 "1. The user sends a DescribeClusterMirrors request with optional
> mirror names (empty means all mirrors)."
> In metadata request, we use a null arrary to represent all topics. It would
> be useful to be consistent here.
>
> OK, KIP updated.
>
>
> > JR53.2 "2. The admin client fans out the request to all brokers and
> merges
> the responses client-side."
> This is not ideal. Not every broker hosts a mirror partition. So sending
> the request to every broker is wasteful. It's probably better to send the
> request to a single broker, which can then contact the right
> MirrorFetcherManager and ClusterMirrorCoordinator.
>
> Agree, KIP updated.
>
>
> > JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING, PAUSED,
> STOPPED the only validate states? The state table has other states too.
> Ditto for MirrorPartitionState.
> The DesiredState field in MirrorTopicStateChangeRecord is to define the
> target state the user wants. And the desired state users can assign is
> MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as STOPPING
> or LOG_TRUNCATION… etc because they are more like intermediate states.
>
> About MirrorPartitionState record, it’ll record any state in the state
> table that the current partition is. This is the real state for each
> partition. It’s like in the kubernetes world, when we change the replica
> field of the deployment yaml from 3 -> 5 (desired state), it won’t increase
> to 5 immediately. The current replica (MirrorPartitionState record) is
> still 3, then 4 (maybe), and finally 5.
>
>
> Thank you,
> Luke
>
> On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
>> Hi, Fede,
>>
>> Thanks for the reply. A few more comments.
>>
>> JR31.3 "Regarding state caching: the mirror partition cache lives in
>> MirrorMetadataManager. It is populated in two ways: by replaying
>> __mirror_state partitions this broker leads, and by fetching state
>> from remote coordinators via ReadMirrorStates RPCs."
>> Hmm, a given broker only caches the mirror partition state for partitions
>> hashed into the __mirror_state partition hosted on this broker, right? Why
>> does it need to fetch the state from other remote coordinators?
>>
>> JR36. I am still not quite sure why we need MirrorLeaderEpoch in the Fetch
>> request. In the current Fetch request, in addition to CurrentLeaderEpoch,
>> it has another field LastFetchedEpoch, which captures the last leader
>> epoch
>> in the data and is used to resolve the inconsistency between the follower
>> and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
>>
>> JR48. "The NumPartitions field is needed so the controller can create the
>> destination topic if it does not already exist. When a topic is being
>> mirrored for the first time, the destination cluster has no knowledge
>> of the source topic's partition count, so the request must carry it."
>> Hmm, this seems inconsistent. When a mirror is started, the source topic
>> may not exist yet. When the source topic is created later, we can start
>> mirroring without requiring the user to provide the number of partitions.
>>
>> JR51. Start Mirror: I am wondering if we truly need the last mirror epoch
>> for truncation when starting a mirror. When a mirror is stopped at the
>> source, we already bump up the leader epoch and write some additional
>> records for cleanup. It seems that we can just use the DivergingEpoch
>> returned in the initial mirror Fetch request to detect the diverging epoch
>> and use that for truncation at the mirror destination.
>>
>> JR52. Stop Mirror:
>> JR52.1 "3. If patterns are provided, the controller removes matching
>> entries from mirror.topics.include or adds them to mirror.topics.exclude
>> on
>> the CLUSTER_MIRROR resource."
>> Why is it 'or' and not 'and'?
>> JR52.2 "7. On subsequent metadata refresh cycles, MirrorMetadataManager
>> discovers new source topics matching the persisted include/exclude
>> patterns
>> and repeats steps 3 through 7 for each."
>> The broker will send a create topic request to the controller. How does
>> the
>> controller associate it with the mirror name to create a
>> `MirrorTopicStateChangeRecord`?
>>
>> JR53. Describe Mirrors:
>> JR53.1 "1. The user sends a DescribeClusterMirrors request with optional
>> mirror names (empty means all mirrors)."
>> In metadata request, we use a null arrary to represent all topics. It
>> would
>> be useful to be consistent here.
>> JR53.2 "2. The admin client fans out the request to all brokers and merges
>> the responses client-side."
>> This is not ideal. Not every broker hosts a mirror partition. So sending
>> the request to every broker is wasteful. It's probably better to send the
>> request to a single broker, which can then contact the right
>> MirrorFetcherManager and ClusterMirrorCoordinator.
>>
>> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING, PAUSED,
>> STOPPED the only validate states? The state table has other states too.
>> Ditto for MirrorPartitionState.
>>
>> Jun
>>
>> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <fe...@gmail.com>
>> wrote:
>>
>> > Hi Andrew, yes, validation is called at the start of createTopic. If
>> > the topic ID is invalid, the request fails with INVALID_REQUEST. If
>> > the topic ID is already used by a different topic name, it fails with
>> > TOPIC_ALREADY_EXISTS. In both cases, only the individual topic in the
>> > batch fails; other topics in the same CreateTopics request continue
>> > processing normally. Added this information to the KIP.
>> >
>> >
>> >
>> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <
>> schofielaj@apache.org>
>> > wrote:
>> > >
>> > > Hi Fede,
>> > > One more small question.
>> > >
>> > > AS37: If the controller fails validation for the MirrorInfo.TopicId
>> > added to the CreateTopic request, what does it do? I suspect that it
>> fails
>> > the request with a particular error code.
>> > >
>> > > Thanks,
>> > > Andrew
>> > >
>> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
>> > > > Hi all,
>> > > >
>> > > > We updated the KIP to include Coordinator Runtime configurations and
>> > > > the coordinator state transition validation paragraph. The latter
>> > > > covers the per-partition epoch fencing mechanism (leader epoch and
>> > > > state epoch) that complements the existing per-topic StateOffset
>> > > > fencing for lifecycle operations.
>> > > >
>> > > > Thanks,
>> > > > Fede
>> > > >
>> >
>>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Jun,
Thanks for the great comments again.
Answering your question below.
> JR31.3 "Regarding state caching: the mirror partition cache lives in
MirrorMetadataManager. It is populated in two ways: by replaying
__mirror_state partitions this broker leads, and by fetching state
from remote coordinators via ReadMirrorStates RPCs."
Hmm, a given broker only caches the mirror partition state for partitions
hashed into the __mirror_state partition hosted on this broker, right? Why
does it need to fetch the state from other remote coordinators?
> a given broker only caches the mirror partition state for partitions
hashed into the __mirror_state partition hosted on this broker, right?
Right.
> Why does it need to fetch the state from other remote coordinators?
Because the data partition leader needs to know the current mirror state of
the partitions it leads. On a leadership change, the metadata delta is
published to the MirrorMetadataManager, which queries the current mirror
state for its led partitions from the coordinator (local or remote), and
processes tasks based on that state. This is the reason why it fetches the
state from other remote coordinator. This part of cache will be cleared
after the leadership change.
> JR36. I am still not quite sure why we need MirrorLeaderEpoch in the Fetch
request. In the current Fetch request, in addition to CurrentLeaderEpoch,
it has another field LastFetchedEpoch, which captures the last leader epoch
in the data and is used to resolve the inconsistency between the follower
and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
Unfortunately, the LastFetchedEpoch cannot be used to do the validation to
avoid the issue in KAFKA-18723. If we reject any fetched batches with epoch
higher than the LastFetchedEpoch, it means the follower can never accept
new appended batches with higher epoch in the leader node.
Currently, the CurrentLeaderEpoch field serves 2 purposes in the fetch API:
1. The receiver (leader node) uses it to validate it matches the leader
epoch in the leader node
2. The sender (follower node) uses it to validate the epoch in the fetched
batches are all <= this value.
For purpose (1), the CurrentLeaderEpoch is still serving for followers in
the destination cluster. For purpose (2), my understanding of the
validation for KAFKA-18723 is like: the follower knows the
CurrentLeaderEpoch X is the highest epoch the leader node should contain in
the log batches when fetch request is sent. Any log batch beyond this epoch
in the fetch response means the follower’s metadata is stale. The batches
need to be rejected and follower needs to update metadata for next fetch.
In the Cluster Mirroring’s world, the MirrorLeaderEpoch serves this purpose
to let the followers know the highest epoch in the leader node log in each
fetch response. Once the info is stale in the follower side, it updates it
from fetch response, and send next fetch request with correct
MirrorLeaderEpoch to make sure no inconsistent data is possible like
KAFKA-18723.
I tried not to add new field in Fetch API, but this is the best solution I
can think of.
> JR48. "The NumPartitions field is needed so the controller can create the
destination topic if it does not already exist. When a topic is being
mirrored for the first time, the destination cluster has no knowledge
of the source topic's partition count, so the request must carry it."
Hmm, this seems inconsistent. When a mirror is started, the source topic
may not exist yet. When the source topic is created later, we can start
mirroring without requiring the user to provide the number of partitions.
> When the source topic is created later, we can start mirroring without
requiring the user to provide the number of partitions.
This is because this happens in periodical sync up in the destination
cluster, and the numPartitions and other topic info in the source cluster
is already retrieved during the sync up process. Also, if users are
creating mirror topics via admin API, this info can also be retrieved
inside the admin client if not provided.
> JR51. Start Mirror: I am wondering if we truly need the last mirror epoch
for truncation when starting a mirror. When a mirror is stopped at the
source, we already bump up the leader epoch and write some additional
records for cleanup. It seems that we can just use the DivergingEpoch
returned in the initial mirror Fetch request to detect the diverging epoch
and use that for truncation at the mirror destination.
That’s a good question. I thought about this before but unfortunately it
doesn’t work. The reason we need the last mirror epoch record is because
the destination cluster might not completely sync with the source cluster
before STOPPED. And this un-mirrored data might confuse the replication
protocol when reverse mirror. For example,
Cluster A (source) —> Cluster B (dest)
foo-0 leader log foo-0 leader log
Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
Offset 2: from A (LE: 10)
While cluster B mirrors until offset 1, the cluster A crashes and failover
to cluster B. B bumps leader epoch to 10, appends a MIRROR_PID_RESET with
LE: 10. Then cluster A starts to reverse mirror from cluster B, the log
will be like this:
Cluster A (dest) <-- Cluster B (source)
foo-0 leader log: foo-0 leader log:
Offset 0: from A (LE: 0) Offset 0: from A (LE: 0)
Offset 1: from A (LE: 0) Offset 1: from A (LE: 0)
Offset 2: from A (LE: 10) Offset 2: PID_RESET (LE: 10)
Offset 3: from B (LE: 10) Offset 3: from B (LE: 10)
As you can see, cluster A (new destination cluster) doesn’t know the offset
2 is not belonging to the cluster B, which causes inconsistent logs. So
like what we described in the KIP, the last mirror epoch is the
synchronization point between source and destination. Any data beyond the
point is treated as unknown to the source cluster and should be truncated.
> JR52. Stop Mirror:
> JR52.1 "3. If patterns are provided, the controller removes matching
entries from mirror.topics.include or adds them to mirror.topics.exclude on
the CLUSTER_MIRROR resource."
Why is it 'or' and not 'and'?
Yes, that should be AND. KIP updated. Thanks.
> JR52.2 "7. On subsequent metadata refresh cycles, MirrorMetadataManager
discovers new source topics matching the persisted include/exclude patterns
and repeats steps 3 through 7 for each."
The broker will send a create topic request to the controller. How does the
controller associate it with the mirror name to create a
`MirrorTopicStateChangeRecord`?
No, we don’t send a create topic request to the controller here. Actually
we send a StartMirrorTopics request to the controller, which contains the
mirror name and topic metadata like NumPartitions, topicId… etc. So the
controller can create the non-existed topic as explained in JR48, and
associate it to MirrorTopicStateChangeRecord.
> JR53. Describe Mirrors:
> JR53.1 "1. The user sends a DescribeClusterMirrors request with optional
mirror names (empty means all mirrors)."
In metadata request, we use a null arrary to represent all topics. It would
be useful to be consistent here.
OK, KIP updated.
> JR53.2 "2. The admin client fans out the request to all brokers and merges
the responses client-side."
This is not ideal. Not every broker hosts a mirror partition. So sending
the request to every broker is wasteful. It's probably better to send the
request to a single broker, which can then contact the right
MirrorFetcherManager and ClusterMirrorCoordinator.
Agree, KIP updated.
> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING, PAUSED,
STOPPED the only validate states? The state table has other states too.
Ditto for MirrorPartitionState.
The DesiredState field in MirrorTopicStateChangeRecord is to define the
target state the user wants. And the desired state users can assign is
MIRRORING/PAUSED/STOPPED. Users cannot assign a desired state as STOPPING
or LOG_TRUNCATION… etc because they are more like intermediate states.
About MirrorPartitionState record, it’ll record any state in the state
table that the current partition is. This is the real state for each
partition. It’s like in the kubernetes world, when we change the replica
field of the deployment yaml from 3 -> 5 (desired state), it won’t increase
to 5 immediately. The current replica (MirrorPartitionState record) is
still 3, then 4 (maybe), and finally 5.
Thank you,
Luke
On Tue, Aug 4, 2026 at 3:21 AM Jun Rao via dev <de...@kafka.apache.org> wrote:
> Hi, Fede,
>
> Thanks for the reply. A few more comments.
>
> JR31.3 "Regarding state caching: the mirror partition cache lives in
> MirrorMetadataManager. It is populated in two ways: by replaying
> __mirror_state partitions this broker leads, and by fetching state
> from remote coordinators via ReadMirrorStates RPCs."
> Hmm, a given broker only caches the mirror partition state for partitions
> hashed into the __mirror_state partition hosted on this broker, right? Why
> does it need to fetch the state from other remote coordinators?
>
> JR36. I am still not quite sure why we need MirrorLeaderEpoch in the Fetch
> request. In the current Fetch request, in addition to CurrentLeaderEpoch,
> it has another field LastFetchedEpoch, which captures the last leader epoch
> in the data and is used to resolve the inconsistency between the follower
> and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
>
> JR48. "The NumPartitions field is needed so the controller can create the
> destination topic if it does not already exist. When a topic is being
> mirrored for the first time, the destination cluster has no knowledge
> of the source topic's partition count, so the request must carry it."
> Hmm, this seems inconsistent. When a mirror is started, the source topic
> may not exist yet. When the source topic is created later, we can start
> mirroring without requiring the user to provide the number of partitions.
>
> JR51. Start Mirror: I am wondering if we truly need the last mirror epoch
> for truncation when starting a mirror. When a mirror is stopped at the
> source, we already bump up the leader epoch and write some additional
> records for cleanup. It seems that we can just use the DivergingEpoch
> returned in the initial mirror Fetch request to detect the diverging epoch
> and use that for truncation at the mirror destination.
>
> JR52. Stop Mirror:
> JR52.1 "3. If patterns are provided, the controller removes matching
> entries from mirror.topics.include or adds them to mirror.topics.exclude on
> the CLUSTER_MIRROR resource."
> Why is it 'or' and not 'and'?
> JR52.2 "7. On subsequent metadata refresh cycles, MirrorMetadataManager
> discovers new source topics matching the persisted include/exclude patterns
> and repeats steps 3 through 7 for each."
> The broker will send a create topic request to the controller. How does the
> controller associate it with the mirror name to create a
> `MirrorTopicStateChangeRecord`?
>
> JR53. Describe Mirrors:
> JR53.1 "1. The user sends a DescribeClusterMirrors request with optional
> mirror names (empty means all mirrors)."
> In metadata request, we use a null arrary to represent all topics. It would
> be useful to be consistent here.
> JR53.2 "2. The admin client fans out the request to all brokers and merges
> the responses client-side."
> This is not ideal. Not every broker hosts a mirror partition. So sending
> the request to every broker is wasteful. It's probably better to send the
> request to a single broker, which can then contact the right
> MirrorFetcherManager and ClusterMirrorCoordinator.
>
> JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING, PAUSED,
> STOPPED the only validate states? The state table has other states too.
> Ditto for MirrorPartitionState.
>
> Jun
>
> On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <fe...@gmail.com>
> wrote:
>
> > Hi Andrew, yes, validation is called at the start of createTopic. If
> > the topic ID is invalid, the request fails with INVALID_REQUEST. If
> > the topic ID is already used by a different topic name, it fails with
> > TOPIC_ALREADY_EXISTS. In both cases, only the individual topic in the
> > batch fails; other topics in the same CreateTopics request continue
> > processing normally. Added this information to the KIP.
> >
> >
> >
> > On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <schofielaj@apache.org
> >
> > wrote:
> > >
> > > Hi Fede,
> > > One more small question.
> > >
> > > AS37: If the controller fails validation for the MirrorInfo.TopicId
> > added to the CreateTopic request, what does it do? I suspect that it
> fails
> > the request with a particular error code.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On 2026/07/30 17:13:46 Federico Valeri wrote:
> > > > Hi all,
> > > >
> > > > We updated the KIP to include Coordinator Runtime configurations and
> > > > the coordinator state transition validation paragraph. The latter
> > > > covers the per-partition epoch fencing mechanism (leader epoch and
> > > > state epoch) that complements the existing per-topic StateOffset
> > > > fencing for lifecycle operations.
> > > >
> > > > Thanks,
> > > > Fede
> > > >
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Fede,
Thanks for the reply. A few more comments.
JR31.3 "Regarding state caching: the mirror partition cache lives in
MirrorMetadataManager. It is populated in two ways: by replaying
__mirror_state partitions this broker leads, and by fetching state
from remote coordinators via ReadMirrorStates RPCs."
Hmm, a given broker only caches the mirror partition state for partitions
hashed into the __mirror_state partition hosted on this broker, right? Why
does it need to fetch the state from other remote coordinators?
JR36. I am still not quite sure why we need MirrorLeaderEpoch in the Fetch
request. In the current Fetch request, in addition to CurrentLeaderEpoch,
it has another field LastFetchedEpoch, which captures the last leader epoch
in the data and is used to resolve the inconsistency between the follower
and the leader. Does it achieve the same purpose as MirrorLeaderEpoch?
JR48. "The NumPartitions field is needed so the controller can create the
destination topic if it does not already exist. When a topic is being
mirrored for the first time, the destination cluster has no knowledge
of the source topic's partition count, so the request must carry it."
Hmm, this seems inconsistent. When a mirror is started, the source topic
may not exist yet. When the source topic is created later, we can start
mirroring without requiring the user to provide the number of partitions.
JR51. Start Mirror: I am wondering if we truly need the last mirror epoch
for truncation when starting a mirror. When a mirror is stopped at the
source, we already bump up the leader epoch and write some additional
records for cleanup. It seems that we can just use the DivergingEpoch
returned in the initial mirror Fetch request to detect the diverging epoch
and use that for truncation at the mirror destination.
JR52. Stop Mirror:
JR52.1 "3. If patterns are provided, the controller removes matching
entries from mirror.topics.include or adds them to mirror.topics.exclude on
the CLUSTER_MIRROR resource."
Why is it 'or' and not 'and'?
JR52.2 "7. On subsequent metadata refresh cycles, MirrorMetadataManager
discovers new source topics matching the persisted include/exclude patterns
and repeats steps 3 through 7 for each."
The broker will send a create topic request to the controller. How does the
controller associate it with the mirror name to create a
`MirrorTopicStateChangeRecord`?
JR53. Describe Mirrors:
JR53.1 "1. The user sends a DescribeClusterMirrors request with optional
mirror names (empty means all mirrors)."
In metadata request, we use a null arrary to represent all topics. It would
be useful to be consistent here.
JR53.2 "2. The admin client fans out the request to all brokers and merges
the responses client-side."
This is not ideal. Not every broker hosts a mirror partition. So sending
the request to every broker is wasteful. It's probably better to send the
request to a single broker, which can then contact the right
MirrorFetcherManager and ClusterMirrorCoordinator.
JR54. MirrorTopicStateChangeRecord.DesiredState: Are MIRRORING, PAUSED,
STOPPED the only validate states? The state table has other states too.
Ditto for MirrorPartitionState.
Jun
On Fri, Jul 31, 2026 at 7:52 AM Federico Valeri <fe...@gmail.com>
wrote:
> Hi Andrew, yes, validation is called at the start of createTopic. If
> the topic ID is invalid, the request fails with INVALID_REQUEST. If
> the topic ID is already used by a different topic name, it fails with
> TOPIC_ALREADY_EXISTS. In both cases, only the individual topic in the
> batch fails; other topics in the same CreateTopics request continue
> processing normally. Added this information to the KIP.
>
>
>
> On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <sc...@apache.org>
> wrote:
> >
> > Hi Fede,
> > One more small question.
> >
> > AS37: If the controller fails validation for the MirrorInfo.TopicId
> added to the CreateTopic request, what does it do? I suspect that it fails
> the request with a particular error code.
> >
> > Thanks,
> > Andrew
> >
> > On 2026/07/30 17:13:46 Federico Valeri wrote:
> > > Hi all,
> > >
> > > We updated the KIP to include Coordinator Runtime configurations and
> > > the coordinator state transition validation paragraph. The latter
> > > covers the per-partition epoch fencing mechanism (leader epoch and
> > > state epoch) that complements the existing per-topic StateOffset
> > > fencing for lifecycle operations.
> > >
> > > Thanks,
> > > Fede
> > >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Andrew, yes, validation is called at the start of createTopic. If
the topic ID is invalid, the request fails with INVALID_REQUEST. If
the topic ID is already used by a different topic name, it fails with
TOPIC_ALREADY_EXISTS. In both cases, only the individual topic in the
batch fails; other topics in the same CreateTopics request continue
processing normally. Added this information to the KIP.
On Fri, Jul 31, 2026 at 10:53 AM Andrew Schofield <sc...@apache.org> wrote:
>
> Hi Fede,
> One more small question.
>
> AS37: If the controller fails validation for the MirrorInfo.TopicId added to the CreateTopic request, what does it do? I suspect that it fails the request with a particular error code.
>
> Thanks,
> Andrew
>
> On 2026/07/30 17:13:46 Federico Valeri wrote:
> > Hi all,
> >
> > We updated the KIP to include Coordinator Runtime configurations and
> > the coordinator state transition validation paragraph. The latter
> > covers the per-partition epoch fencing mechanism (leader epoch and
> > state epoch) that complements the existing per-topic StateOffset
> > fencing for lifecycle operations.
> >
> > Thanks,
> > Fede
> >
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Andrew Schofield <sc...@apache.org>.
Hi Fede,
One more small question.
AS37: If the controller fails validation for the MirrorInfo.TopicId added to the CreateTopic request, what does it do? I suspect that it fails the request with a particular error code.
Thanks,
Andrew
On 2026/07/30 17:13:46 Federico Valeri wrote:
> Hi all,
>
> We updated the KIP to include Coordinator Runtime configurations and
> the coordinator state transition validation paragraph. The latter
> covers the per-partition epoch fencing mechanism (leader epoch and
> state epoch) that complements the existing per-topic StateOffset
> fencing for lifecycle operations.
>
> Thanks,
> Fede
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi all,
We updated the KIP to include Coordinator Runtime configurations and
the coordinator state transition validation paragraph. The latter
covers the per-partition epoch fencing mechanism (leader epoch and
state epoch) that complements the existing per-topic StateOffset
fencing for lifecycle operations.
Thanks,
Fede
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Jun, we updated the KIP to reflect the following answers and
hopefully clear most doubts. We also added a "Scalability
Considerations" paragraph at the bottom.
On Thu, Jul 16, 2026 at 2:01 AM Jun Rao via dev <de...@kafka.apache.org> wrote:
>
> Hi, Fede,
>
> Thanks for the reply. A few more comments.
>
> JR26. The mirror partition states table link is not clickable.
Ok. This should be fixed now.
> JR28. Hmm, I am a bit confused. start/stop/pause etc changes the state at
> the mirror/topic level, but mirror coordinator manages the mirror partition
> state. Which broker does the validation? Is it the mirror coordinator or
> the MirrorMetadataManager?
One correction first: start/stop/pause/resume changes the state at the
mirror level, not at the topic level. The partition state machine is
managed separately by the ClusterMirrorCoordinator. This is similar to
an operator reconciling the desired state with the actual cluster
state.
The validation is done by MirrorMetadataManager, not by the
coordinator directly. The request is sent to any broker, and that
broker's MirrorMetadataManager validates that all affected partitions
are in a valid state for the requested operation before forwarding to
the controller. It does this in two steps: first it checks partitions
whose __mirror_state records are coordinated locally (from its
in-memory cache), then for partitions coordinated by other brokers, it
sends ReadMirrorStates RPCs to the responsible coordinators. If all
partitions pass, the broker stamps the current metadata offset on the
request and forwards it to the controller.
On the controller side, two more things happen. First, the controller
checks that no mirror state changed between the broker validation and
the controller processing, using the stamped metadata offset as an
optimistic lock. If any topic's last state change offset exceeds that
value, the entire request is rejected. Second, the controller
validates each topic individually (ID consistency, whether the topic
is already active in another mirror, whether it is in the expected
state, etc.). At this level, individual topics can fail while others
succeed, so the controller operates as a partial-success model.
> JR31.1 Hmm, if only topics are specified in StartMirrorTopicsRequest and
> those topics don't exist in the source yet, topics are not persisted
> anywhere, right? Then how can the MirrorMetadataManager picks up those
> topics when they are created in the source?
The topic names are actually persisted. When the controller processes
a StartMirrorTopics request, it adds every topic (whether it’s literal
or regex) to the "mirror.topics.include” configuration of that mirror
before doing anything else. So even if the topic does not exist on the
source yet and no MirrorTopicStateChangeRecord can be written for it,
the topic name is recorded in the include pattern.
From that point on, MirrorMetadataManager picks it up during periodic
sync. On every metadata refresh cycle, it sends a listTopics request
to the source cluster, matches the results against
mirror.topics.include (filtering out mirror.topics.exclude), and
removes topics that are already being mirrored. Any remaining matches
are new topics to start mirroring. It then issues a StartMirrorTopics
request for those topics with the topic ID and partition count from
the source, so the controller can create the destination topic and
write the MirrorTopicStateChangeRecord.
> JR31.3 OK. So MirrorMetadataManager serves two purposes: (1) mirrors
> topic/group/acl metadata from the source (2) initiates truncation/mirroring
> on mirror partitions. The instance responsible for (1) is the one based on
> mirror name hashed into the _mirror_state partition and the inststance
> responsible for (2) is the one on the broker that is the mirror partition
> leader. This also means that MirrorMetadataManager needs to run on every
> broker, consumes every _mirror_state partition and caches every mirror
> partition state. Is that correct?
Yes, that is mostly correct, with one clarification.
MirrorMetadataManager does run on every broker and it does serve those
two purposes, but the scoping is slightly different from what you
described.
For purpose (1), metadata sync from the source, source topic state
sync (leader caches, deletion detection, missed partition recovery)
runs on every broker. Topic creation, partition scaling, and
coordinator-level sync (topic configs, consumer group offsets, ACLs,
topic discovery by pattern) run only on the broker that leads the
__mirror_state partition determined by hashing the mirror name.
For purpose (2), initiating state transitions (truncation, epoch
fencing, creating fetchers), when a KRaft metadata update arrives,
each broker collects only the partitions where it gained or holds
leadership and triggers state transitions for those partitions. State
writes are routed to the coordinator that leads the relevant
__mirror_state partition for the composite key (mirror name, topic id,
partition number).
Regarding state caching: the mirror partition cache lives in
MirrorMetadataManager. It is populated in two ways: by replaying
__mirror_state partitions this broker leads, and by fetching state
from remote coordinators via ReadMirrorStates RPCs. When this broker
gains __mirror_state leadership, the ClusterMirrorCoordinator replays
the log to rebuild the cache. When it loses leadership, the cache is
cleared entirely.
> JR30. "During failback, the requesting cluster sends its own cluster ID,
> and the receiving cluster finds the matching mirror by scanning which of
> its mirror configs has that cluster ID as source."
> Does that mean that there could only be 1 mirror between a pair of source
> and destination cluster?
No, there can be multiple mirrors between the same pair of clusters.
Each topic on the destination can only belong to one mirror at a time,
but different topics can be mirrored through different mirror
configurations even if they share the same source and destination. For
example, topic "foo" on cluster B can be mirrored from cluster A via
mirror "a-to-b", while topic "bar" on the same cluster B can be
mirrored from the same cluster A via a separate mirror "a-to-b-2".
Regarding the second part: when ClusterId is set in
DescribeClusterMirrors, the MirrorNames field does not need to be
empty. They serve independent purposes. MirrorNames controls which
mirrors to describe (empty means all). ClusterId is a tagged field
used specifically for last mirror epoch lookup during failback. When
set, the receiving broker scans all its mirror configs to find mirrors
whose source cluster ID matches the requesting cluster's ID, then
looks up the stored last mirror epochs for the requested partitions.
If multiple mirrors match (because there are multiple mirrors from the
same source cluster), the broker aggregates the results by taking the
maximum last mirror epoch per partition across all matching mirrors.
So the two fields can be used together or independently.
> Also, when ClusterId is set in
> DescribeClusterMirrors, should the MirrorNames field be empty?
In practice it would be natural to leave MirrorNames empty. But the
protocol does not require it, and setting both fields would produce
both outputs. We actually leave it empty on LME lookup, which means
describe all mirrors, and we use the results to validate that the
source partitions are in STOPPED state before proceeding with failback
(mirror loop protection). This optimization spares one RPC request.
> "Then it does a simple LME lookup in its metadata log using the original
> mirror name."
> Does the requesting cluster use the FindCoordinator RPC to identify the
> ClusterMirrorCoordinator storing the LME for the mirror?
No, the requesting cluster does not use FindCoordinator. It sends a
DescribeClusterMirrors request, and the admin client fans out that
request to all brokers on the target cluster. Each broker reports the
last mirror epochs for partitions it coordinates locally, and the
admin client aggregates the results by taking the maximum last mirror
epoch per partition across all broker responses.
The reason it broadcasts instead of targeting a specific coordinator
is that the __mirror_state topic uses a composite key (mirror name,
topic ID, partition number) to distribute coordination across its
partitions. Two partitions of the same topic can be coordinated by
different brokers, so there is no single coordinator that holds all
the LME data for a given mirror or topic. Broadcasting to all brokers
and merging the results is the simplest way to collect the complete
picture.
Since FindCoordinator is not involved, there is no need to bump its
version or add a new KeyType.
> If so, should we bump up the version of FindCoordinator to support the new KeyType?
We actually don’t need this anymore, so we reverted the change.
Internally, we find the coordinator using the local metadata cache in
the destination cluster.
> JR36. Hmm, how does the follower set the MirrorLeaderEpoch field initially
> and subsequently? Also, when will the fetched batch epoch ever pass the
> expected value in the follower?
The follower initializes MirrorLeaderEpoch to 0 when it starts
fetching a mirror partition. On each fetch response, the leader
includes the latest leader epoch present in its log as the
MirrorLeaderEpoch. The follower then uses its current
MirrorLeaderEpoch (from the previous fetch cycle, not the one just
received) to validate the fetched batches before appending them.
Here is a concrete example. Suppose the destination leader has these
records in its log:
Offset 0: record A (batch epoch = 2)
Offset 1: record B (batch epoch = 2)
The follower sends a fetch request with fetchOffset=0 and
MirrorLeaderEpoch=0. The leader responds with the data and
MirrorLeaderEpoch=2 (the latest epoch in its log). Before appending,
the follower validates the batches using the MirrorLeaderEpoch from
its current state (0, not the 2 it just received). Since the batch
epoch (2) is greater than the MirrorLeaderEpoch used for validation
(0), the records are rejected. After processing completes, the
follower updates its MirrorLeaderEpoch to 2 (from the fetch response)
and retries. On the next fetch, the batch epoch (2) matches the
MirrorLeaderEpoch (2), so validation passes and the records are
appended.
Regarding the second question: in normal replication, the leader and
follower share the same epoch lineage, so the follower can validate
batches against the current leader epoch directly. In cluster
mirroring, the destination log contains batches carrying the source
cluster's leader epochs, which are unrelated to the destination's own
leader epoch. The follower cannot use the destination leader epoch for
validation because it would not match the batch epochs at all.
MirrorLeaderEpoch solves this by letting the leader tell each follower
what the latest epoch in the log currently is, so the follower can
validate that it is not receiving batches from a future epoch it has
not seen yet. The initial fetch always rejects because the follower
starts from 0 and needs one round trip to learn the actual epoch from
the leader.
> Our replication flow uses a similar logic based on leader epoch because the batches obtained
> from the leader are not necessariy committed. In cluster mirroring, only committed records are mirrored.
It is correct. But the issue in KAFKA-18723 is about the “log
truncation” and new records appended before fetch response sent. In
the normal replication, the log truncation can only happen on
“uncommitted data” like you said. But in an unclean leader election,
even the committed data will be truncated. So our solution is to
protect the unclean leader election in the source cluster causing
inconsistent data in the destination cluster. About the case of
unclean leader election causing inconsistent data, you can refer to
the image in the KIP here:
https://cwiki.apache.org/confluence/download/attachments/406620973/image-2026-7-14_11-47-39.png?version=1&modificationDate=1784022460000&api=v2
. It shows that due to the unclean leader election in the source
cluster and a late fetch response, the follower (broker 1) might
append the {E (LE: 1)} into offset 2 if no MirrorLeaderEpoch
validation.
> JR40. "When a state is in FAILED state, we’ll retry it in exponential
> backoff time to re-transition to the PreviousState to retry the state."
> Hmm, if we are trying to re-transition to the previous state, why allowing
> the previous state to transition to the error state in the first place?
Transitioning to FAILED makes the partition's actual status visible to
operators. If a partition stayed in MIRRORING while backing off due to
a connection issue or a transient error, it would look healthy from
the outside, which could mask real problems. By moving to FAILED, the
CLI tool and describe output can surface the error message, the retry
attempt count, and the previous state, giving operators clear
visibility into what went wrong and whether the system is recovering
on its own.
There is also a distinction between retriable and terminal failures.
For transient errors (network issues, temporary unavailability), the
coordinator retries with exponential backoff, transitioning back to
the previous state automatically. But if the retry attempts are
exhausted or the error is non-retriable (e.g. the source cluster ID
changed, indicating a misconfiguration), the partition stays in FAILED
permanently and requires manual intervention. Having an explicit
FAILED state makes this distinction actionable: operators can use the
CLI with the --failed flag to list partitions that need attention, and
metrics can track retry behavior over time.
> JR44. Could you add the description to the flow?
Ok. I added a “Leadership Changes” sub-paragraph to
MirrorClusterCoordinator and MirrorMetadata manager paragraphs.
> JR45. Is there any benefit of introducing this new config instead of just
> reusing sasl.mechanism.inter.broker.protocol?
The goal is to separate the security principal used for mirroring from
the one used for inter-broker communication. If we reused
sasl.mechanism.inter.broker.protocol, users would be forced to grant
additional permissions (GROUP, TOPIC READ/DESCRIBE, and others
required by mirror metadata sync) to the inter-broker principal, which
currently does not need them. This widens the blast radius of that
principal unnecessarily.
By introducing sasl.mechanism.mirror.admin.protocol, we keep the
inter-broker principal scoped to its original responsibilities. The
default value falls back to sasl.mechanism.inter.broker.protocol, so
there is no extra configuration burden for users who are fine with a
shared principal. It simply gives users the option to use a dedicated
principal with narrower permissions for mirroring when their security
model requires it.
> JR47. Workflow:
> JR47.1 For each of the initial requests, could you describe whether it
> lands on an arbitrary broker or a coordinator? If it's the latter, how is
> the coordinator determined?
All mirror operations (start, stop, pause, resume, delete) are sent to
an arbitrary broker. None of them target a coordinator directly. The
receiving broker validates the partition states before forwarding the
request to the controller.
The validation works as follows, using pause as an example. When a
PauseMirrorTopics request arrives at a broker, MirrorMetadataManager
checks that every partition under the requested topics is in a state
that is valid for the operation. For pause, all partitions must be in
MIRRORING state. It first checks partitions whose __mirror_state
records are coordinated locally (from its in-memory cache). For
partitions coordinated by other brokers, it sends ReadMirrorStates
RPCs to those brokers and waits for the responses. If any partition is
not in the expected state, the entire request is rejected with an
error. If all partitions pass, the broker stamps the current metadata
offset on the request and forwards it to the controller. The
controller then performs an optimistic locking check using that offset
to guard against concurrent state changes, and if it passes, writes
the MirrorTopicStateChangeRecord to the metadata log.
The same two-phase validation pattern (local check, then remote
coordinator RPCs, then forward to controller) applies to all mirror
operations. The only difference is which partition states are
considered valid: start requires STOPPED or UNKNOWN, stop requires
MIRRORING or PAUSED, pause requires MIRRORING, and resume requires
PAUSED.
I updated the workflows paragraph to include this information.
> JR47.2 It seems that RPCs like WriteMirrorStates and ReadMirrorStates
> should be used in the workflow, but they are not listed.
WriteMirrorStates and ReadMirrorStates are inter-broker RPCs used
internally by MirrorMetadataManager, not user-facing operations, so
they don't have their own workflow sections. But they play a role in
several of the listed workflows and should be called out where they
appear.
Specifically, ReadMirrorStates shows up in the broker-side validation
step of start, stop, pause, resume, and delete (when the broker needs
partition state from a remote coordinator). WriteMirrorStates shows up
when partition state changes need to be persisted to a __mirror_state
partition coordinated by another broker.
I updated the workflows paragraph to include this information.
> JR48. StartMirrorTopics: Why does it need the NumPartitions field since all
> partitions in the topic will be mirrored?
The NumPartitions field is needed so the controller can create the
destination topic if it does not already exist. When a topic is being
mirrored for the first time, the destination cluster has no knowledge
of the source topic's partition count, so the request must carry it.
If the topic already exists on the destination, the NumPartitions
field is ignored. The CLI populates this field automatically by
describing the source topic before sending the request.
> JR49. "Also it writes a MirrorTopicStateChangeRecord with empty mirror
> name. "
> An empty mirror name is not very intuitive. Should we use a null value
> instead?
Agreed. An empty string is not intuitive for "no mirror". Using null
is clearer and aligns with Kafka's convention for nullable fields in
wire protocol records. KIP updated.
> JR50. Configs: Could you add the type for each new config?
Sure. KIP updated.
> Jun
Thanks
Fede
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Fede,
Thanks for the reply. A few more comments.
JR26. The mirror partition states table link is not clickable.
JR28. Hmm, I am a bit confused. start/stop/pause etc changes the state at
the mirror/topic level, but mirror coordinator manages the mirror partition
state. Which broker does the validation? Is it the mirror coordinator or
the MirrorMetadataManager?
JR31.1 Hmm, if only topics are specified in StartMirrorTopicsRequest and
those topics don't exist in the source yet, topics are not persisted
anywhere, right? Then how can the MirrorMetadataManager picks up those
topics when they are created in the source?
JR31.3 OK. So MirrorMetadataManager serves two purposes: (1) mirrors
topic/group/acl metadata from the source (2) initiates truncation/mirroring
on mirror partitions. The instance responsible for (1) is the one based on
mirror name hashed into the _mirror_state partition and the inststance
responsible for (2) is the one on the broker that is the mirror partition
leader. This also means that MirrorMetadataManager needs to run on every
broker, consumes every _mirror_state partition and caches every mirror
partition state. Is that correct?
JR30. "During failback, the requesting cluster sends its own cluster ID,
and the receiving cluster finds the matching mirror by scanning which of
its mirror configs has that cluster ID as source."
Does that mean that there could only be 1 mirror between a pair of source
and destination cluster? Also, when ClusterId is set in
DescribeClusterMirrors, should the MirrorNames field be empty?
"Then it does a simple LME lookup in its metadata log using the original
mirror name."
Does the requesting cluster use the FindCoordinator RPC to identify the
ClusterMirrorCoordinator storing the LME for the mirror? If so, should we
bump up the version of FindCoordinator to support the new KeyType?
JR36. Hmm, how does the follower set the MirrorLeaderEpoch field initially
and subsequently? Also, when will the fetched batch epoch ever pass the
expected value in the follower? Our replication flow uses a similar logic
based on leader epoch because the batches obtained from the leader are not
necessariy committed. In cluster mirroring, only committed records are
mirrored.
JR40. "When a state is in FAILED state, we’ll retry it in exponential
backoff time to re-transition to the PreviousState to retry the state."
Hmm, if we are trying to re-transition to the previous state, why allowing
the previous state to transition to the error state in the first place?
JR44. Could you add the description to the flow?
JR45. Is there any benefit of introducing this new config instead of just
reusing sasl.mechanism.inter.broker.protocol?
JR47. Workflow:
JR47.1 For each of the initial requests, could you describe whether it
lands on an arbitrary broker or a coordinator? If it's the latter, how is
the coordinator determined?
JR47.2 It seems that RPCs like WriteMirrorStates and ReadMirrorStates
should be used in the workflow, but they are not listed.
JR48. StartMirrorTopics: Why does it need the NumPartitions field since all
partitions in the topic will be mirrored?
JR49. "Also it writes a MirrorTopicStateChangeRecord with empty mirror
name. "
An empty mirror name is not very intuitive. Should we use a null value
instead?
JR50. Configs: Could you add the type for each new config?
Jun
On Wed, Jul 15, 2026 at 2:00 AM Federico Valeri <fe...@gmail.com>
wrote:
> Hi Jun, thanks for the additional and useful feedback.
>
> JR20: Yes. MirrorMetadataManager sends a ListGroups RPC to the
> destination cluster filtering by active states (STABLE,
> PREPARING_REBALANCE, COMPLETING_REBALANCE, ASSIGNING, RECONCILING).
> Groups in those active states are skipped with a warning log message
> during offset sync. Groups in EMPTY or DEAD are implicitly allowed
> (not returned by the filter). KIP updated.
>
> JR21: No, mirrored records skip timestamp validation like how a
> follower does right now. Timestamp validation is skipped for mirrored
> records to preserve source timestamps. We basically rely on the source
> leader validation. KIP updated.
>
> JR22: Yes, they are correct, let me explain. The phrase "unchanged
> during replication" means per-batch appends don't bump, while the
> invariant is maintained by periodic/proactive bumps via a separate
> BumpLeaderEpochs controller RPC (see Leader Epoch Invariant
> paragraph). KIP updated.
>
> JR23: Destination. The mirror.support.unclean.leader.election config
> controls truncation behavior on the destination cluster. This prevents
> a destination ULE from electing a replica that has stale
> pre-truncation data. The source cluster's ULE behavior is not affected
> by this config. KIP updated.
>
> JR24: Agreed. KIP updated to remove the KIP-1327 paragraph.
>
> JR25: Exclude takes precedence. In the config section we have “Topics
> matching the exclude pattern are not mirrored even if they match
> mirror.topics.include. Internal topics are always excluded. Exclude
> always wins over include.” and similar for groups exclude patterns.
> KIP and protocol descriptions are updated to make it more visible.
>
> JR26: Ok. I created a mirror partition states table and linked where
> it is needed.
>
> JR27: Good catch. KIP updated.
>
> JR28: It is used for an optimistic locking guard that prevents a
> state-change operation (start, stop, pause, resume, delete) from going
> through if a mirror's topic state changed between the moment the
> broker did its validation (preconditions) and the moment the
> controller processes the operation.
>
> Mirror partition state transitions involve two separate nodes: a
> broker and the controller. The broker validates preconditions against
> the mirror coordinator, then sends a request to the controller, which
> writes the actual metadata records. There is a window between the
> broker's check and the controller's write where another operation can
> sneak in and change the state, making the original operation invalid.
>
> The StateValidationOffset field contains the cluster metadata offset
> before any validation. It represents the metadata snapshot against
> which the broker made its decision. If the controller detects that the
> MirrorTopicStateChangeRecord offset is strictly greater than
> StateValidationOffset, it rejects the operation with
> InvalidClusterMirrorStatesException.
>
> I added a ClusterMirrorCoordinator sub-paragraph to explain this.
>
> JR29: No RPC, the KIP sentence is slightly misleading. KIP updated.
>
> Each broker only reports partitions it is responsible for (i.e. it is
> the mirror leader for that partition). The ReplicaManager is just a
> pass-through that delegates to the MirroFetcherManager. The actual
> source of lag data is the MirrorFetcherThread, which continuously
> updates lag as it replicates from the source cluster. The admin client
> fans out the DescribeClusterMirrors request to all brokers to get the
> full picture.
>
> JR30: Cluster B does not match by mirror name. It matches by source
> cluster ID. The mirror config stores the source cluster ID at creation
> time, which is then used for validation and incremental mirroring.
> During failback, the requesting cluster sends its own cluster ID, and
> the receiving cluster finds the matching mirror by scanning which of
> its mirror configs has that cluster ID as source. Then it does a
> simple LME lookup in its metadata log using the original mirror name.
>
> I added a ClusterMirrorCoordinator sub-paragraph to explain this.
>
> JR31.1: Topics are already persisted, but not as ConfigRecord entries.
> When startMirrorTopics succeeds, a MirrorTopicStateChangeRecord is
> written to the metadata log for each topic that exists at that point.
> Topics that don't exist yet are picked up later by the periodic auto
> discovery loop, which matches source topics against the persisted
> include/exclude patterns and issues a StartMirrorTopics for any new
> matches. In short, the patterns are the durable config records of
> “what should be mirrored”. Individual topics are the resolved result.
>
> JR31.2: Yes, the include/exclude topic pattern configs are persisted
> and then the topics will be created if not exist. The KIP is updated
> to avoid any confusion.
>
> JR31.3: Every broker has its own MirrorMetadataManager instance that
> receives the same metadata update, but each instance only processes
> partitions for which that broker is the partition leader. Truncation
> and state transitions are performed locally by the leader broker. The
> same applies to stop: each broker's MMM handles the stopping process
> only for its own leader partitions.
>
> JR32: Yes, a stopped topic's mirror association persists in snapshots
> indefinitely until someone deletes the mirror. Once you delete the
> mirror, you lose the ability to leverage delta failback, but you can
> still do a failback from scratch.
>
> JR33.1/JR34: MirrorTopicStateChangeRecord (metadata log) is not
> tombstoned. It is cleared by writing a new record with an empty mirror
> name. The effect is that the topic's mirror association is removed
> from the TopicImage. When the next metadata snapshot is written, the
> snapshot writer skips topics with a blank mirror name. The KIP is
> updated.
>
> The records in __mirror_state (partition state and LME) are the ones
> that get tombstoned, and they do have keys (MirrorPartitionStateKey
> and LastMirrorEpochsKey).
>
> JR34.1: Sure. KIP Updated.
>
> JR34.2: When cluster A does a failback and sends
> DescribeClusterMirrors to its source (cluster B), it sets clusterId to
> A's own cluster ID. Cluster B then uses that to find which of its
> mirror configs has A as the source, as we discussed earlier.
>
> JR35: In a regular topic, if a topic is deleted and one of the replica
> brokers is offline, the topic deletion completes. Later when the topic
> re-creates with the same name, it’ll be assigned with a different
> topic ID. So when the fenced broker becomes online, it can detect the
> topic ID change and make the existing data folder as “stray”. On the
> other hand, a mirror topic is deleted and re-created with “the same
> source topic ID” while a broker is offline, that broker still has log
> directories tagged with the old topic ID. When it comes back online,
> it cannot distinguish between the stale log directory from the
> previous incarnation and the active one from the new creation. It
> would treat the stale directory as valid, causing data corruption or
> inconsistency.
>
> JR36: Your understanding is correct. The MirrorLeaderEpoch is not for
> fetching from the source cluster. It is for destination internal
> replication, where follower brokers fetch from the mirror leader.
>
> The MirrorLeaderEpoch field in the Fetch request tells the follower
> "this request was sent when the mirror epoch was N." The follower
> uses that epoch (not the regular leader epoch) to validate the fetched
> batches before appending them. If the fetched batch epoch has passed
> what the follower expected, the batches are rejected.
>
> JR37: Good catch. They are not necessary. It was related to the old
> design. KIP updated.
>
> JR38: Metadata synchronization operates at the mirror level rather
> than the partition level, so it uses a separate coordinator assignment
> based on the mirror name alone. That is:
>
> leader of partition [Utils.abs(mirrorName.hashCode()) % partitionNum]
> in __mirror_state topic
>
> Only the broker assigned as the metadata coordinator for a given
> mirror performs synchronization, and it applies changes only to the
> mirror partitions it manages. This avoids both redundant
> synchronization across brokers and unnecessary updates to partitions
> managed by other coordinators.
>
> JR39: TopicId is not needed for StopMirrorTopics. The admin client
> does not populate it, and the controller resolves the topic by name.
> The field appears in the schema but is unused. We removed it now, as
> well as the topicId field in PauseMirrorTopics and ResumeMirrorTopics
> requests.
>
> JR40: PreviousState is used for the FAILED retry mechanism. When a
> state is in FAILED state, we’ll retry it in exponential backoff time
> to re-transition to the PreviousState to retry the state. So the data
> partition leader will get this info from the coordinator and then
> apply the retry mechanism. We updated the KIP to explain it.
>
> JR41: As you suggest, we added a dedicated LeaderEpoch field to
> PartitionChangeRecord (version 3+). The controller computes the final
> epoch value and writes it into the record. All nodes just read it from
> PartitionRegistration.merge() when replay the PartitionChangeRecord,
> with no local derivation needed. For backward compatibility with older
> metadata versions (v0-v2), merge() falls back to the previous implicit
> derivation (leader change = epoch + 1) when the field is not set and
> leader is changed. KIP updated.
>
> JR42: Indeed, we should be consistent in all places to use topicId
> instead of topicName. We updated to use topicID in
> MirrorPartitionStateValue and LastMirrorEpochsValue. KIP is updated.
>
> JR43: Yes, we support all the sasl.* and ssl.* configurations for
> authentication with the source cluster. We’ve added 2 placeholder
> entries into the mirror config table.
>
> JR44: When a data partition leader changes, the onMetadataUpdate
> callback in the new leader node will be invoked and then it’ll get the
> current mirror partition state from the mirror coordinator, either
> read from local log, or sending ReadMirrorStates API to the
> coordinator node and wait for the response async. After the state is
> retrieved, it’ll process the current partition state tasks. Ex:
> MIRRORING state is to start a mirror fetcher thread to fetch from the
> source cluster.
>
> When a coordinator partition leader changes, the leader will load all
> the logs locally and start to do what a coordinator should do.
>
> JR45: This configuration is used for the admin client in the
> destination cluster (MirrorMetadataManager) to talk to the brokers and
> controllers in the same cluster. For example, when syncing up the
> consumer group offsets, the admin client needs to talk to the group
> coordinator in the destination cluster to commit/alter the offset.
>
> JR46: This information is in the “Early Access” sub-paragraph of the
> “Release Phases” paragraph, but I created an introduction to define
> them properly.
>
> Thanks
> Fede
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Jun, thanks for the additional and useful feedback.
JR20: Yes. MirrorMetadataManager sends a ListGroups RPC to the
destination cluster filtering by active states (STABLE,
PREPARING_REBALANCE, COMPLETING_REBALANCE, ASSIGNING, RECONCILING).
Groups in those active states are skipped with a warning log message
during offset sync. Groups in EMPTY or DEAD are implicitly allowed
(not returned by the filter). KIP updated.
JR21: No, mirrored records skip timestamp validation like how a
follower does right now. Timestamp validation is skipped for mirrored
records to preserve source timestamps. We basically rely on the source
leader validation. KIP updated.
JR22: Yes, they are correct, let me explain. The phrase "unchanged
during replication" means per-batch appends don't bump, while the
invariant is maintained by periodic/proactive bumps via a separate
BumpLeaderEpochs controller RPC (see Leader Epoch Invariant
paragraph). KIP updated.
JR23: Destination. The mirror.support.unclean.leader.election config
controls truncation behavior on the destination cluster. This prevents
a destination ULE from electing a replica that has stale
pre-truncation data. The source cluster's ULE behavior is not affected
by this config. KIP updated.
JR24: Agreed. KIP updated to remove the KIP-1327 paragraph.
JR25: Exclude takes precedence. In the config section we have “Topics
matching the exclude pattern are not mirrored even if they match
mirror.topics.include. Internal topics are always excluded. Exclude
always wins over include.” and similar for groups exclude patterns.
KIP and protocol descriptions are updated to make it more visible.
JR26: Ok. I created a mirror partition states table and linked where
it is needed.
JR27: Good catch. KIP updated.
JR28: It is used for an optimistic locking guard that prevents a
state-change operation (start, stop, pause, resume, delete) from going
through if a mirror's topic state changed between the moment the
broker did its validation (preconditions) and the moment the
controller processes the operation.
Mirror partition state transitions involve two separate nodes: a
broker and the controller. The broker validates preconditions against
the mirror coordinator, then sends a request to the controller, which
writes the actual metadata records. There is a window between the
broker's check and the controller's write where another operation can
sneak in and change the state, making the original operation invalid.
The StateValidationOffset field contains the cluster metadata offset
before any validation. It represents the metadata snapshot against
which the broker made its decision. If the controller detects that the
MirrorTopicStateChangeRecord offset is strictly greater than
StateValidationOffset, it rejects the operation with
InvalidClusterMirrorStatesException.
I added a ClusterMirrorCoordinator sub-paragraph to explain this.
JR29: No RPC, the KIP sentence is slightly misleading. KIP updated.
Each broker only reports partitions it is responsible for (i.e. it is
the mirror leader for that partition). The ReplicaManager is just a
pass-through that delegates to the MirroFetcherManager. The actual
source of lag data is the MirrorFetcherThread, which continuously
updates lag as it replicates from the source cluster. The admin client
fans out the DescribeClusterMirrors request to all brokers to get the
full picture.
JR30: Cluster B does not match by mirror name. It matches by source
cluster ID. The mirror config stores the source cluster ID at creation
time, which is then used for validation and incremental mirroring.
During failback, the requesting cluster sends its own cluster ID, and
the receiving cluster finds the matching mirror by scanning which of
its mirror configs has that cluster ID as source. Then it does a
simple LME lookup in its metadata log using the original mirror name.
I added a ClusterMirrorCoordinator sub-paragraph to explain this.
JR31.1: Topics are already persisted, but not as ConfigRecord entries.
When startMirrorTopics succeeds, a MirrorTopicStateChangeRecord is
written to the metadata log for each topic that exists at that point.
Topics that don't exist yet are picked up later by the periodic auto
discovery loop, which matches source topics against the persisted
include/exclude patterns and issues a StartMirrorTopics for any new
matches. In short, the patterns are the durable config records of
“what should be mirrored”. Individual topics are the resolved result.
JR31.2: Yes, the include/exclude topic pattern configs are persisted
and then the topics will be created if not exist. The KIP is updated
to avoid any confusion.
JR31.3: Every broker has its own MirrorMetadataManager instance that
receives the same metadata update, but each instance only processes
partitions for which that broker is the partition leader. Truncation
and state transitions are performed locally by the leader broker. The
same applies to stop: each broker's MMM handles the stopping process
only for its own leader partitions.
JR32: Yes, a stopped topic's mirror association persists in snapshots
indefinitely until someone deletes the mirror. Once you delete the
mirror, you lose the ability to leverage delta failback, but you can
still do a failback from scratch.
JR33.1/JR34: MirrorTopicStateChangeRecord (metadata log) is not
tombstoned. It is cleared by writing a new record with an empty mirror
name. The effect is that the topic's mirror association is removed
from the TopicImage. When the next metadata snapshot is written, the
snapshot writer skips topics with a blank mirror name. The KIP is
updated.
The records in __mirror_state (partition state and LME) are the ones
that get tombstoned, and they do have keys (MirrorPartitionStateKey
and LastMirrorEpochsKey).
JR34.1: Sure. KIP Updated.
JR34.2: When cluster A does a failback and sends
DescribeClusterMirrors to its source (cluster B), it sets clusterId to
A's own cluster ID. Cluster B then uses that to find which of its
mirror configs has A as the source, as we discussed earlier.
JR35: In a regular topic, if a topic is deleted and one of the replica
brokers is offline, the topic deletion completes. Later when the topic
re-creates with the same name, it’ll be assigned with a different
topic ID. So when the fenced broker becomes online, it can detect the
topic ID change and make the existing data folder as “stray”. On the
other hand, a mirror topic is deleted and re-created with “the same
source topic ID” while a broker is offline, that broker still has log
directories tagged with the old topic ID. When it comes back online,
it cannot distinguish between the stale log directory from the
previous incarnation and the active one from the new creation. It
would treat the stale directory as valid, causing data corruption or
inconsistency.
JR36: Your understanding is correct. The MirrorLeaderEpoch is not for
fetching from the source cluster. It is for destination internal
replication, where follower brokers fetch from the mirror leader.
The MirrorLeaderEpoch field in the Fetch request tells the follower
"this request was sent when the mirror epoch was N." The follower
uses that epoch (not the regular leader epoch) to validate the fetched
batches before appending them. If the fetched batch epoch has passed
what the follower expected, the batches are rejected.
JR37: Good catch. They are not necessary. It was related to the old
design. KIP updated.
JR38: Metadata synchronization operates at the mirror level rather
than the partition level, so it uses a separate coordinator assignment
based on the mirror name alone. That is:
leader of partition [Utils.abs(mirrorName.hashCode()) % partitionNum]
in __mirror_state topic
Only the broker assigned as the metadata coordinator for a given
mirror performs synchronization, and it applies changes only to the
mirror partitions it manages. This avoids both redundant
synchronization across brokers and unnecessary updates to partitions
managed by other coordinators.
JR39: TopicId is not needed for StopMirrorTopics. The admin client
does not populate it, and the controller resolves the topic by name.
The field appears in the schema but is unused. We removed it now, as
well as the topicId field in PauseMirrorTopics and ResumeMirrorTopics
requests.
JR40: PreviousState is used for the FAILED retry mechanism. When a
state is in FAILED state, we’ll retry it in exponential backoff time
to re-transition to the PreviousState to retry the state. So the data
partition leader will get this info from the coordinator and then
apply the retry mechanism. We updated the KIP to explain it.
JR41: As you suggest, we added a dedicated LeaderEpoch field to
PartitionChangeRecord (version 3+). The controller computes the final
epoch value and writes it into the record. All nodes just read it from
PartitionRegistration.merge() when replay the PartitionChangeRecord,
with no local derivation needed. For backward compatibility with older
metadata versions (v0-v2), merge() falls back to the previous implicit
derivation (leader change = epoch + 1) when the field is not set and
leader is changed. KIP updated.
JR42: Indeed, we should be consistent in all places to use topicId
instead of topicName. We updated to use topicID in
MirrorPartitionStateValue and LastMirrorEpochsValue. KIP is updated.
JR43: Yes, we support all the sasl.* and ssl.* configurations for
authentication with the source cluster. We’ve added 2 placeholder
entries into the mirror config table.
JR44: When a data partition leader changes, the onMetadataUpdate
callback in the new leader node will be invoked and then it’ll get the
current mirror partition state from the mirror coordinator, either
read from local log, or sending ReadMirrorStates API to the
coordinator node and wait for the response async. After the state is
retrieved, it’ll process the current partition state tasks. Ex:
MIRRORING state is to start a mirror fetcher thread to fetch from the
source cluster.
When a coordinator partition leader changes, the leader will load all
the logs locally and start to do what a coordinator should do.
JR45: This configuration is used for the admin client in the
destination cluster (MirrorMetadataManager) to talk to the brokers and
controllers in the same cluster. For example, when syncing up the
consumer group offsets, the admin client needs to talk to the group
coordinator in the destination cluster to commit/alter the offset.
JR46: This information is in the “Early Access” sub-paragraph of the
“Release Phases” paragraph, but I created an introduction to define
them properly.
Thanks
Fede
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by vaquar khan <va...@gmail.com>.
Hi Luke, Federico, Omnia, and the whole team,
Before anything else - thank you. We have been iterating on this KIP for
months now, and I deeply appreciate how thoroughly and seriously you have
engaged with every architectural and security concern raised throughout
this review process. Digging into each edge case, updating the design
specification, and opening the tracking JIRAs has ensured a robust design
for a change of this scale.
*VK12 (Data-Safety Clamp): Confirmed Closed.*I see the defensive clamp
fully integrated into both the MirrorMetadataManager section and the Group
Offsets section:
max(dest log start offset, min(dest LEO, source committed offset))
The inclusion of the worked execution example completely resolves my
data-safety concerns regarding offset boundaries. Thank you for folding
this critical validation layer into the core design.
*VK10 (Group-Namespace Validation & KAFKA-20799): Resolved.*Your reasoning
is structurally sound, and I verified the behavior against the core
implementation to be certain. Because group names are unique across
consumer/share/streams types by design (with GroupMetadataManager throwing
a GroupIdNotFoundException for cross-type access) and synced offset commits
route through the standard OffsetCommit path, an isolated pre-flight gate
in mirroring would be incomplete.
The correct structural fix belongs at the group-namespace level in core
Kafka. I am completely aligned with dropping the localized gate framework;
KAFKA-20799 is precisely the right architectural home for this ecosystem
fix.
To assist with the implementation of KAFKA-20799, here is how I view the
priority structure for production readiness (non-blocking for this KIP):
*Actionable Error Telemetry:* Enhance the exception payload to explicitly
include the colliding group name, its type in each cluster, and the
associated mirror name, rather than suppressing it behind a bare
GroupIdNotFoundException. This transforms an ambiguous failover-time skip
into a self-explaining, actionable diagnostic event.
*Operational Metrics: *Introduce a dedicated metric counter tracking
group-offset-sync collisions/errors so platform operators can alert on
collisions deterministically instead of parsing distributed logs.
*On-Demand Diagnostics (Lowest Priority):* Expose an on-demand
describe/list capability to surface point-in-time name collisions, which is
highly valuable for identifying pre-existing conflicts during initial
environment setup.
Item (1) delivers the primary architectural value; the remaining items can
land according to your core roadmap. I would be glad to serve as the
reviewer for the KAFKA-20799 PR once it is ready for code review.
With VK12 verified and VK10 properly decoupled and tracked in KAFKA-20799,
I have no remaining architectural concerns. You have my full endorsement on
KIP-1279. This is a highly significant advancement for Kafka's distributed
mirroring capabilities, executed with excellent rigor.
Outstanding work by the entire team.
Regards,
Vaquar Khan
https://www.linkedin.com/in/vaquar-khan-b695577/
On Mon, 13 Jul 2026 at 05:26, Luke Chen <sh...@gmail.com> wrote:
> Hi Vaquar,
>
> Thanks for your comments.
>
> VK12:
> Sorry, it's my bad that I forgot to update in the discussion thread and the
> KIP.
> Yes, we adopted your suggestion but forgot to update the KIP.
> I've just updated it now. Sorry for the confusion.
>
> VK10:
> "(a) returning a precise error for cross-type collisions"
> As you know, this is an existing logic, not related to this KIP.
> I've opened KAFKA-20799 <https://issues.apache.org/jira/browse/KAFKA-20799>
> to
> improve the error handling for group name collisions.
>
> "(b) adding a pre-flight validation check to the CLI utility"
> I'm wondering how much it helps because the group name is not fixed during
> the mirroring period.
> Even if we did a validation before starting the mirror, it is still
> possible the newly created consumer group name, either in source or
> destination cluster, will have a collision. Also, the
> "mirror.groups.include"
> is a dynamic configuration. Users can update it anytime.
> For example: source cluster has 0 consumer/share group, destination cluster
> has 1 "share group" called: "test"
> 1. "mirror.groups.include=.*": Users start the mirror, and later create a
> consumer group "test" in the source cluster, which collides with the one in
> the destination cluster when sync up.
> 2. "mirror.groups.include=dummy": Users start the mirror, and later create
> a consumer group "test" in the source cluster. No group offset sync up
> happens. But later, the "mirror.groups.include" config is updated to
> "mirror.groups.include=test",
> so the name collision still happens.
>
> For case (1), when mirror starts, we don't know if the source cluster will
> create a classic consumer group named "test" in the future. There is
> nothing we can do.
> For case (2), even if we add a validation each time the
> "mirror.groups.include/exclude"
> config updates. That still doesn't resolve this issue because we don't have
> a way to distinguish an offset commit is from mirror sync up or from user
> creation. For example, "mirror.groups.include=.*" changes to
> "mirror.groups.include=test".
> When the change triggers, should we treat the existing "test" group in the
> destination cluster as "name collision" or we should pass the validation
> because the "test" could be created and synced in the mirror? We have no
> way to distinguish it.
>
> My thought is to improve the option (a) in KAFKA-20799
> <https://issues.apache.org/jira/browse/KAFKA-20799> and make the error log
> clear to users.
> What do you think?
>
> Thank you,
> Luke
>
> On Mon, Jul 13, 2026 at 7:54 AM vaquar khan <va...@gmail.com> wrote:
>
> > Hi Federico, Luke, Omnia, and all,
> >
> > Thanks for the continued work on this. The KIP has come a long way. I
> want
> > to close out the items we have settled so the record is clean, then
> address
> > the remaining data-safety item where the current text does not yet
> > implement the necessary architectural guard.
> >
> >
> > *RESOLVED / CLOSED ON MY END*
> > - VK2 (Poison pill blast radius): Resolved. Partition-level FAILED
> > isolation in the fetcher pool is the correct model.
> >
> > - VK5 (Infinite loops): Resolved. Read-only enforcement on mirrored
> > topics makes A->B->A physically impossible.
> >
> > - VK6 (Unclean-election divergence): Accepted as a documented
> > limitation, and the updated ULE sub-design thoroughly addresses my
> initial
> > concerns.
> >
> > - VK7 (Tiered storage): Accepted for the initial release; the follow-up
> > plan is reasonable.
> >
> > - VK1 (Thundering herd / OOM): Resolved. Bounding peak memory
> > by num_fetcher_threads * response_max_bytes, rather than partitions *
> > 1MB, ensures predictable resource consumption.
> >
> > - VK3 (Control-plane hotspot): Resolved. Distributing
> > __mirror_state transitions via the (mirrorName, topicId, partition) hash
> > mirrors the proven scalability model of __consumer_offsets.
> >
> > - VK4 / VK8 (PID rewriting): Resolved, and superior to my original
> > proposal. Dropping PID rewriting via the MIRROR_PID_RESET control record
> > completely bypasses the hasProducerId() issue (where
> > AbstractRecordBatch.hasProducerId() evaluates NO_PRODUCER_ID <
> > producerId(), meaning a negative PID would have bypassed
> > ProducerStateManager and incorrectly defaulted the LSO to the high
> > watermark).
> >
> > - VK11 (TransactionIndex): Resolved. Confirmed the index is rebuilt
> > locally on append rather than copied over the wire.
> >
> > - VK9 (CRC / zero-copy): Moot once PID rewriting was removed.
> >
> > Thank you for working through these eight architectural items.
> >
> >
> > *STILL OPEN: VK12 (Offset-sync read-side data loss)*
> > Luke noted on June 2 that the proposed formula makes sense, which I
> > appreciate. However, the current Group Offsets text only implements the
> > active-group guard ("skip groups that are active on the destination in
> any
> > state other than EMPTY or DEAD"). While this guard is necessary, it is
> > insufficient to prevent data loss for INACTIVE or dead groups during
> > replication lag.
> >
> > Verifying this against the core coordinator implementation reveals a
> > critical vulnerability when operators follow the KIP's guidance to
> > set auto.offset.reset=latest:
> >
> > 1. The synced offset for an inactive group can legitimately exceed the
> > destination log end offset (LEO) due to replication lag. The
> > OffsetCommit path in the group coordinator does not validate a committed
> > offset against the partition LEO, meaning an out-of-range offset is
> > persisted as-is.
> >
> > 2. Upon failover, a resuming consumer reading from this offset receives
> an
> > OFFSET_OUT_OF_RANGE exception.
> >
> > 3. Under the auto.offset.reset=latest strategy, the reset resolves via
> > AutoOffsetResetStrategy.timestamp() to
> > ListOffsetsRequest.LATEST_TIMESTAMP, which returns the current
> destination
> > LEO.
> >
> > 4. Consequently, the consumer skips every record that arrived between
> > the stale committed position and the reset point. This results in
> > silent, unrecoverable read-side data loss precisely during a cluster
> > failover the exact scenario this KIP is designed to protect against.
> >
> > Because the affected group is inactive on the destination, the
> > active-group guard offers no protection here. The only way to guarantee
> > data safety is to enforce a structural invariant at sync time using a
> > double-clamp:
> >
> > syncedOffset = max(destinationLogStartOffset, min(sourceCommitted,
> > destinationLEO))
> >
> > - The min(..., destinationLEO) upper bound eliminates the
> > OFFSET_OUT_OF_RANGE exception window entirely, preventing forced resets
> and
> > data skipping.
> > - The max(destinationLogStartOffset, ...) lower bound prevents
> > group rewinding if an offset falls below the destination retention log
> > start offset.
> >
> > Under this invariant, the worst-case scenario is bounded
> > re-processing proportional to the replication lag at failover, which
> aligns
> > with Kafka's at-least-once delivery contract. This is strictly safer than
> > endorsing a configuration path that silently drops data.
> >
> > Can we update the Group Offsets section to explicitly define this
> > double-clamp as a coordinator-enforced invariant alongside the
> active-group
> > guard?
> >
> >
> > *STILL OPEN: VK10 (Group-name namespace collision / JR18)*
> > The current text specifies that an offset commit fails
> > with GroupIdNotFoundException when a consumer group and a share group
> share
> > a name.
> > As confirmed in the coordinator logic, share groups and consumer groups
> > occupy a single namespace, and cross-type access throws this exception.
> > This introduces two issues during operational failovers:
> >
> > 1. The error semantics are incorrect. The group ID does exist; it is
> > simply the wrong type. Throwing "not found" misleads operators under
> fire.
> > I agree with Jun's point in JR18 that an exception like
> > GroupIdInUseException is required.
> >
> > 2. Discovering name collisions during an emergency failover introduces
> > extreme operational risk. Introducing a pre-migration validation check
> to
> > the CLI (e.g., kafka-cluster-mirrors.sh) to scan and flag source group
> > and share-group collisions before mirroring begins would move this risk
> > from production runtime to set up-time.
> >
> > Can we commit to (a) returning a precise error for cross-type
> > collisions,and (b) adding a pre-flight validation check to the CLI
> utility?
> >
> > Once the double-clamp invariant for VK12 is integrated into the design
> text
> > and we have a clear resolution for VK10, I have no further blockers and
> > will gladly support the vote.
> >
> > Regards,
> > Vaquar Khan
> > https://www.linkedin.com/in/vaquar-khan-b695577/
> >
> > On Sun, 12 Jul 2026 at 12:54, Jun Rao via dev <de...@kafka.apache.org>
> > wrote:
> >
> > > Hi, Federico,
> > >
> > > Thanks for the updated KIP. A few more comments.
> > >
> > > JR20. "Groups that are active on the destination cluster (i.e., in any
> > > state other than EMPTY or DEAD) are skipped to avoid regressing offsets
> > > after failover."
> > > Does MirrorMetadataManager need to send an RPC to the group coordinator
> > to
> > > know the group's state?
> > >
> > > JR21. Are mirrored records validated against
> > > log.message.timestamp.before.max.ms and
> > log.message.timestamp.after.max.ms
> > > on the destination cluster?
> > >
> > > JR22. "Leader Epoch Bump (LEB): The leader epoch in the destination
> > cluster
> > > remains unchanged during replication. That means it is possible that
> the
> > > fetched batch carries a leader epoch that does not match with the local
> > > leader epoch."
> > > Is that correct? In the Leader Epoch Invariant section, there is logic
> to
> > > makes sure SLE > DLE can't happen.
> > >
> > > JR23. "A new dynamic topic level configuration
> > > mirror.support.unclean.leader.election (boolean, default false)
> controls
> > > whether LME truncation waits for all assigned replicas to converge, not
> > > just current ISR members. When enabled, the leader will not complete
> > > truncation until every replica has joined the ISR and truncated past
> the
> > > LME offset. This prevents subsequent unclean leader elections from
> > > introducing undetectable log divergence between clusters."
> > > Is this to prevent issues with ULE on the source or destination?
> > >
> > > JR24. "KIP-1327 introduces immutable creation timestamps on both
> > partitions
> > > and consumer groups to establish a happen-before relationship that
> > > determines which offset reset policy applies when a consumer group
> > > encounters a partition with no committed offset."
> > > This needs adjustment since we decided not to pursue KIP-1327.
> > >
> > > JR25. IncludePatterns/ExcludePatterns: Which one takes precedence if
> both
> > > are specified?
> > >
> > > JR26. For each record and RPC involving a state field, could you add a
> > > description of the all possible states?
> > >
> > > JR27. WriteMirrorStatesRequest
> > > { "name": "", "type": "int32", "versions": "0+", "default":
> "-1",
> > > "about": "The last mirror leader epoch, or -1 if not
> > available."
> > > },
> > > The field name is empty.
> > >
> > > JR28. Could you describe how the StateValidationOffset field is being
> > used?
> > >
> > > JR29. "The ReplicaManager which provides source offset, destination
> > offset,
> > > and lag for each partition."
> > > Which RPC is used to query the ReplicaManager?
> > >
> > > JR30. Failover: The original destination b stores LME under the mirror
> > name
> > > a-to-b. During fallback, the original source a creates a new mirror
> name
> > > b-to-a and uses it in DescribeClusterMirrors to obtain LME. How can the
> > > original destination cluster b find the right LME if the stored and
> > > requested mirror names don't match?
> > >
> > > JR31. Start Mirror Topics:
> > > JR31.1 "Controller persists include/exclude patterns as ConfigRecord
> > > entries on the MIRROR resource in the metadata log."
> > > Should we persist topics too? It's possible that the source topic
> doesn't
> > > exist yet when a mirror is started.
> > > JR31.2 "For each topic, the controller creates it on the destination if
> > it
> > > does not already exist, updates include/exclude topic pattern configs"
> > > What are include/exclude topic pattern configs? In the previous step,
> the
> > > controller already persists include/exclude patterns as ConfigRecord
> > > entries on the MIRROR resource in the metadata log.
> > > JR31.3 "Brokers receive the metadata update. The MirrorMetadataManager
> > > detects the new mirrorName and DesiredState from the TopicImage and
> > queries
> > > the coordinator for the current partition state.
> > > Partitions transition from UNKNOWN to LOG_TRUNCATION. During this
> state,
> > > LME truncation runs and waits for all ISR members (or all replicas if
> ULE
> > > is enabled)."
> > > The MirrorMetadataManager instance responsible for a mirror is not
> > > necessarily the same broker that leads a mirror partition. How can the
> > > MirrorMetadataManager instance perform truncation? Ditto for Stop
> Mirror
> > > Topics.
> > >
> > > JR32. Stop Mirror Topics: "For each topic, the controller writes a
> > > MirrorTopicStateChangeRecord with DesiredState=1 (stop)."
> > > Is this record kept forever in the metadata snapshot?
> > >
> > > JR33. Delete Mirror
> > > JR33.1 he controller tombstones => The controller tombstones
> > > JR34. "The mirror state records in __mirror_state internal topic are
> also
> > > tombstoned."
> > > Could you describe how MirrorTopicStateChangeRecord is tombstoned since
> > it
> > > doesn't have a key?
> > >
> > > JR34. AdminClient:
> > > JR34.1 Could you define the public facing classes such as
> > > CreateClusterMirrorResult, StartMirrorTopicsResult,
> > > PauseMirrorTopicsOptions, etc?
> > > JR34.2 describeClusterMirrors: How does it populate the clusterId field
> > in
> > > DescribeClusterMirrorsRequest?
> > >
> > > JR35. CreateTopic : "The replicas for the partition assignment are all
> > > active and not in fenced or controlled shutdown. This is to make sure
> > when
> > > a topic gets deleted and re-created with the same topic ID, the stale
> > > offline log dir won’t be treated as the active log dir after it becomes
> > > online"
> > > Could you explain why this is only needed for mirror topics, but not
> > > regular topics?
> > >
> > > JR36. Fetch: Could you explain why we need to add the MirrorLeaderEpoch
> > > field? I thought the destination cluster just issues a regular fetch
> > > request like a consumer.
> > >
> > > JR37. CreateClusterMirror: "The broker validates that the mirror name
> is
> > > not already in use, contains only permitted characters, and does not
> end
> > > with .stopped or .paused suffix."
> > > Could you explain why the suffix restriction is necessary?
> > >
> > > JR38. MirrorMetadataManager: Could you explain how mirrors are assigned
> > to
> > > MirrorMetadataManager instances? Is it based on the _mrror_state topic
> > > partitions?
> > >
> > > JR39. StopMirrorTopics: Do we need TopicId? How does the AdminClient
> get
> > > it?
> > >
> > > JR40. ReadMirrorStatesResponse: Could you explain how PreviousState is
> > > being used?
> > >
> > > JR41. PartitionChangeRecord.MinLeaderEpoch: Currently, leader epochs
> are
> > > implicitly derived from PartitionChangeRecord. By adding this field,
> the
> > > controller will sometimes derive the epoch and sometimes use the
> > explicitly
> > > stored epoch, which can be confusing. Would it be better to always
> store
> > > the leader epoch explicitly in PartitionChangeRecord?
> > >
> > > JR42. MirrorPartitionStateValue: Why does it use topicName, not
> topicId?
> > >
> > > JR43. Mirror config: What about authentication (sasl, ssl, etc) related
> > > configs?
> > >
> > > JR44. Could you also describe how a partition leader picks up the
> > mirroring
> > > flow when it changes?
> > >
> > > JR45. Broker config: sasl.mechanism.mirror.admin.protocol
> > > Is that for mirror-related RPCs between brokers and controllers?
> > >
> > > JR46. Compatibility
> > > Since this KIP changes the metadata record format, we need an MV bump.
> > For
> > > mirror-related records and internal RPCs, should we introduce a
> > > mirror-specific feature like GV/TV to gate them?
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jul 7, 2026 at 6:40 AM Federico Valeri <fe...@gmail.com>
> > > wrote:
> > >
> > > > Hi Michael, thanks for your review.
> > > >
> > > > MM1: It was a bit confusing but the coordinator is the scheduler and
> > > > the manager is the executor. That said, given that the manager
> already
> > > > has a scheduler, we moved that logic there and updated the KIP.
> > > >
> > > > MM2: Good point. A deleted source topic would now cause the mirror
> > > > partitions on the destination cluster to move to FAILED state. You
> can
> > > > easily identify them using the describe command with the --failed
> > > > flag.
> > > >
> > > > MM3: We have a log message at warn level. I think this is enough.
> > > >
> > > > MM4: Correct. KIP updated.
> > > >
> > > > MM5: Correct. KIP updated.
> > > >
> > > > MM6: The isolation level is hardcoded to READ_UNCOMMITTED. Not sure
> > > > about the actual use case, but we can certainly make it configurable
> > > > in a follow-up KIP.
> > > >
> > > > MM7: For quota related topic configurations,
> > > > follower.replication.throttled.replicas and
> > > > leader.replication.throttled.replicas are excluded by default in MM2
> > > > and we apply the same logic. mirror.support.unclean.leader.election
> > > > and mirror.replication.throttled.replicas are included by default
> > > > because they are mirror related configurations.
> > > >
> > > > MM8: Updated the field description to: Topic patterns to remove from
> > > > mirror.topics.include or add to mirror.topics.exclude.
> > > >
> > > > MM9: Not needed. KIP updated.
> > > >
> > > > MM10: The config is needed because the listener can have multiple
> > > > enabled SASL mechanisms and the listener name alone doesn't select
> > > > one. The naming follows the existing
> > > > sasl.mechanism.inter.broker.protocol convention, so renaming would be
> > > > inconsistent with that precedent.
> > > >
> > > > MM11: Correct. KIP updated.
> > > >
> > > > MM12: No, when mirror.support.unclean.leader.election=true, we’ll
> wait
> > > > for all replicas, not just ISR replicas to complete truncation.
> > > >
> > > > MM13: Correct. KIP updated.
> > > >
> > > > MM14: We missed the “MIRROR_TOPIC_NOT_STOPPED”, so now: start on non
> > > > stopped throws MIRROR_TOPIC_NOT_STOPPED. Stop on paused throws
> > > > MIRROR_TOPIC_ALREADY_PAUSED. Pause on stopped throws
> > > > MIRROR_TOPIC_BEING_STOPPED. Resume on non paused throws
> > > > MIRROR_TOPIC_NOT_PAUSED. Stop on stopped throws
> > > > TOPIC_NOT_IN_CLUSTER_MIRROR.
> > > >
> > > >
> > > >
> > > > On Fri, Jul 3, 2026 at 5:26 PM Mickael Maison <
> > mickael.maison@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > Thanks for the KIP!
> > > > > This will definitively be a great addition to Kafka.
> > > > >
> > > > > MM1: In the description of the main components,
> > > > > ClusterMirrorCoordinator has this responsibility:
> > > > > "Metadata Refresh Scheduling: The coordinator schedules periodic
> > > > > metadata refresh operations by invoking a metadata manager every 30
> > > > > seconds by default. This ensures that configuration updates and
> group
> > > > > offset commits in the source cluster are continuously propagated to
> > > > > the destination cluster. The refresh interval is configurable via
> > > > > mirror.metadata.refresh.interval.ms."
> > > > > Is that accurate because MirrorMetadataManager has pretty much the
> > > > > same responsibility.
> > > > >
> > > > > MM2: About topic deletions, the KIP states: "the operator would
> need
> > > > > to manually remove the topic from the mirror"
> > > > > Is there a way to find topics that have been deleted in the source
> > > > > cluster for an operator to do that action?
> > > > >
> > > > > MM3: About group offsets: It states: "Groups with active members on
> > > > > the destination are skipped."
> > > > > Should these be logged too?
> > > > >
> > > > > MM4: In the security control section, the DeleteClusterMirror RPC
> > > > > requires Alter on the ClusterMirror resource. Should we use Delete
> > > > > instead of Alter?
> > > > > On non-cluster resources, we typically use Delete for deletion. See
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://kafka.apache.org/43/security/authorization-and-acls/*operations-and-resources-on-protocols__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqU3AKr-48$
> > > > >
> > > > > MM5: In Destination cluster permissions, I think FindCoordinator
> > > > > should also need Describe on the Group resources for regular
> consumer
> > > > > groups.
> > > > >
> > > > > MM6: In Exactly-Once Semantics, it states: "The mirror fetcher
> thread
> > > > > uses READ_UNCOMMITTED isolation"
> > > > > Could this be configurable? If so we can do this in a follow-up KIP
> > > > >
> > > > > MM7: It seems quotas are not mirrored between the clusters?
> > > > > Is that omission intentional?
> > > > >
> > > > > MM8: In StopMirrorTopicsRequest I'm not sure I understand what the
> > > > > Patterns field is?
> > > > > Is it a pattern for the topics to stop? If so the description seems
> > odd
> > > > >
> > > > > MM9: In
> > > >
> > >
> >
> StopMirrorTopicsResponse/StartMirrorTopicsResponse/PauseMirrorTopicsResponse/ResumeMirrorTopicsResponse
> > > > > do we really need the mirror name?
> > > > >
> > > > > MM10: In Configuration / Broker:
> > > > > Can you clarify why we need sasl.mechanism.mirror.admin.protocol?
> > > > > Can't we retrieve the protocol from mirror.admin.listener.name or
> > > > > inter.broker.listener.name?
> > > > > If we really need it, should it be named
> > > > > mirror.admin.sasl.mechanism.protocol to match the other names?
> > > > >
> > > > > MM11: In Configuration / Mirror
> > > > > Should it be mirror.acls.include instead of mirror.acl.include to
> > > > > match mirror.groups.include and mirror.topics.include?
> > > > >
> > > > > MM12: Is the description of mirror.support.unclean.leader.election
> > > > inverted?
> > > > >
> > > > > MM13: The description of CLUSTER_MIRROR_NOT_EMPTY states: "The
> > cluster
> > > > > mirror still has active or non-removed topics"
> > > > > Should it be "The cluster mirror still has active or non-stopped
> > > topics"?
> > > > >
> > > > > MM14: We have the MIRROR_TOPIC_BEING_STOPPED error when stopping
> > topic
> > > > > already in stopping state? What happens when pausing topics already
> > in
> > > > > pausing state?
> > > > > Related, I assume when stopping a stopped topic, you get
> > > > > TOPIC_NOT_IN_CLUSTER_MIRROR
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Mon, Jun 22, 2026 at 7:47 PM Andrew Schofield <
> > > schofielaj@apache.org>
> > > > wrote:
> > > > > >
> > > > > > Hi Fede,
> > > > > > Thanks for your replies.
> > > > > >
> > > > > > AS30: Yes, that's right. Two different end markers for the same
> > > > transaction, and essentially unbounded duration of the inconsistency
> > > since
> > > > KIP-939. I agree that it's a class of transaction state divergence,
> > like
> > > > KAFKA-20716. For KIP-1279, the most important thing is that consumers
> > of
> > > > the mirrored topic must not get confused by the markers. For example,
> > > when
> > > > the Fetch response contains the list of aborted transactions,
> provided
> > we
> > > > are sure that it can be built properly without confusion from the
> extra
> > > > marker, then we are good. It came to mind because the way this is
> > > > represented in the protocol for the responses is somewhat fragile I
> > feel,
> > > > so I'm a bit paranoid that a mis-alignment could break it.
> > > > > >
> > > > > > Thanks,
> > > > > > Andrew
> > > > > >
> > > > > > On 2026/06/19 12:16:52 Federico Valeri wrote:
> > > > > > > Hi Andrew, me, again.
> > > > > > >
> > > > > > > Given that READ_COMMITTED consumers are unaffected because the
> > > ABORT
> > > > > > > is already recorded in the transaction index and the dangling
> > > COMMIT
> > > > > > > produces no CompletedTxn (no index and LSO updates), the
> > practical
> > > > > > > impact is limited. However, __transaction_state records the
> > > > > > > transaction as COMMITTED while the partition log has it as
> > aborted.
> > > > > > > While analyzing this, we discovered that this type of
> divergence
> > is
> > > > > > > not unique to mirroring. KAFKA-20716 documents the same
> category
> > of
> > > > > > > inconsistency after unclean leader elections, where lost
> > > transaction
> > > > > > > markers cause the LSO to get stuck (much bigger problem). We
> > > believe
> > > > > > > that solving the transaction state divergence issue would also
> > > close
> > > > > > > this mirroring scenario. Wdyt?
> > > > > > >
> > > > > > > On Thu, Jun 18, 2026 at 10:36 PM Federico Valeri <
> > > > fedevaleri@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi Andrew, about AS30, I had a closer look at KIP-939, and I
> > > think
> > > > > > > > this is what you mean:
> > > > > > > >
> > > > > > > > The problem is that a WriteTxnMarker request for a PID
> unknown
> > to
> > > > the
> > > > > > > > ProducerStateManager (cleared by MIRROR_PID_RESET) does not
> > fail.
> > > > The
> > > > > > > > end transaction marker is still appended, but transaction
> index
> > > and
> > > > > > > > LSO are unaffected (ProducerStateManager.prepareUpdate,
> > > > > > > > ProducerAppendInfo.appendEndTxnMarker). This may lead to a
> > > > situation
> > > > > > > > where we have two different end txn markers for the same
> > > > transaction:
> > > > > > > > ABORT from the stop mirroring operation, followed by a COMMIT
> > > from
> > > > the
> > > > > > > > local Txn Coordinator retrying its WriteTxnMarker after the
> > > > partition
> > > > > > > > becomes writable again.
> > > > > > > >
> > > > > > > > READ_COMMITTED consumers still see the earlier ABORT in the
> > > > > > > > transaction index and skip the aborted data. The visible
> > outcome
> > > > for
> > > > > > > > consumers is correct. However, __transaction_state on the
> > > original
> > > > > > > > source cluster now records the transaction as COMMITTED while
> > the
> > > > > > > > partition log has it as aborted, creating a silent
> > inconsistency
> > > > that
> > > > > > > > tools querying transaction state will not detect. The window
> > for
> > > > this
> > > > > > > > race is bounded by transaction.max.timeout.ms (default 15
> > > > minutes),
> > > > > > > > but becomes unbounded with KIP-939 because it allows prepared
> > > > > > > > transactions to sit indefinitely.
> > > > > > > >
> > > > > > > > We could move the coordinator state to “complete abort” at
> the
> > > > cost of
> > > > > > > > an RPC. This would prevent any retry, but that would break
> the
> > > 2PC
> > > > > > > > contract (Kafka must hold the prepared state until the
> external
> > > > > > > > coordinator decides). Also, before stop there would be a
> number
> > > of
> > > > > > > > retries constantly going on. We definitely need a better
> > > solution.
> > > > > > > >
> > > > > > > > On Thu, Jun 18, 2026 at 11:36 AM Federico Valeri <
> > > > fedevaleri@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi Andrew, thanks for your careful review.
> > > > > > > > >
> > > > > > > > > AS28: We reworked that section and added some diagrams to
> > > > illustrate
> > > > > > > > > problem and solutions. Let us know if it is better.
> > > > > > > > >
> > > > > > > > > AS30: We updated that table, but we are not sure about the
> > > > problem you
> > > > > > > > > mention. Can you clarify and send us the sequence of
> actions
> > > that
> > > > > > > > > would lead to that problem?
> > > > > > > > >
> > > > > > > > > AS32: You are right. Replaced
> > > > StartMirrorTopicsRequestData.TopicData
> > > > > > > > > with a Controller.MirrorTopicMetadata record. Also renamed
> > > > TopicData
> > > > > > > > > to TopicMetadata in all schemas.
> > > > > > > > >
> > > > > > > > > AS33: Good idea. Added per-topic error messages in the RPC
> > > > responses
> > > > > > > > > you mentioned.
> > > > > > > > >
> > > > > > > > > AS29, AS31, AS34 (both of them), AS35, AS36: These small
> > issues
> > > > are
> > > > > > > > > all fixed now.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jun 9, 2026 at 9:58 PM Andrew Schofield <
> > > > schofielaj@apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Fede and friends,
> > > > > > > > > > This KIP is shaping up nicely. The change to the way that
> > > > state management for the mirrors is done is much more elegant. The
> > > removal
> > > > of the PID mapping is also a nice improvement.
> > > > > > > > > >
> > > > > > > > > > Inevitably, I have some more comments.
> > > > > > > > > >
> > > > > > > > > > AS28: I understand KAFKA-18723 and how leader epochs work
> > in
> > > > that case, but the enhancements that you made to the Fetch RPC and
> how
> > > they
> > > > work escapes me. Please could we have some diagrams to illustrate the
> > > > various epochs?
> > > > > > > > > >
> > > > > > > > > > AS29: I like the addition of share group offset syncing.
> > You
> > > > need AlterShareGroupOffsets and DescribeShareGroupOffsets in the
> tables
> > > of
> > > > RPCs and permissions.
> > > > > > > > > >
> > > > > > > > > > AS30: It is certainly true that EOS is not supported.
> Using
> > > > cluster mirroring on transactional data does risk breaking
> > transactional
> > > > atomicity for transactions in flight at the time that mirroring was
> > > > stopped. That's a known limitation, and it's an effect of per-topic
> > async
> > > > replication. Writing ABORT markers when stop mirroring is triggered
> is
> > > > interesting. I just wonder whether it's truly safe.
> > > > > > > > > >
> > > > > > > > > > Could you enhance the tables of records in the
> Exactly-Once
> > > > Semantics section to include leader epochs? I think that the leader
> > epoch
> > > > for the ABORT markers will have been bumped as the mirroring stopped
> > and
> > > > the partitions became writable. However, I worry that someone might
> > > switch
> > > > the mirroring back in the other direction before the transaction
> which
> > > was
> > > > rudely partially aborted has completed on the original source
> cluster,
> > > and
> > > > that the effect of the transaction coordinator completing the
> > > transaction,
> > > > potentially writing duplicate transaction markers after the ABORT
> > markers
> > > > which will have been replicated back again.
> > > > > > > > > >
> > > > > > > > > > This is probably only a realistic possibility when
> KIP-939
> > is
> > > > present, but it is accepted and I'm sure it will land soon enough. I
> > > would
> > > > say this KIP should be prepared for the existence of long-running
> > > > transactions so we don't dig a hole to fall into later.
> > > > > > > > > >
> > > > > > > > > > AS31: In the section on the Admin Client, the description
> > for
> > > > Admin.createClusterMirror(String, Map, CreateClusterMirrorOptions) is
> > > > duplicated.
> > > > > > > > > >
> > > > > > > > > > AS32: The use of StartMirrorTopicsRequestData.TopicData
> in
> > an
> > > > external interface looks odd.
> > > > > > > > > >
> > > > > > > > > > AS33: I suggest per-topic error messages in the RPC
> > responses
> > > > such as PauseMirrorTopics so you can pass back more context, such as
> > the
> > > > state if the request failed due to a state mismatch.
> > > > > > > > > >
> > > > > > > > > > AS34: There is an unnamed field in
> > > > DescribeClusterMirrorsResponse. The same in LastMirrorEpochsValue.
> > > > > > > > > >
> > > > > > > > > > AS34: Some of the fields in BumpLeaderEpochsRequest start
> > > with
> > > > lower-case letters. Also, there are quite a few fields in the RPCs in
> > > > general that should be version "0+" which are "0".
> > > > > > > > > >
> > > > > > > > > > AS35: Sometimes the partition states in the RPCs are
> > string,
> > > > but sometimes int8.
> > > > > > > > > >
> > > > > > > > > > AS36: The base level for share group support is probably
> > 4.1.
> > > > All of the RPCs were present in 4.1, although they were only enabled
> by
> > > > default in 4.2.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Andrew
> > > > > > > > > >
> > > > > > > > > > On 2026/06/04 11:01:36 Federico Valeri wrote:
> > > > > > > > > > > Hello, additional updates and replies:
> > > > > > > > > > >
> > > > > > > > > > > VK12: Consumer data loss risk is a good point. We
> updated
> > > > the "Group
> > > > > > > > > > > Offsets" paragraph.
> > > > > > > > > > >
> > > > > > > > > > > Some of you raised the point that using mirror.name
> > > > suffixes for pause
> > > > > > > > > > > and stop operations is a bit inelegant and we agreed.
> Now
> > > we
> > > > propose a
> > > > > > > > > > > better approach:
> > > > > > > > > > >
> > > > > > > > > > > We replaced the topic config based approach (
> mirror.name
> > > and
> > > > > > > > > > > .stopped/.paused suffix conventions) with a first class
> > > > metadata
> > > > > > > > > > > record for tracking mirror topic state changes. A new
> > > > > > > > > > > MirrorTopicStateChangeRecord is added to the metadata
> log
> > > > with three
> > > > > > > > > > > fields: TopicId (uuid), MirrorName (string), and
> > > > DesiredState (int8,
> > > > > > > > > > > where 0 = start mirroring, 1 = stop, 2 = pause). When a
> > > user
> > > > calls
> > > > > > > > > > > startMirrorTopics, stopMirrorTopics, or
> > pauseMirrorTopics,
> > > > the
> > > > > > > > > > > controller writes this record to the metadata log
> instead
> > > of
> > > > > > > > > > > manipulating config suffixes. Brokers receive the
> record
> > > via
> > > > metadata
> > > > > > > > > > > updates and react accordingly, triggering partition
> level
> > > > state
> > > > > > > > > > > transitions through the existing MirrorPartitionState
> > state
> > > > machine.
> > > > > > > > > > > More details in the updated KIP.
> > > > > > > > > > >
> > > > > > > > > > > Let us know what you think.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Fede
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 2, 2026 at 8:50 AM Luke Chen <
> > > showuon@gmail.com>
> > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Viquar,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the comment.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding VK12, thanks for raising the issue.
> > > > > > > > > > > > We didn't think about that.
> > > > > > > > > > > > I agree that compared with data re-processing, it's
> > worse
> > > > to have data loss.
> > > > > > > > > > > > The proposed formula makes sense to me.
> > > > > > > > > > > > *syncedOffset = max(destinationLogStartOffset,
> > > > min(sourceCommitted,
> > > > > > > > > > > > destinationLEO))*
> > > > > > > > > > > >
> > > > > > > > > > > > Let us have some discussion internally and then
> update
> > > the
> > > > KIP and reply to
> > > > > > > > > > > > the thread.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Luke
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 20, 2026 at 2:39 AM Federico Valeri <
> > > > fedevaleri@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Rajini,
> > > > > > > > > > > > >
> > > > > > > > > > > > > RS14: Done. Missed that, sorry.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks again.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 19, 2026 at 7:28 PM Rajini Sivaram <
> > > > rajinisivaram@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Federico,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the updates. Just one minor point,
> apart
> > > > from that, looks
> > > > > > > > > > > > > good.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > RS14: KIP still shows "bin/kafka-configs.sh
> > > > --bootstrap-server :9094
> > > > > > > > > > > > > > --entity-type mirrors".
> > > > > > > > > > > > > > Will be good to change that to `--entity-type
> > > > cluster-mirrors`.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Rajini
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, May 19, 2026 at 5:28 PM vaquar khan <
> > > > vaquar.khan@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Federico and team,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you for your detailed response on May 11,
> > > > 2026. I greatly
> > > > > > > > > > > > > appreciate
> > > > > > > > > > > > > > > the collaborative effort over the past few
> months
> > > to
> > > > harden the
> > > > > > > > > > > > > KIP-1279
> > > > > > > > > > > > > > > architecture.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > * 1. Resolved Architectural & Stability
> > > > Vulnerabilities*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I am pleased to confirm that the KIP has
> > > > successfully integrated fixes
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the critical vulnerabilities I flagged, thereby
> > > > protecting the
> > > > > > > > > > > > > cluster's
> > > > > > > > > > > > > > > state machine, memory bounds, and control
> plane.
> > > > Consider the following
> > > > > > > > > > > > > > > items fully resolved on my end:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *VK4 / VK8 (Negative PID Bug & State Machine
> > > > Failure):* I previously
> > > > > > > > > > > > > > > identified that the proposed
> > -(sourceProducerId
> > > +
> > > > 2) PID rewriting
> > > > > > > > > > > > > > > formula would fundamentally break
> > > hasProducerId()
> > > > in
> > > > > > > > > > > > > > > AbstractRecordBatch.java, causing the broker
> > to
> > > > bypass
> > > > > > > > > > > > > > > ProducerStateManager.update() and default
> the
> > > > Last Stable Offset
> > > > > > > > > > > > > (LSO)
> > > > > > > > > > > > > > > to the High Watermark. I am glad to see the
> > KIP
> > > > authors abandoned
> > > > > > > > > > > > > PID
> > > > > > > > > > > > > > > mapping entirely and adopted the
> deterministic
> > > > MIRROR_PID_RESET
> > > > > > > > > > > > > control
> > > > > > > > > > > > > > > record barrier. This perfectly protects
> > Kafka's
> > > > exactly-once
> > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *VK11 (TransactionIndex Rebuild Ambiguity):*
> > > > Thank you for
> > > > > > > > > > > > > officially
> > > > > > > > > > > > > > > confirming that the TransactionIndex is
> > strictly
> > > > rebuilt locally
> > > > > > > > > > > > > during
> > > > > > > > > > > > > > > log appends rather than copied
> byte-for-byte.
> > > > This resolves my
> > > > > > > > > > > > > concern
> > > > > > > > > > > > > > > regarding PID mismatches inducing consumer
> > read
> > > > anomalies.
> > > > > > > > > > > > > > > -
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *VK1 (Thundering Herd / OOM Heap
> Allocation):*
> > > > Your clarification
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > fetcher threads multiplex partitions meaning
> > the
> > > > memory footprint is
> > > > > > > > > > > > > > > strictly bounded by num_fetcher_threads *
> > > > response_max_bytes rather
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > a concurrent 1MB buffer per partition fully
> > > > resolves my concern
> > > > > > > > > > > > > > > regarding
> > > > > > > > > > > > > > > 50GB broker-wide OOM spikes during mass
> > > partition
> > > > wake-ups.
> > > > > > > > > > > > > > > -
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *VK3 (Control Plane Hotspots):* I had
> flagged
> > > the
> > > > severe risk of
> > > > > > > > > > > > > > > metadata saturation on a single broker
> during
> > a
> > > > "link flap" event.
> > > > > > > > > > > > > Your
> > > > > > > > > > > > > > > confirmation that the __mirror_state topic
> > > > utilizes a compound hash
> > > > > > > > > > > > > > > of (mirrorName,
> > > > > > > > > > > > > > > topicId, partition number) mathematically
> > > ensures
> > > > that the 50,000+
> > > > > > > > > > > > > state
> > > > > > > > > > > > > > > transitions will be safely distributed
> across
> > > the
> > > > cluster,
> > > > > > > > > > > > > neutralizing
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > single-node hotspot risk.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. Outstanding Critical Blocker: VK12 (Offset
> > Sync
> > > > Data Loss)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding VK12, there is a fundamental
> > > > misunderstanding in your
> > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > reply. You stated: *"The scenario described can
> > > only
> > > > occur if offsets
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > force-written to an active group, which the
> > design
> > > > prevents."*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > My concern has absolutely nothing to do with
> > > > overwriting active
> > > > > > > > > > > > > groups. My
> > > > > > > > > > > > > > > concern applies strictly to the normal
> > > > synchronization of inactive/dead
> > > > > > > > > > > > > > > groups, and is based directly on the race
> > condition
> > > > currently
> > > > > > > > > > > > > documented in
> > > > > > > > > > > > > > > the official KIP-1279 text:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *"During offset synchronization, the committed
> > > > offset in the
> > > > > > > > > > > > > destination
> > > > > > > > > > > > > > > cluster may temporarily exceed the current log
> > end
> > > > offset (LEO) of the
> > > > > > > > > > > > > > > mirror topic... consumers attempting to resume
> > from
> > > > offset 100 will
> > > > > > > > > > > > > receive
> > > > > > > > > > > > > > > an OffsetOutOfRangeException. To handle this
> > > > gracefully, consumers
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > configure auto.offset.reset=latest..."*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If a failover happens during this documented
> > > > divergence window, a
> > > > > > > > > > > > > > > reconnecting consumer will hit the
> > > > OffsetOutOfRangeException. If the
> > > > > > > > > > > > > > > downstream consumer follows the KIP's official
> > > > advice and relies on
> > > > > > > > > > > > > > > auto.offset.reset=latest, the consumer will
> jump
> > to
> > > > the absolute newest
> > > > > > > > > > > > > > > offset on the partition. *This completely skips
> > any
> > > > newly produced
> > > > > > > > > > > > > records
> > > > > > > > > > > > > > > that arrived between the failover and the
> > consumer
> > > > reconnecting,
> > > > > > > > > > > > > resulting
> > > > > > > > > > > > > > > in silent, unrecoverable data loss for the
> > > > downstream application.*
> > > > > > > > > > > > > > > *Proposed Resolution: Double-Clamped Offset
> > Safety
> > > > Invariant* Instead
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > requiring consumers to use
> > auto.offset.reset=latest
> > > > and endorsing a
> > > > > > > > > > > > > known
> > > > > > > > > > > > > > > data loss vector, I propose that the
> > > > ClusterMirrorCoordinator enforce a
> > > > > > > > > > > > > > > pre-persist offset validation invariant during
> > > > offset synchronization.
> > > > > > > > > > > > > > > Before any translated offset is committed to
> the
> > > > destination cluster,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > coordinator must apply a double-clamp:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *syncedOffset = max(destinationLSO,
> > > > min(sourceCommitted,
> > > > > > > > > > > > > destinationLEO))*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Where destinationLSO is the Log Start Offset
> > > > (earliest readable
> > > > > > > > > > > > > position
> > > > > > > > > > > > > > > post-retention) and destinationLEO is the Log
> End
> > > > Offset (latest
> > > > > > > > > > > > > replicated
> > > > > > > > > > > > > > > position) of the destination partition.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This guarantees that every persisted offset
> falls
> > > > within the physically
> > > > > > > > > > > > > > > valid range eliminating both the
> > > > OffsetOutOfRangeException caused by
> > > > > > > > > > > > > > > exceeding the LEO during replication lag, and
> the
> > > > expired-offset rewind
> > > > > > > > > > > > > > > caused by falling below the LSO due to
> > destination
> > > > retention policies.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The worst-case trade-off under this invariant
> is
> > > > bounded re-processing
> > > > > > > > > > > > > > > proportional to the replication lag at failover
> > > time
> > > > not total
> > > > > > > > > > > > > partition
> > > > > > > > > > > > > > > depth which is perfectly consistent with
> Kafka's
> > > > documented
> > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > delivery guarantees. Broad enterprise
> production
> > > > evidence from
> > > > > > > > > > > > > large-scale
> > > > > > > > > > > > > > > cross-cluster failovers confirms that state
> > checks
> > > > alone (preventing
> > > > > > > > > > > > > active
> > > > > > > > > > > > > > > group overwrites) are insufficient; strict
> offset
> > > > bounds clamping is
> > > > > > > > > > > > > > > required to achieve enterprise-grade data
> > > integrity.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I look forward to your thoughts on implementing
> > > this
> > > > final offset
> > > > > > > > > > > > > capping
> > > > > > > > > > > > > > > logic. Once VK12 is patched, I believe this
> > > > architecture will be
> > > > > > > > > > > > > > > exceptionally robust and ready for enterprise
> > > > deployment.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Viquar Khan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, 18 May 2026 at 14:23, Rajini Sivaram <
> > > > rajinisivaram@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Federico,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the updates! The KIP is looking
> > good.
> > > A
> > > > few more small
> > > > > > > > > > > > > > > comments.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > RS13: A couple of places still refer to
> > > > `kafka-mirror.sh` like under
> > > > > > > > > > > > > > > > `Failover Process`. Can we change them to
> > > > > > > > > > > > > `*kafka-cluster-mirrors.sh*`?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > RS14: Should we change `--entity-type
> mirrors`
> > > for
> > > > kafka-configs to
> > > > > > > > > > > > > be `
> > > > > > > > > > > > > > > > --entity-type *cluster-mirrors*` to be
> > > consistent?
> > > > Also,
> > > > > > > > > > > > > > > > CLUSTER_MIRROR((byte)
> > > > > > > > > > > > > > > > 64, "mirror"); could be `*cluster-mirror*`?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > RS15: It may be useful to rename
> > > > `mirror.topic.num.partitions` and `
> > > > > > > > > > > > > > > > mirror.topic.replication.factor` since they
> are
> > > > very similar to `
> > > > > > > > > > > > > > > > mirror.topic.properties.exclude`, but the
> > > > `mirror.topic` prefix
> > > > > > > > > > > > > refers to
> > > > > > > > > > > > > > > > different topics (the internal topic for the
> > > first
> > > > two and actual
> > > > > > > > > > > > > mirror
> > > > > > > > > > > > > > > > topics for the other one).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > RS16: ACL Sync: KIP says "Deletes ACLs that
> > exist
> > > > in destination but
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > source using DeleteAcls request."
> > > > > > > > > > > > > > > > What happens if someone creates an ACL on the
> > > > destination to deny
> > > > > > > > > > > > > > > > User:Alice access to all topics?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. If that ACL also existed on the source
> > > > cluster and then it was
> > > > > > > > > > > > > > > > removed, will it get removed from the
> > > > destination?
> > > > > > > > > > > > > > > > 2. If that ACL never existed on the source
> > > > cluster, will it get
> > > > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > > from the destination?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > RS17: The table in the Source ACLs section
> > says:
> > > > > > > > > > > > > "DescribeClusterMirrors
> > > > > > > > > > > > > > > > MC Read Cluster Log
> truncation"
> > > > > > > > > > > > > > > > Should that be `ClusterMirror:Read` instead
> of
> > > > `Cluster:Read`?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Rajini
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, May 15, 2026 at 8:41 AM Federico
> > Valeri <
> > > > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Rajini, we finally addressed the API and
> > > tool
> > > > naming
> > > > > > > > > > > > > refactoring as
> > > > > > > > > > > > > > > > > you suggested in RS6. Please take a look
> when
> > > > you have time.
> > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:17 PM Federico
> > > Valeri <
> > > > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hello all, I want to highlight a couple
> of
> > > new
> > > > paragraphs:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Leader Epoch Invariant: Cluster
> > mirroring
> > > > enforces the
> > > > > > > > > > > > > invariant
> > > > > > > > > > > > > > > > > > that the destination leader epoch must
> > always
> > > > be greater than or
> > > > > > > > > > > > > > > equal
> > > > > > > > > > > > > > > > > > to the source leader epoch (DLE>=SLE).
> > > Without
> > > > this, consumers
> > > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > > destination cluster can get stuck in an
> > > > infinite metadata refresh
> > > > > > > > > > > > > > > loop
> > > > > > > > > > > > > > > > > > when they encounter committed offsets
> > > carrying
> > > > source epochs
> > > > > > > > > > > > > higher
> > > > > > > > > > > > > > > > > > than the local epoch. The invariant is
> > > > maintained through three
> > > > > > > > > > > > > > > > > > mechanisms: reactive bumping (epoch
> fencing
> > > > triggered when SLE >
> > > > > > > > > > > > > DLE
> > > > > > > > > > > > > > > > > > during fetch), proactive bumping
> (scheduled
> > > > when SLE approaches
> > > > > > > > > > > > > DLE
> > > > > > > > > > > > > > > > > > within a threshold), and periodic bumping
> > > > (checked during
> > > > > > > > > > > > > coordinator
> > > > > > > > > > > > > > > > > > metadata sync).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973*KIP1279:ClusterMirroring-LeaderEpochInvariant__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqUcl_uWkU$
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. Group Offsets: The coordinator
> > > periodically
> > > > syncs consumer and
> > > > > > > > > > > > > > > > > > share group offsets from the source
> cluster
> > > to
> > > > the destination
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > mirrored topics. Groups are filtered by
> > > > configurable
> > > > > > > > > > > > > include/exclude
> > > > > > > > > > > > > > > > > > patterns, and offsets are only synced for
> > > > groups that are not
> > > > > > > > > > > > > > > > > > currently active on the destination
> > cluster,
> > > > preventing
> > > > > > > > > > > > > overwrites of
> > > > > > > > > > > > > > > > > > local consumer progress. Because source
> and
> > > > destination share the
> > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > topic offsets (no offset translation),
> > synced
> > > > offsets can be used
> > > > > > > > > > > > > > > > > > directly without mapping.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973*KIP1279:ClusterMirroring-GroupOffsets__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqU-JubnOA$
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > These new paragraphs directly address
> some
> > of
> > > > your questions,
> > > > > > > > > > > > > but let
> > > > > > > > > > > > > > > > > > me list them here:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > JR2: Yes, we removed the incorrect phrase
> > and
> > > > added more details
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the paragraph.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > JR4: When source cluster topic has tiered
> > > > storage enabled, CM
> > > > > > > > > > > > > works
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > mirroring remote and local log into
> > > > destination cluster. When
> > > > > > > > > > > > > > > > > > destination cluster topic has tiered
> > storage
> > > > enabled, CM fails in
> > > > > > > > > > > > > > > > > > PREPARING state because the LME may be in
> > > > remote storage, but
> > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > > fine if already MIRRORING because no
> > > > truncation is needed.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > JR11: See "Leader Epoch Invariant"
> > paragraph
> > > > mentioned above.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > JR13: We can't support stateful Streams
> > > > application because
> > > > > > > > > > > > > > > > > > asynchronous replication cannot preserve
> > the
> > > > transactional
> > > > > > > > > > > > > boundaries
> > > > > > > > > > > > > > > > > > between input offset commits, state store
> > > > mutations written to
> > > > > > > > > > > > > > > > > > changelog topics, and intermediate
> records
> > > > written to repartition
> > > > > > > > > > > > > > > > > > topics. The synchronous extension of this
> > > > design will be able to
> > > > > > > > > > > > > > > > > > support them. Existing Features
> Integration
> > > > paragraph updated.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > JR18: See "Group Offsets" paragraph
> > mentioned
> > > > above.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > IY1: See "Group Offsets" paragraph
> > mentioned
> > > > above.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:08 PM Federico
> > > > Valeri <
> > > > > > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Vaquar,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > VK4/VK8: We don't do PID mapping
> anymore.
> > > > The KIP was updated
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > time ago with the new approach based on
> > the
> > > > new PID reset
> > > > > > > > > > > > > control
> > > > > > > > > > > > > > > > > > > record.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > VK11: The transaction index is always
> > built
> > > > locally during log
> > > > > > > > > > > > > > > > append,
> > > > > > > > > > > > > > > > > > > never copied.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > VK1: The 50,000 * 1MB = 50GB
> calculation
> > > > misunderstands the
> > > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > > model. Fetcher threads don't allocate
> one
> > > > buffer per partition.
> > > > > > > > > > > > > > > > Actual
> > > > > > > > > > > > > > > > > > > peak memory is roughly
> > num_fetcher_threads
> > > *
> > > > > > > > > > > > > response_max_bytes,
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > num_partitions * partition_max_bytes.
> > With
> > > 1
> > > > fetcher thread
> > > > > > > > > > > > > and the
> > > > > > > > > > > > > > > > > > > default response max, the memory
> > footprint
> > > > is modest
> > > > > > > > > > > > > regardless of
> > > > > > > > > > > > > > > > > > > partition count. We are leveraging the
> > same
> > > > proven pattern
> > > > > > > > > > > > > used by
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > internal replication.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > VK3: The __mirror_state topic uses
> > > > hash-based partitioning
> > > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > > mirrorName, topicId and partition
> number.
> > > > With the production
> > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > of 50 partitions, 50,000 partition
> > > > transitions distribute
> > > > > > > > > > > > > across
> > > > > > > > > > > > > > > ~50
> > > > > > > > > > > > > > > > > > > partition leaders on different brokers,
> > not
> > > > a single broker.
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > the same proven pattern as
> > > > __consumer_offsets, which handles
> > > > > > > > > > > > > > > millions
> > > > > > > > > > > > > > > > > > > of commits.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > VK12: The scenario described can only
> > occur
> > > > if offsets are
> > > > > > > > > > > > > > > > > > > force-written to an active group, which
> > the
> > > > design prevents.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Luke Chen <sh...@gmail.com>.
Hi Vaquar,
Thanks for your comments.
VK12:
Sorry, it's my bad that I forgot to update in the discussion thread and the
KIP.
Yes, we adopted your suggestion but forgot to update the KIP.
I've just updated it now. Sorry for the confusion.
VK10:
"(a) returning a precise error for cross-type collisions"
As you know, this is an existing logic, not related to this KIP.
I've opened KAFKA-20799 <https://issues.apache.org/jira/browse/KAFKA-20799> to
improve the error handling for group name collisions.
"(b) adding a pre-flight validation check to the CLI utility"
I'm wondering how much it helps because the group name is not fixed during
the mirroring period.
Even if we did a validation before starting the mirror, it is still
possible the newly created consumer group name, either in source or
destination cluster, will have a collision. Also, the "mirror.groups.include"
is a dynamic configuration. Users can update it anytime.
For example: source cluster has 0 consumer/share group, destination cluster
has 1 "share group" called: "test"
1. "mirror.groups.include=.*": Users start the mirror, and later create a
consumer group "test" in the source cluster, which collides with the one in
the destination cluster when sync up.
2. "mirror.groups.include=dummy": Users start the mirror, and later create
a consumer group "test" in the source cluster. No group offset sync up
happens. But later, the "mirror.groups.include" config is updated to
"mirror.groups.include=test",
so the name collision still happens.
For case (1), when mirror starts, we don't know if the source cluster will
create a classic consumer group named "test" in the future. There is
nothing we can do.
For case (2), even if we add a validation each time the
"mirror.groups.include/exclude"
config updates. That still doesn't resolve this issue because we don't have
a way to distinguish an offset commit is from mirror sync up or from user
creation. For example, "mirror.groups.include=.*" changes to
"mirror.groups.include=test".
When the change triggers, should we treat the existing "test" group in the
destination cluster as "name collision" or we should pass the validation
because the "test" could be created and synced in the mirror? We have no
way to distinguish it.
My thought is to improve the option (a) in KAFKA-20799
<https://issues.apache.org/jira/browse/KAFKA-20799> and make the error log
clear to users.
What do you think?
Thank you,
Luke
On Mon, Jul 13, 2026 at 7:54 AM vaquar khan <va...@gmail.com> wrote:
> Hi Federico, Luke, Omnia, and all,
>
> Thanks for the continued work on this. The KIP has come a long way. I want
> to close out the items we have settled so the record is clean, then address
> the remaining data-safety item where the current text does not yet
> implement the necessary architectural guard.
>
>
> *RESOLVED / CLOSED ON MY END*
> - VK2 (Poison pill blast radius): Resolved. Partition-level FAILED
> isolation in the fetcher pool is the correct model.
>
> - VK5 (Infinite loops): Resolved. Read-only enforcement on mirrored
> topics makes A->B->A physically impossible.
>
> - VK6 (Unclean-election divergence): Accepted as a documented
> limitation, and the updated ULE sub-design thoroughly addresses my initial
> concerns.
>
> - VK7 (Tiered storage): Accepted for the initial release; the follow-up
> plan is reasonable.
>
> - VK1 (Thundering herd / OOM): Resolved. Bounding peak memory
> by num_fetcher_threads * response_max_bytes, rather than partitions *
> 1MB, ensures predictable resource consumption.
>
> - VK3 (Control-plane hotspot): Resolved. Distributing
> __mirror_state transitions via the (mirrorName, topicId, partition) hash
> mirrors the proven scalability model of __consumer_offsets.
>
> - VK4 / VK8 (PID rewriting): Resolved, and superior to my original
> proposal. Dropping PID rewriting via the MIRROR_PID_RESET control record
> completely bypasses the hasProducerId() issue (where
> AbstractRecordBatch.hasProducerId() evaluates NO_PRODUCER_ID <
> producerId(), meaning a negative PID would have bypassed
> ProducerStateManager and incorrectly defaulted the LSO to the high
> watermark).
>
> - VK11 (TransactionIndex): Resolved. Confirmed the index is rebuilt
> locally on append rather than copied over the wire.
>
> - VK9 (CRC / zero-copy): Moot once PID rewriting was removed.
>
> Thank you for working through these eight architectural items.
>
>
> *STILL OPEN: VK12 (Offset-sync read-side data loss)*
> Luke noted on June 2 that the proposed formula makes sense, which I
> appreciate. However, the current Group Offsets text only implements the
> active-group guard ("skip groups that are active on the destination in any
> state other than EMPTY or DEAD"). While this guard is necessary, it is
> insufficient to prevent data loss for INACTIVE or dead groups during
> replication lag.
>
> Verifying this against the core coordinator implementation reveals a
> critical vulnerability when operators follow the KIP's guidance to
> set auto.offset.reset=latest:
>
> 1. The synced offset for an inactive group can legitimately exceed the
> destination log end offset (LEO) due to replication lag. The
> OffsetCommit path in the group coordinator does not validate a committed
> offset against the partition LEO, meaning an out-of-range offset is
> persisted as-is.
>
> 2. Upon failover, a resuming consumer reading from this offset receives an
> OFFSET_OUT_OF_RANGE exception.
>
> 3. Under the auto.offset.reset=latest strategy, the reset resolves via
> AutoOffsetResetStrategy.timestamp() to
> ListOffsetsRequest.LATEST_TIMESTAMP, which returns the current destination
> LEO.
>
> 4. Consequently, the consumer skips every record that arrived between
> the stale committed position and the reset point. This results in
> silent, unrecoverable read-side data loss precisely during a cluster
> failover the exact scenario this KIP is designed to protect against.
>
> Because the affected group is inactive on the destination, the
> active-group guard offers no protection here. The only way to guarantee
> data safety is to enforce a structural invariant at sync time using a
> double-clamp:
>
> syncedOffset = max(destinationLogStartOffset, min(sourceCommitted,
> destinationLEO))
>
> - The min(..., destinationLEO) upper bound eliminates the
> OFFSET_OUT_OF_RANGE exception window entirely, preventing forced resets and
> data skipping.
> - The max(destinationLogStartOffset, ...) lower bound prevents
> group rewinding if an offset falls below the destination retention log
> start offset.
>
> Under this invariant, the worst-case scenario is bounded
> re-processing proportional to the replication lag at failover, which aligns
> with Kafka's at-least-once delivery contract. This is strictly safer than
> endorsing a configuration path that silently drops data.
>
> Can we update the Group Offsets section to explicitly define this
> double-clamp as a coordinator-enforced invariant alongside the active-group
> guard?
>
>
> *STILL OPEN: VK10 (Group-name namespace collision / JR18)*
> The current text specifies that an offset commit fails
> with GroupIdNotFoundException when a consumer group and a share group share
> a name.
> As confirmed in the coordinator logic, share groups and consumer groups
> occupy a single namespace, and cross-type access throws this exception.
> This introduces two issues during operational failovers:
>
> 1. The error semantics are incorrect. The group ID does exist; it is
> simply the wrong type. Throwing "not found" misleads operators under fire.
> I agree with Jun's point in JR18 that an exception like
> GroupIdInUseException is required.
>
> 2. Discovering name collisions during an emergency failover introduces
> extreme operational risk. Introducing a pre-migration validation check to
> the CLI (e.g., kafka-cluster-mirrors.sh) to scan and flag source group
> and share-group collisions before mirroring begins would move this risk
> from production runtime to set up-time.
>
> Can we commit to (a) returning a precise error for cross-type
> collisions,and (b) adding a pre-flight validation check to the CLI utility?
>
> Once the double-clamp invariant for VK12 is integrated into the design text
> and we have a clear resolution for VK10, I have no further blockers and
> will gladly support the vote.
>
> Regards,
> Vaquar Khan
> https://www.linkedin.com/in/vaquar-khan-b695577/
>
> On Sun, 12 Jul 2026 at 12:54, Jun Rao via dev <de...@kafka.apache.org>
> wrote:
>
> > Hi, Federico,
> >
> > Thanks for the updated KIP. A few more comments.
> >
> > JR20. "Groups that are active on the destination cluster (i.e., in any
> > state other than EMPTY or DEAD) are skipped to avoid regressing offsets
> > after failover."
> > Does MirrorMetadataManager need to send an RPC to the group coordinator
> to
> > know the group's state?
> >
> > JR21. Are mirrored records validated against
> > log.message.timestamp.before.max.ms and
> log.message.timestamp.after.max.ms
> > on the destination cluster?
> >
> > JR22. "Leader Epoch Bump (LEB): The leader epoch in the destination
> cluster
> > remains unchanged during replication. That means it is possible that the
> > fetched batch carries a leader epoch that does not match with the local
> > leader epoch."
> > Is that correct? In the Leader Epoch Invariant section, there is logic to
> > makes sure SLE > DLE can't happen.
> >
> > JR23. "A new dynamic topic level configuration
> > mirror.support.unclean.leader.election (boolean, default false) controls
> > whether LME truncation waits for all assigned replicas to converge, not
> > just current ISR members. When enabled, the leader will not complete
> > truncation until every replica has joined the ISR and truncated past the
> > LME offset. This prevents subsequent unclean leader elections from
> > introducing undetectable log divergence between clusters."
> > Is this to prevent issues with ULE on the source or destination?
> >
> > JR24. "KIP-1327 introduces immutable creation timestamps on both
> partitions
> > and consumer groups to establish a happen-before relationship that
> > determines which offset reset policy applies when a consumer group
> > encounters a partition with no committed offset."
> > This needs adjustment since we decided not to pursue KIP-1327.
> >
> > JR25. IncludePatterns/ExcludePatterns: Which one takes precedence if both
> > are specified?
> >
> > JR26. For each record and RPC involving a state field, could you add a
> > description of the all possible states?
> >
> > JR27. WriteMirrorStatesRequest
> > { "name": "", "type": "int32", "versions": "0+", "default": "-1",
> > "about": "The last mirror leader epoch, or -1 if not
> available."
> > },
> > The field name is empty.
> >
> > JR28. Could you describe how the StateValidationOffset field is being
> used?
> >
> > JR29. "The ReplicaManager which provides source offset, destination
> offset,
> > and lag for each partition."
> > Which RPC is used to query the ReplicaManager?
> >
> > JR30. Failover: The original destination b stores LME under the mirror
> name
> > a-to-b. During fallback, the original source a creates a new mirror name
> > b-to-a and uses it in DescribeClusterMirrors to obtain LME. How can the
> > original destination cluster b find the right LME if the stored and
> > requested mirror names don't match?
> >
> > JR31. Start Mirror Topics:
> > JR31.1 "Controller persists include/exclude patterns as ConfigRecord
> > entries on the MIRROR resource in the metadata log."
> > Should we persist topics too? It's possible that the source topic doesn't
> > exist yet when a mirror is started.
> > JR31.2 "For each topic, the controller creates it on the destination if
> it
> > does not already exist, updates include/exclude topic pattern configs"
> > What are include/exclude topic pattern configs? In the previous step, the
> > controller already persists include/exclude patterns as ConfigRecord
> > entries on the MIRROR resource in the metadata log.
> > JR31.3 "Brokers receive the metadata update. The MirrorMetadataManager
> > detects the new mirrorName and DesiredState from the TopicImage and
> queries
> > the coordinator for the current partition state.
> > Partitions transition from UNKNOWN to LOG_TRUNCATION. During this state,
> > LME truncation runs and waits for all ISR members (or all replicas if ULE
> > is enabled)."
> > The MirrorMetadataManager instance responsible for a mirror is not
> > necessarily the same broker that leads a mirror partition. How can the
> > MirrorMetadataManager instance perform truncation? Ditto for Stop Mirror
> > Topics.
> >
> > JR32. Stop Mirror Topics: "For each topic, the controller writes a
> > MirrorTopicStateChangeRecord with DesiredState=1 (stop)."
> > Is this record kept forever in the metadata snapshot?
> >
> > JR33. Delete Mirror
> > JR33.1 he controller tombstones => The controller tombstones
> > JR34. "The mirror state records in __mirror_state internal topic are also
> > tombstoned."
> > Could you describe how MirrorTopicStateChangeRecord is tombstoned since
> it
> > doesn't have a key?
> >
> > JR34. AdminClient:
> > JR34.1 Could you define the public facing classes such as
> > CreateClusterMirrorResult, StartMirrorTopicsResult,
> > PauseMirrorTopicsOptions, etc?
> > JR34.2 describeClusterMirrors: How does it populate the clusterId field
> in
> > DescribeClusterMirrorsRequest?
> >
> > JR35. CreateTopic : "The replicas for the partition assignment are all
> > active and not in fenced or controlled shutdown. This is to make sure
> when
> > a topic gets deleted and re-created with the same topic ID, the stale
> > offline log dir won’t be treated as the active log dir after it becomes
> > online"
> > Could you explain why this is only needed for mirror topics, but not
> > regular topics?
> >
> > JR36. Fetch: Could you explain why we need to add the MirrorLeaderEpoch
> > field? I thought the destination cluster just issues a regular fetch
> > request like a consumer.
> >
> > JR37. CreateClusterMirror: "The broker validates that the mirror name is
> > not already in use, contains only permitted characters, and does not end
> > with .stopped or .paused suffix."
> > Could you explain why the suffix restriction is necessary?
> >
> > JR38. MirrorMetadataManager: Could you explain how mirrors are assigned
> to
> > MirrorMetadataManager instances? Is it based on the _mrror_state topic
> > partitions?
> >
> > JR39. StopMirrorTopics: Do we need TopicId? How does the AdminClient get
> > it?
> >
> > JR40. ReadMirrorStatesResponse: Could you explain how PreviousState is
> > being used?
> >
> > JR41. PartitionChangeRecord.MinLeaderEpoch: Currently, leader epochs are
> > implicitly derived from PartitionChangeRecord. By adding this field, the
> > controller will sometimes derive the epoch and sometimes use the
> explicitly
> > stored epoch, which can be confusing. Would it be better to always store
> > the leader epoch explicitly in PartitionChangeRecord?
> >
> > JR42. MirrorPartitionStateValue: Why does it use topicName, not topicId?
> >
> > JR43. Mirror config: What about authentication (sasl, ssl, etc) related
> > configs?
> >
> > JR44. Could you also describe how a partition leader picks up the
> mirroring
> > flow when it changes?
> >
> > JR45. Broker config: sasl.mechanism.mirror.admin.protocol
> > Is that for mirror-related RPCs between brokers and controllers?
> >
> > JR46. Compatibility
> > Since this KIP changes the metadata record format, we need an MV bump.
> For
> > mirror-related records and internal RPCs, should we introduce a
> > mirror-specific feature like GV/TV to gate them?
> >
> > Jun
> >
> >
> > On Tue, Jul 7, 2026 at 6:40 AM Federico Valeri <fe...@gmail.com>
> > wrote:
> >
> > > Hi Michael, thanks for your review.
> > >
> > > MM1: It was a bit confusing but the coordinator is the scheduler and
> > > the manager is the executor. That said, given that the manager already
> > > has a scheduler, we moved that logic there and updated the KIP.
> > >
> > > MM2: Good point. A deleted source topic would now cause the mirror
> > > partitions on the destination cluster to move to FAILED state. You can
> > > easily identify them using the describe command with the --failed
> > > flag.
> > >
> > > MM3: We have a log message at warn level. I think this is enough.
> > >
> > > MM4: Correct. KIP updated.
> > >
> > > MM5: Correct. KIP updated.
> > >
> > > MM6: The isolation level is hardcoded to READ_UNCOMMITTED. Not sure
> > > about the actual use case, but we can certainly make it configurable
> > > in a follow-up KIP.
> > >
> > > MM7: For quota related topic configurations,
> > > follower.replication.throttled.replicas and
> > > leader.replication.throttled.replicas are excluded by default in MM2
> > > and we apply the same logic. mirror.support.unclean.leader.election
> > > and mirror.replication.throttled.replicas are included by default
> > > because they are mirror related configurations.
> > >
> > > MM8: Updated the field description to: Topic patterns to remove from
> > > mirror.topics.include or add to mirror.topics.exclude.
> > >
> > > MM9: Not needed. KIP updated.
> > >
> > > MM10: The config is needed because the listener can have multiple
> > > enabled SASL mechanisms and the listener name alone doesn't select
> > > one. The naming follows the existing
> > > sasl.mechanism.inter.broker.protocol convention, so renaming would be
> > > inconsistent with that precedent.
> > >
> > > MM11: Correct. KIP updated.
> > >
> > > MM12: No, when mirror.support.unclean.leader.election=true, we’ll wait
> > > for all replicas, not just ISR replicas to complete truncation.
> > >
> > > MM13: Correct. KIP updated.
> > >
> > > MM14: We missed the “MIRROR_TOPIC_NOT_STOPPED”, so now: start on non
> > > stopped throws MIRROR_TOPIC_NOT_STOPPED. Stop on paused throws
> > > MIRROR_TOPIC_ALREADY_PAUSED. Pause on stopped throws
> > > MIRROR_TOPIC_BEING_STOPPED. Resume on non paused throws
> > > MIRROR_TOPIC_NOT_PAUSED. Stop on stopped throws
> > > TOPIC_NOT_IN_CLUSTER_MIRROR.
> > >
> > >
> > >
> > > On Fri, Jul 3, 2026 at 5:26 PM Mickael Maison <
> mickael.maison@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > Thanks for the KIP!
> > > > This will definitively be a great addition to Kafka.
> > > >
> > > > MM1: In the description of the main components,
> > > > ClusterMirrorCoordinator has this responsibility:
> > > > "Metadata Refresh Scheduling: The coordinator schedules periodic
> > > > metadata refresh operations by invoking a metadata manager every 30
> > > > seconds by default. This ensures that configuration updates and group
> > > > offset commits in the source cluster are continuously propagated to
> > > > the destination cluster. The refresh interval is configurable via
> > > > mirror.metadata.refresh.interval.ms."
> > > > Is that accurate because MirrorMetadataManager has pretty much the
> > > > same responsibility.
> > > >
> > > > MM2: About topic deletions, the KIP states: "the operator would need
> > > > to manually remove the topic from the mirror"
> > > > Is there a way to find topics that have been deleted in the source
> > > > cluster for an operator to do that action?
> > > >
> > > > MM3: About group offsets: It states: "Groups with active members on
> > > > the destination are skipped."
> > > > Should these be logged too?
> > > >
> > > > MM4: In the security control section, the DeleteClusterMirror RPC
> > > > requires Alter on the ClusterMirror resource. Should we use Delete
> > > > instead of Alter?
> > > > On non-cluster resources, we typically use Delete for deletion. See
> > > >
> > >
> >
> https://urldefense.com/v3/__https://kafka.apache.org/43/security/authorization-and-acls/*operations-and-resources-on-protocols__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqU3AKr-48$
> > > >
> > > > MM5: In Destination cluster permissions, I think FindCoordinator
> > > > should also need Describe on the Group resources for regular consumer
> > > > groups.
> > > >
> > > > MM6: In Exactly-Once Semantics, it states: "The mirror fetcher thread
> > > > uses READ_UNCOMMITTED isolation"
> > > > Could this be configurable? If so we can do this in a follow-up KIP
> > > >
> > > > MM7: It seems quotas are not mirrored between the clusters?
> > > > Is that omission intentional?
> > > >
> > > > MM8: In StopMirrorTopicsRequest I'm not sure I understand what the
> > > > Patterns field is?
> > > > Is it a pattern for the topics to stop? If so the description seems
> odd
> > > >
> > > > MM9: In
> > >
> >
> StopMirrorTopicsResponse/StartMirrorTopicsResponse/PauseMirrorTopicsResponse/ResumeMirrorTopicsResponse
> > > > do we really need the mirror name?
> > > >
> > > > MM10: In Configuration / Broker:
> > > > Can you clarify why we need sasl.mechanism.mirror.admin.protocol?
> > > > Can't we retrieve the protocol from mirror.admin.listener.name or
> > > > inter.broker.listener.name?
> > > > If we really need it, should it be named
> > > > mirror.admin.sasl.mechanism.protocol to match the other names?
> > > >
> > > > MM11: In Configuration / Mirror
> > > > Should it be mirror.acls.include instead of mirror.acl.include to
> > > > match mirror.groups.include and mirror.topics.include?
> > > >
> > > > MM12: Is the description of mirror.support.unclean.leader.election
> > > inverted?
> > > >
> > > > MM13: The description of CLUSTER_MIRROR_NOT_EMPTY states: "The
> cluster
> > > > mirror still has active or non-removed topics"
> > > > Should it be "The cluster mirror still has active or non-stopped
> > topics"?
> > > >
> > > > MM14: We have the MIRROR_TOPIC_BEING_STOPPED error when stopping
> topic
> > > > already in stopping state? What happens when pausing topics already
> in
> > > > pausing state?
> > > > Related, I assume when stopping a stopped topic, you get
> > > > TOPIC_NOT_IN_CLUSTER_MIRROR
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Mon, Jun 22, 2026 at 7:47 PM Andrew Schofield <
> > schofielaj@apache.org>
> > > wrote:
> > > > >
> > > > > Hi Fede,
> > > > > Thanks for your replies.
> > > > >
> > > > > AS30: Yes, that's right. Two different end markers for the same
> > > transaction, and essentially unbounded duration of the inconsistency
> > since
> > > KIP-939. I agree that it's a class of transaction state divergence,
> like
> > > KAFKA-20716. For KIP-1279, the most important thing is that consumers
> of
> > > the mirrored topic must not get confused by the markers. For example,
> > when
> > > the Fetch response contains the list of aborted transactions, provided
> we
> > > are sure that it can be built properly without confusion from the extra
> > > marker, then we are good. It came to mind because the way this is
> > > represented in the protocol for the responses is somewhat fragile I
> feel,
> > > so I'm a bit paranoid that a mis-alignment could break it.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > On 2026/06/19 12:16:52 Federico Valeri wrote:
> > > > > > Hi Andrew, me, again.
> > > > > >
> > > > > > Given that READ_COMMITTED consumers are unaffected because the
> > ABORT
> > > > > > is already recorded in the transaction index and the dangling
> > COMMIT
> > > > > > produces no CompletedTxn (no index and LSO updates), the
> practical
> > > > > > impact is limited. However, __transaction_state records the
> > > > > > transaction as COMMITTED while the partition log has it as
> aborted.
> > > > > > While analyzing this, we discovered that this type of divergence
> is
> > > > > > not unique to mirroring. KAFKA-20716 documents the same category
> of
> > > > > > inconsistency after unclean leader elections, where lost
> > transaction
> > > > > > markers cause the LSO to get stuck (much bigger problem). We
> > believe
> > > > > > that solving the transaction state divergence issue would also
> > close
> > > > > > this mirroring scenario. Wdyt?
> > > > > >
> > > > > > On Thu, Jun 18, 2026 at 10:36 PM Federico Valeri <
> > > fedevaleri@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Andrew, about AS30, I had a closer look at KIP-939, and I
> > think
> > > > > > > this is what you mean:
> > > > > > >
> > > > > > > The problem is that a WriteTxnMarker request for a PID unknown
> to
> > > the
> > > > > > > ProducerStateManager (cleared by MIRROR_PID_RESET) does not
> fail.
> > > The
> > > > > > > end transaction marker is still appended, but transaction index
> > and
> > > > > > > LSO are unaffected (ProducerStateManager.prepareUpdate,
> > > > > > > ProducerAppendInfo.appendEndTxnMarker). This may lead to a
> > > situation
> > > > > > > where we have two different end txn markers for the same
> > > transaction:
> > > > > > > ABORT from the stop mirroring operation, followed by a COMMIT
> > from
> > > the
> > > > > > > local Txn Coordinator retrying its WriteTxnMarker after the
> > > partition
> > > > > > > becomes writable again.
> > > > > > >
> > > > > > > READ_COMMITTED consumers still see the earlier ABORT in the
> > > > > > > transaction index and skip the aborted data. The visible
> outcome
> > > for
> > > > > > > consumers is correct. However, __transaction_state on the
> > original
> > > > > > > source cluster now records the transaction as COMMITTED while
> the
> > > > > > > partition log has it as aborted, creating a silent
> inconsistency
> > > that
> > > > > > > tools querying transaction state will not detect. The window
> for
> > > this
> > > > > > > race is bounded by transaction.max.timeout.ms (default 15
> > > minutes),
> > > > > > > but becomes unbounded with KIP-939 because it allows prepared
> > > > > > > transactions to sit indefinitely.
> > > > > > >
> > > > > > > We could move the coordinator state to “complete abort” at the
> > > cost of
> > > > > > > an RPC. This would prevent any retry, but that would break the
> > 2PC
> > > > > > > contract (Kafka must hold the prepared state until the external
> > > > > > > coordinator decides). Also, before stop there would be a number
> > of
> > > > > > > retries constantly going on. We definitely need a better
> > solution.
> > > > > > >
> > > > > > > On Thu, Jun 18, 2026 at 11:36 AM Federico Valeri <
> > > fedevaleri@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi Andrew, thanks for your careful review.
> > > > > > > >
> > > > > > > > AS28: We reworked that section and added some diagrams to
> > > illustrate
> > > > > > > > problem and solutions. Let us know if it is better.
> > > > > > > >
> > > > > > > > AS30: We updated that table, but we are not sure about the
> > > problem you
> > > > > > > > mention. Can you clarify and send us the sequence of actions
> > that
> > > > > > > > would lead to that problem?
> > > > > > > >
> > > > > > > > AS32: You are right. Replaced
> > > StartMirrorTopicsRequestData.TopicData
> > > > > > > > with a Controller.MirrorTopicMetadata record. Also renamed
> > > TopicData
> > > > > > > > to TopicMetadata in all schemas.
> > > > > > > >
> > > > > > > > AS33: Good idea. Added per-topic error messages in the RPC
> > > responses
> > > > > > > > you mentioned.
> > > > > > > >
> > > > > > > > AS29, AS31, AS34 (both of them), AS35, AS36: These small
> issues
> > > are
> > > > > > > > all fixed now.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 9, 2026 at 9:58 PM Andrew Schofield <
> > > schofielaj@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > Hi Fede and friends,
> > > > > > > > > This KIP is shaping up nicely. The change to the way that
> > > state management for the mirrors is done is much more elegant. The
> > removal
> > > of the PID mapping is also a nice improvement.
> > > > > > > > >
> > > > > > > > > Inevitably, I have some more comments.
> > > > > > > > >
> > > > > > > > > AS28: I understand KAFKA-18723 and how leader epochs work
> in
> > > that case, but the enhancements that you made to the Fetch RPC and how
> > they
> > > work escapes me. Please could we have some diagrams to illustrate the
> > > various epochs?
> > > > > > > > >
> > > > > > > > > AS29: I like the addition of share group offset syncing.
> You
> > > need AlterShareGroupOffsets and DescribeShareGroupOffsets in the tables
> > of
> > > RPCs and permissions.
> > > > > > > > >
> > > > > > > > > AS30: It is certainly true that EOS is not supported. Using
> > > cluster mirroring on transactional data does risk breaking
> transactional
> > > atomicity for transactions in flight at the time that mirroring was
> > > stopped. That's a known limitation, and it's an effect of per-topic
> async
> > > replication. Writing ABORT markers when stop mirroring is triggered is
> > > interesting. I just wonder whether it's truly safe.
> > > > > > > > >
> > > > > > > > > Could you enhance the tables of records in the Exactly-Once
> > > Semantics section to include leader epochs? I think that the leader
> epoch
> > > for the ABORT markers will have been bumped as the mirroring stopped
> and
> > > the partitions became writable. However, I worry that someone might
> > switch
> > > the mirroring back in the other direction before the transaction which
> > was
> > > rudely partially aborted has completed on the original source cluster,
> > and
> > > that the effect of the transaction coordinator completing the
> > transaction,
> > > potentially writing duplicate transaction markers after the ABORT
> markers
> > > which will have been replicated back again.
> > > > > > > > >
> > > > > > > > > This is probably only a realistic possibility when KIP-939
> is
> > > present, but it is accepted and I'm sure it will land soon enough. I
> > would
> > > say this KIP should be prepared for the existence of long-running
> > > transactions so we don't dig a hole to fall into later.
> > > > > > > > >
> > > > > > > > > AS31: In the section on the Admin Client, the description
> for
> > > Admin.createClusterMirror(String, Map, CreateClusterMirrorOptions) is
> > > duplicated.
> > > > > > > > >
> > > > > > > > > AS32: The use of StartMirrorTopicsRequestData.TopicData in
> an
> > > external interface looks odd.
> > > > > > > > >
> > > > > > > > > AS33: I suggest per-topic error messages in the RPC
> responses
> > > such as PauseMirrorTopics so you can pass back more context, such as
> the
> > > state if the request failed due to a state mismatch.
> > > > > > > > >
> > > > > > > > > AS34: There is an unnamed field in
> > > DescribeClusterMirrorsResponse. The same in LastMirrorEpochsValue.
> > > > > > > > >
> > > > > > > > > AS34: Some of the fields in BumpLeaderEpochsRequest start
> > with
> > > lower-case letters. Also, there are quite a few fields in the RPCs in
> > > general that should be version "0+" which are "0".
> > > > > > > > >
> > > > > > > > > AS35: Sometimes the partition states in the RPCs are
> string,
> > > but sometimes int8.
> > > > > > > > >
> > > > > > > > > AS36: The base level for share group support is probably
> 4.1.
> > > All of the RPCs were present in 4.1, although they were only enabled by
> > > default in 4.2.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Andrew
> > > > > > > > >
> > > > > > > > > On 2026/06/04 11:01:36 Federico Valeri wrote:
> > > > > > > > > > Hello, additional updates and replies:
> > > > > > > > > >
> > > > > > > > > > VK12: Consumer data loss risk is a good point. We updated
> > > the "Group
> > > > > > > > > > Offsets" paragraph.
> > > > > > > > > >
> > > > > > > > > > Some of you raised the point that using mirror.name
> > > suffixes for pause
> > > > > > > > > > and stop operations is a bit inelegant and we agreed. Now
> > we
> > > propose a
> > > > > > > > > > better approach:
> > > > > > > > > >
> > > > > > > > > > We replaced the topic config based approach (mirror.name
> > and
> > > > > > > > > > .stopped/.paused suffix conventions) with a first class
> > > metadata
> > > > > > > > > > record for tracking mirror topic state changes. A new
> > > > > > > > > > MirrorTopicStateChangeRecord is added to the metadata log
> > > with three
> > > > > > > > > > fields: TopicId (uuid), MirrorName (string), and
> > > DesiredState (int8,
> > > > > > > > > > where 0 = start mirroring, 1 = stop, 2 = pause). When a
> > user
> > > calls
> > > > > > > > > > startMirrorTopics, stopMirrorTopics, or
> pauseMirrorTopics,
> > > the
> > > > > > > > > > controller writes this record to the metadata log instead
> > of
> > > > > > > > > > manipulating config suffixes. Brokers receive the record
> > via
> > > metadata
> > > > > > > > > > updates and react accordingly, triggering partition level
> > > state
> > > > > > > > > > transitions through the existing MirrorPartitionState
> state
> > > machine.
> > > > > > > > > > More details in the updated KIP.
> > > > > > > > > >
> > > > > > > > > > Let us know what you think.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Fede
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 2, 2026 at 8:50 AM Luke Chen <
> > showuon@gmail.com>
> > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Viquar,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the comment.
> > > > > > > > > > >
> > > > > > > > > > > Regarding VK12, thanks for raising the issue.
> > > > > > > > > > > We didn't think about that.
> > > > > > > > > > > I agree that compared with data re-processing, it's
> worse
> > > to have data loss.
> > > > > > > > > > > The proposed formula makes sense to me.
> > > > > > > > > > > *syncedOffset = max(destinationLogStartOffset,
> > > min(sourceCommitted,
> > > > > > > > > > > destinationLEO))*
> > > > > > > > > > >
> > > > > > > > > > > Let us have some discussion internally and then update
> > the
> > > KIP and reply to
> > > > > > > > > > > the thread.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Luke
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 20, 2026 at 2:39 AM Federico Valeri <
> > > fedevaleri@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Rajini,
> > > > > > > > > > > >
> > > > > > > > > > > > RS14: Done. Missed that, sorry.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks again.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 19, 2026 at 7:28 PM Rajini Sivaram <
> > > rajinisivaram@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Federico,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the updates. Just one minor point, apart
> > > from that, looks
> > > > > > > > > > > > good.
> > > > > > > > > > > > >
> > > > > > > > > > > > > RS14: KIP still shows "bin/kafka-configs.sh
> > > --bootstrap-server :9094
> > > > > > > > > > > > > --entity-type mirrors".
> > > > > > > > > > > > > Will be good to change that to `--entity-type
> > > cluster-mirrors`.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rajini
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 19, 2026 at 5:28 PM vaquar khan <
> > > vaquar.khan@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Federico and team,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for your detailed response on May 11,
> > > 2026. I greatly
> > > > > > > > > > > > appreciate
> > > > > > > > > > > > > > the collaborative effort over the past few months
> > to
> > > harden the
> > > > > > > > > > > > KIP-1279
> > > > > > > > > > > > > > architecture.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > * 1. Resolved Architectural & Stability
> > > Vulnerabilities*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am pleased to confirm that the KIP has
> > > successfully integrated fixes
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the critical vulnerabilities I flagged, thereby
> > > protecting the
> > > > > > > > > > > > cluster's
> > > > > > > > > > > > > > state machine, memory bounds, and control plane.
> > > Consider the following
> > > > > > > > > > > > > > items fully resolved on my end:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *VK4 / VK8 (Negative PID Bug & State Machine
> > > Failure):* I previously
> > > > > > > > > > > > > > identified that the proposed
> -(sourceProducerId
> > +
> > > 2) PID rewriting
> > > > > > > > > > > > > > formula would fundamentally break
> > hasProducerId()
> > > in
> > > > > > > > > > > > > > AbstractRecordBatch.java, causing the broker
> to
> > > bypass
> > > > > > > > > > > > > > ProducerStateManager.update() and default the
> > > Last Stable Offset
> > > > > > > > > > > > (LSO)
> > > > > > > > > > > > > > to the High Watermark. I am glad to see the
> KIP
> > > authors abandoned
> > > > > > > > > > > > PID
> > > > > > > > > > > > > > mapping entirely and adopted the deterministic
> > > MIRROR_PID_RESET
> > > > > > > > > > > > control
> > > > > > > > > > > > > > record barrier. This perfectly protects
> Kafka's
> > > exactly-once
> > > > > > > > > > > > semantics.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *VK11 (TransactionIndex Rebuild Ambiguity):*
> > > Thank you for
> > > > > > > > > > > > officially
> > > > > > > > > > > > > > confirming that the TransactionIndex is
> strictly
> > > rebuilt locally
> > > > > > > > > > > > during
> > > > > > > > > > > > > > log appends rather than copied byte-for-byte.
> > > This resolves my
> > > > > > > > > > > > concern
> > > > > > > > > > > > > > regarding PID mismatches inducing consumer
> read
> > > anomalies.
> > > > > > > > > > > > > > -
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *VK1 (Thundering Herd / OOM Heap Allocation):*
> > > Your clarification
> > > > > > > > > > > > that
> > > > > > > > > > > > > > fetcher threads multiplex partitions meaning
> the
> > > memory footprint is
> > > > > > > > > > > > > > strictly bounded by num_fetcher_threads *
> > > response_max_bytes rather
> > > > > > > > > > > > than
> > > > > > > > > > > > > > a concurrent 1MB buffer per partition fully
> > > resolves my concern
> > > > > > > > > > > > > > regarding
> > > > > > > > > > > > > > 50GB broker-wide OOM spikes during mass
> > partition
> > > wake-ups.
> > > > > > > > > > > > > > -
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *VK3 (Control Plane Hotspots):* I had flagged
> > the
> > > severe risk of
> > > > > > > > > > > > > > metadata saturation on a single broker during
> a
> > > "link flap" event.
> > > > > > > > > > > > Your
> > > > > > > > > > > > > > confirmation that the __mirror_state topic
> > > utilizes a compound hash
> > > > > > > > > > > > > > of (mirrorName,
> > > > > > > > > > > > > > topicId, partition number) mathematically
> > ensures
> > > that the 50,000+
> > > > > > > > > > > > state
> > > > > > > > > > > > > > transitions will be safely distributed across
> > the
> > > cluster,
> > > > > > > > > > > > neutralizing
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > single-node hotspot risk.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. Outstanding Critical Blocker: VK12 (Offset
> Sync
> > > Data Loss)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding VK12, there is a fundamental
> > > misunderstanding in your
> > > > > > > > > > > > previous
> > > > > > > > > > > > > > reply. You stated: *"The scenario described can
> > only
> > > occur if offsets
> > > > > > > > > > > > are
> > > > > > > > > > > > > > force-written to an active group, which the
> design
> > > prevents."*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > My concern has absolutely nothing to do with
> > > overwriting active
> > > > > > > > > > > > groups. My
> > > > > > > > > > > > > > concern applies strictly to the normal
> > > synchronization of inactive/dead
> > > > > > > > > > > > > > groups, and is based directly on the race
> condition
> > > currently
> > > > > > > > > > > > documented in
> > > > > > > > > > > > > > the official KIP-1279 text:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *"During offset synchronization, the committed
> > > offset in the
> > > > > > > > > > > > destination
> > > > > > > > > > > > > > cluster may temporarily exceed the current log
> end
> > > offset (LEO) of the
> > > > > > > > > > > > > > mirror topic... consumers attempting to resume
> from
> > > offset 100 will
> > > > > > > > > > > > receive
> > > > > > > > > > > > > > an OffsetOutOfRangeException. To handle this
> > > gracefully, consumers
> > > > > > > > > > > > should
> > > > > > > > > > > > > > configure auto.offset.reset=latest..."*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If a failover happens during this documented
> > > divergence window, a
> > > > > > > > > > > > > > reconnecting consumer will hit the
> > > OffsetOutOfRangeException. If the
> > > > > > > > > > > > > > downstream consumer follows the KIP's official
> > > advice and relies on
> > > > > > > > > > > > > > auto.offset.reset=latest, the consumer will jump
> to
> > > the absolute newest
> > > > > > > > > > > > > > offset on the partition. *This completely skips
> any
> > > newly produced
> > > > > > > > > > > > records
> > > > > > > > > > > > > > that arrived between the failover and the
> consumer
> > > reconnecting,
> > > > > > > > > > > > resulting
> > > > > > > > > > > > > > in silent, unrecoverable data loss for the
> > > downstream application.*
> > > > > > > > > > > > > > *Proposed Resolution: Double-Clamped Offset
> Safety
> > > Invariant* Instead
> > > > > > > > > > > > of
> > > > > > > > > > > > > > requiring consumers to use
> auto.offset.reset=latest
> > > and endorsing a
> > > > > > > > > > > > known
> > > > > > > > > > > > > > data loss vector, I propose that the
> > > ClusterMirrorCoordinator enforce a
> > > > > > > > > > > > > > pre-persist offset validation invariant during
> > > offset synchronization.
> > > > > > > > > > > > > > Before any translated offset is committed to the
> > > destination cluster,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > coordinator must apply a double-clamp:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *syncedOffset = max(destinationLSO,
> > > min(sourceCommitted,
> > > > > > > > > > > > destinationLEO))*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Where destinationLSO is the Log Start Offset
> > > (earliest readable
> > > > > > > > > > > > position
> > > > > > > > > > > > > > post-retention) and destinationLEO is the Log End
> > > Offset (latest
> > > > > > > > > > > > replicated
> > > > > > > > > > > > > > position) of the destination partition.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This guarantees that every persisted offset falls
> > > within the physically
> > > > > > > > > > > > > > valid range eliminating both the
> > > OffsetOutOfRangeException caused by
> > > > > > > > > > > > > > exceeding the LEO during replication lag, and the
> > > expired-offset rewind
> > > > > > > > > > > > > > caused by falling below the LSO due to
> destination
> > > retention policies.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The worst-case trade-off under this invariant is
> > > bounded re-processing
> > > > > > > > > > > > > > proportional to the replication lag at failover
> > time
> > > not total
> > > > > > > > > > > > partition
> > > > > > > > > > > > > > depth which is perfectly consistent with Kafka's
> > > documented
> > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > delivery guarantees. Broad enterprise production
> > > evidence from
> > > > > > > > > > > > large-scale
> > > > > > > > > > > > > > cross-cluster failovers confirms that state
> checks
> > > alone (preventing
> > > > > > > > > > > > active
> > > > > > > > > > > > > > group overwrites) are insufficient; strict offset
> > > bounds clamping is
> > > > > > > > > > > > > > required to achieve enterprise-grade data
> > integrity.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I look forward to your thoughts on implementing
> > this
> > > final offset
> > > > > > > > > > > > capping
> > > > > > > > > > > > > > logic. Once VK12 is patched, I believe this
> > > architecture will be
> > > > > > > > > > > > > > exceptionally robust and ready for enterprise
> > > deployment.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Viquar Khan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, 18 May 2026 at 14:23, Rajini Sivaram <
> > > rajinisivaram@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Federico,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the updates! The KIP is looking
> good.
> > A
> > > few more small
> > > > > > > > > > > > > > comments.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > RS13: A couple of places still refer to
> > > `kafka-mirror.sh` like under
> > > > > > > > > > > > > > > `Failover Process`. Can we change them to
> > > > > > > > > > > > `*kafka-cluster-mirrors.sh*`?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > RS14: Should we change `--entity-type mirrors`
> > for
> > > kafka-configs to
> > > > > > > > > > > > be `
> > > > > > > > > > > > > > > --entity-type *cluster-mirrors*` to be
> > consistent?
> > > Also,
> > > > > > > > > > > > > > > CLUSTER_MIRROR((byte)
> > > > > > > > > > > > > > > 64, "mirror"); could be `*cluster-mirror*`?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > RS15: It may be useful to rename
> > > `mirror.topic.num.partitions` and `
> > > > > > > > > > > > > > > mirror.topic.replication.factor` since they are
> > > very similar to `
> > > > > > > > > > > > > > > mirror.topic.properties.exclude`, but the
> > > `mirror.topic` prefix
> > > > > > > > > > > > refers to
> > > > > > > > > > > > > > > different topics (the internal topic for the
> > first
> > > two and actual
> > > > > > > > > > > > mirror
> > > > > > > > > > > > > > > topics for the other one).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > RS16: ACL Sync: KIP says "Deletes ACLs that
> exist
> > > in destination but
> > > > > > > > > > > > not
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > source using DeleteAcls request."
> > > > > > > > > > > > > > > What happens if someone creates an ACL on the
> > > destination to deny
> > > > > > > > > > > > > > > User:Alice access to all topics?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. If that ACL also existed on the source
> > > cluster and then it was
> > > > > > > > > > > > > > > removed, will it get removed from the
> > > destination?
> > > > > > > > > > > > > > > 2. If that ACL never existed on the source
> > > cluster, will it get
> > > > > > > > > > > > > > removed
> > > > > > > > > > > > > > > from the destination?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > RS17: The table in the Source ACLs section
> says:
> > > > > > > > > > > > "DescribeClusterMirrors
> > > > > > > > > > > > > > > MC Read Cluster Log truncation"
> > > > > > > > > > > > > > > Should that be `ClusterMirror:Read` instead of
> > > `Cluster:Read`?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Rajini
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, May 15, 2026 at 8:41 AM Federico
> Valeri <
> > > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Rajini, we finally addressed the API and
> > tool
> > > naming
> > > > > > > > > > > > refactoring as
> > > > > > > > > > > > > > > > you suggested in RS6. Please take a look when
> > > you have time.
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:17 PM Federico
> > Valeri <
> > > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hello all, I want to highlight a couple of
> > new
> > > paragraphs:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Leader Epoch Invariant: Cluster
> mirroring
> > > enforces the
> > > > > > > > > > > > invariant
> > > > > > > > > > > > > > > > > that the destination leader epoch must
> always
> > > be greater than or
> > > > > > > > > > > > > > equal
> > > > > > > > > > > > > > > > > to the source leader epoch (DLE>=SLE).
> > Without
> > > this, consumers
> > > > > > > > > > > > on the
> > > > > > > > > > > > > > > > > destination cluster can get stuck in an
> > > infinite metadata refresh
> > > > > > > > > > > > > > loop
> > > > > > > > > > > > > > > > > when they encounter committed offsets
> > carrying
> > > source epochs
> > > > > > > > > > > > higher
> > > > > > > > > > > > > > > > > than the local epoch. The invariant is
> > > maintained through three
> > > > > > > > > > > > > > > > > mechanisms: reactive bumping (epoch fencing
> > > triggered when SLE >
> > > > > > > > > > > > DLE
> > > > > > > > > > > > > > > > > during fetch), proactive bumping (scheduled
> > > when SLE approaches
> > > > > > > > > > > > DLE
> > > > > > > > > > > > > > > > > within a threshold), and periodic bumping
> > > (checked during
> > > > > > > > > > > > coordinator
> > > > > > > > > > > > > > > > > metadata sync).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973*KIP1279:ClusterMirroring-LeaderEpochInvariant__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqUcl_uWkU$
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. Group Offsets: The coordinator
> > periodically
> > > syncs consumer and
> > > > > > > > > > > > > > > > > share group offsets from the source cluster
> > to
> > > the destination
> > > > > > > > > > > > for
> > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > mirrored topics. Groups are filtered by
> > > configurable
> > > > > > > > > > > > include/exclude
> > > > > > > > > > > > > > > > > patterns, and offsets are only synced for
> > > groups that are not
> > > > > > > > > > > > > > > > > currently active on the destination
> cluster,
> > > preventing
> > > > > > > > > > > > overwrites of
> > > > > > > > > > > > > > > > > local consumer progress. Because source and
> > > destination share the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > topic offsets (no offset translation),
> synced
> > > offsets can be used
> > > > > > > > > > > > > > > > > directly without mapping.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973*KIP1279:ClusterMirroring-GroupOffsets__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqU-JubnOA$
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > These new paragraphs directly address some
> of
> > > your questions,
> > > > > > > > > > > > but let
> > > > > > > > > > > > > > > > > me list them here:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > JR2: Yes, we removed the incorrect phrase
> and
> > > added more details
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the paragraph.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > JR4: When source cluster topic has tiered
> > > storage enabled, CM
> > > > > > > > > > > > works
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > mirroring remote and local log into
> > > destination cluster. When
> > > > > > > > > > > > > > > > > destination cluster topic has tiered
> storage
> > > enabled, CM fails in
> > > > > > > > > > > > > > > > > PREPARING state because the LME may be in
> > > remote storage, but
> > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > fine if already MIRRORING because no
> > > truncation is needed.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > JR11: See "Leader Epoch Invariant"
> paragraph
> > > mentioned above.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > JR13: We can't support stateful Streams
> > > application because
> > > > > > > > > > > > > > > > > asynchronous replication cannot preserve
> the
> > > transactional
> > > > > > > > > > > > boundaries
> > > > > > > > > > > > > > > > > between input offset commits, state store
> > > mutations written to
> > > > > > > > > > > > > > > > > changelog topics, and intermediate records
> > > written to repartition
> > > > > > > > > > > > > > > > > topics. The synchronous extension of this
> > > design will be able to
> > > > > > > > > > > > > > > > > support them. Existing Features Integration
> > > paragraph updated.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > JR18: See "Group Offsets" paragraph
> mentioned
> > > above.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > IY1: See "Group Offsets" paragraph
> mentioned
> > > above.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:08 PM Federico
> > > Valeri <
> > > > > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Vaquar,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > VK4/VK8: We don't do PID mapping anymore.
> > > The KIP was updated
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > time ago with the new approach based on
> the
> > > new PID reset
> > > > > > > > > > > > control
> > > > > > > > > > > > > > > > > > record.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > VK11: The transaction index is always
> built
> > > locally during log
> > > > > > > > > > > > > > > append,
> > > > > > > > > > > > > > > > > > never copied.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > VK1: The 50,000 * 1MB = 50GB calculation
> > > misunderstands the
> > > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > > model. Fetcher threads don't allocate one
> > > buffer per partition.
> > > > > > > > > > > > > > > Actual
> > > > > > > > > > > > > > > > > > peak memory is roughly
> num_fetcher_threads
> > *
> > > > > > > > > > > > response_max_bytes,
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > num_partitions * partition_max_bytes.
> With
> > 1
> > > fetcher thread
> > > > > > > > > > > > and the
> > > > > > > > > > > > > > > > > > default response max, the memory
> footprint
> > > is modest
> > > > > > > > > > > > regardless of
> > > > > > > > > > > > > > > > > > partition count. We are leveraging the
> same
> > > proven pattern
> > > > > > > > > > > > used by
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > internal replication.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > VK3: The __mirror_state topic uses
> > > hash-based partitioning
> > > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > > mirrorName, topicId and partition number.
> > > With the production
> > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > of 50 partitions, 50,000 partition
> > > transitions distribute
> > > > > > > > > > > > across
> > > > > > > > > > > > > > ~50
> > > > > > > > > > > > > > > > > > partition leaders on different brokers,
> not
> > > a single broker.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > the same proven pattern as
> > > __consumer_offsets, which handles
> > > > > > > > > > > > > > millions
> > > > > > > > > > > > > > > > > > of commits.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > VK12: The scenario described can only
> occur
> > > if offsets are
> > > > > > > > > > > > > > > > > > force-written to an active group, which
> the
> > > design prevents.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by vaquar khan <va...@gmail.com>.
Hi Federico, Luke, Omnia, and all,
Thanks for the continued work on this. The KIP has come a long way. I want
to close out the items we have settled so the record is clean, then address
the remaining data-safety item where the current text does not yet
implement the necessary architectural guard.
*RESOLVED / CLOSED ON MY END*
- VK2 (Poison pill blast radius): Resolved. Partition-level FAILED
isolation in the fetcher pool is the correct model.
- VK5 (Infinite loops): Resolved. Read-only enforcement on mirrored
topics makes A->B->A physically impossible.
- VK6 (Unclean-election divergence): Accepted as a documented
limitation, and the updated ULE sub-design thoroughly addresses my initial
concerns.
- VK7 (Tiered storage): Accepted for the initial release; the follow-up
plan is reasonable.
- VK1 (Thundering herd / OOM): Resolved. Bounding peak memory
by num_fetcher_threads * response_max_bytes, rather than partitions *
1MB, ensures predictable resource consumption.
- VK3 (Control-plane hotspot): Resolved. Distributing
__mirror_state transitions via the (mirrorName, topicId, partition) hash
mirrors the proven scalability model of __consumer_offsets.
- VK4 / VK8 (PID rewriting): Resolved, and superior to my original
proposal. Dropping PID rewriting via the MIRROR_PID_RESET control record
completely bypasses the hasProducerId() issue (where
AbstractRecordBatch.hasProducerId() evaluates NO_PRODUCER_ID <
producerId(), meaning a negative PID would have bypassed
ProducerStateManager and incorrectly defaulted the LSO to the high
watermark).
- VK11 (TransactionIndex): Resolved. Confirmed the index is rebuilt
locally on append rather than copied over the wire.
- VK9 (CRC / zero-copy): Moot once PID rewriting was removed.
Thank you for working through these eight architectural items.
*STILL OPEN: VK12 (Offset-sync read-side data loss)*
Luke noted on June 2 that the proposed formula makes sense, which I
appreciate. However, the current Group Offsets text only implements the
active-group guard ("skip groups that are active on the destination in any
state other than EMPTY or DEAD"). While this guard is necessary, it is
insufficient to prevent data loss for INACTIVE or dead groups during
replication lag.
Verifying this against the core coordinator implementation reveals a
critical vulnerability when operators follow the KIP's guidance to
set auto.offset.reset=latest:
1. The synced offset for an inactive group can legitimately exceed the
destination log end offset (LEO) due to replication lag. The
OffsetCommit path in the group coordinator does not validate a committed
offset against the partition LEO, meaning an out-of-range offset is
persisted as-is.
2. Upon failover, a resuming consumer reading from this offset receives an
OFFSET_OUT_OF_RANGE exception.
3. Under the auto.offset.reset=latest strategy, the reset resolves via
AutoOffsetResetStrategy.timestamp() to
ListOffsetsRequest.LATEST_TIMESTAMP, which returns the current destination
LEO.
4. Consequently, the consumer skips every record that arrived between
the stale committed position and the reset point. This results in
silent, unrecoverable read-side data loss precisely during a cluster
failover the exact scenario this KIP is designed to protect against.
Because the affected group is inactive on the destination, the
active-group guard offers no protection here. The only way to guarantee
data safety is to enforce a structural invariant at sync time using a
double-clamp:
syncedOffset = max(destinationLogStartOffset, min(sourceCommitted,
destinationLEO))
- The min(..., destinationLEO) upper bound eliminates the
OFFSET_OUT_OF_RANGE exception window entirely, preventing forced resets and
data skipping.
- The max(destinationLogStartOffset, ...) lower bound prevents
group rewinding if an offset falls below the destination retention log
start offset.
Under this invariant, the worst-case scenario is bounded
re-processing proportional to the replication lag at failover, which aligns
with Kafka's at-least-once delivery contract. This is strictly safer than
endorsing a configuration path that silently drops data.
Can we update the Group Offsets section to explicitly define this
double-clamp as a coordinator-enforced invariant alongside the active-group
guard?
*STILL OPEN: VK10 (Group-name namespace collision / JR18)*
The current text specifies that an offset commit fails
with GroupIdNotFoundException when a consumer group and a share group share
a name.
As confirmed in the coordinator logic, share groups and consumer groups
occupy a single namespace, and cross-type access throws this exception.
This introduces two issues during operational failovers:
1. The error semantics are incorrect. The group ID does exist; it is
simply the wrong type. Throwing "not found" misleads operators under fire.
I agree with Jun's point in JR18 that an exception like
GroupIdInUseException is required.
2. Discovering name collisions during an emergency failover introduces
extreme operational risk. Introducing a pre-migration validation check to
the CLI (e.g., kafka-cluster-mirrors.sh) to scan and flag source group
and share-group collisions before mirroring begins would move this risk
from production runtime to set up-time.
Can we commit to (a) returning a precise error for cross-type
collisions,and (b) adding a pre-flight validation check to the CLI utility?
Once the double-clamp invariant for VK12 is integrated into the design text
and we have a clear resolution for VK10, I have no further blockers and
will gladly support the vote.
Regards,
Vaquar Khan
https://www.linkedin.com/in/vaquar-khan-b695577/
On Sun, 12 Jul 2026 at 12:54, Jun Rao via dev <de...@kafka.apache.org> wrote:
> Hi, Federico,
>
> Thanks for the updated KIP. A few more comments.
>
> JR20. "Groups that are active on the destination cluster (i.e., in any
> state other than EMPTY or DEAD) are skipped to avoid regressing offsets
> after failover."
> Does MirrorMetadataManager need to send an RPC to the group coordinator to
> know the group's state?
>
> JR21. Are mirrored records validated against
> log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms
> on the destination cluster?
>
> JR22. "Leader Epoch Bump (LEB): The leader epoch in the destination cluster
> remains unchanged during replication. That means it is possible that the
> fetched batch carries a leader epoch that does not match with the local
> leader epoch."
> Is that correct? In the Leader Epoch Invariant section, there is logic to
> makes sure SLE > DLE can't happen.
>
> JR23. "A new dynamic topic level configuration
> mirror.support.unclean.leader.election (boolean, default false) controls
> whether LME truncation waits for all assigned replicas to converge, not
> just current ISR members. When enabled, the leader will not complete
> truncation until every replica has joined the ISR and truncated past the
> LME offset. This prevents subsequent unclean leader elections from
> introducing undetectable log divergence between clusters."
> Is this to prevent issues with ULE on the source or destination?
>
> JR24. "KIP-1327 introduces immutable creation timestamps on both partitions
> and consumer groups to establish a happen-before relationship that
> determines which offset reset policy applies when a consumer group
> encounters a partition with no committed offset."
> This needs adjustment since we decided not to pursue KIP-1327.
>
> JR25. IncludePatterns/ExcludePatterns: Which one takes precedence if both
> are specified?
>
> JR26. For each record and RPC involving a state field, could you add a
> description of the all possible states?
>
> JR27. WriteMirrorStatesRequest
> { "name": "", "type": "int32", "versions": "0+", "default": "-1",
> "about": "The last mirror leader epoch, or -1 if not available."
> },
> The field name is empty.
>
> JR28. Could you describe how the StateValidationOffset field is being used?
>
> JR29. "The ReplicaManager which provides source offset, destination offset,
> and lag for each partition."
> Which RPC is used to query the ReplicaManager?
>
> JR30. Failover: The original destination b stores LME under the mirror name
> a-to-b. During fallback, the original source a creates a new mirror name
> b-to-a and uses it in DescribeClusterMirrors to obtain LME. How can the
> original destination cluster b find the right LME if the stored and
> requested mirror names don't match?
>
> JR31. Start Mirror Topics:
> JR31.1 "Controller persists include/exclude patterns as ConfigRecord
> entries on the MIRROR resource in the metadata log."
> Should we persist topics too? It's possible that the source topic doesn't
> exist yet when a mirror is started.
> JR31.2 "For each topic, the controller creates it on the destination if it
> does not already exist, updates include/exclude topic pattern configs"
> What are include/exclude topic pattern configs? In the previous step, the
> controller already persists include/exclude patterns as ConfigRecord
> entries on the MIRROR resource in the metadata log.
> JR31.3 "Brokers receive the metadata update. The MirrorMetadataManager
> detects the new mirrorName and DesiredState from the TopicImage and queries
> the coordinator for the current partition state.
> Partitions transition from UNKNOWN to LOG_TRUNCATION. During this state,
> LME truncation runs and waits for all ISR members (or all replicas if ULE
> is enabled)."
> The MirrorMetadataManager instance responsible for a mirror is not
> necessarily the same broker that leads a mirror partition. How can the
> MirrorMetadataManager instance perform truncation? Ditto for Stop Mirror
> Topics.
>
> JR32. Stop Mirror Topics: "For each topic, the controller writes a
> MirrorTopicStateChangeRecord with DesiredState=1 (stop)."
> Is this record kept forever in the metadata snapshot?
>
> JR33. Delete Mirror
> JR33.1 he controller tombstones => The controller tombstones
> JR34. "The mirror state records in __mirror_state internal topic are also
> tombstoned."
> Could you describe how MirrorTopicStateChangeRecord is tombstoned since it
> doesn't have a key?
>
> JR34. AdminClient:
> JR34.1 Could you define the public facing classes such as
> CreateClusterMirrorResult, StartMirrorTopicsResult,
> PauseMirrorTopicsOptions, etc?
> JR34.2 describeClusterMirrors: How does it populate the clusterId field in
> DescribeClusterMirrorsRequest?
>
> JR35. CreateTopic : "The replicas for the partition assignment are all
> active and not in fenced or controlled shutdown. This is to make sure when
> a topic gets deleted and re-created with the same topic ID, the stale
> offline log dir won’t be treated as the active log dir after it becomes
> online"
> Could you explain why this is only needed for mirror topics, but not
> regular topics?
>
> JR36. Fetch: Could you explain why we need to add the MirrorLeaderEpoch
> field? I thought the destination cluster just issues a regular fetch
> request like a consumer.
>
> JR37. CreateClusterMirror: "The broker validates that the mirror name is
> not already in use, contains only permitted characters, and does not end
> with .stopped or .paused suffix."
> Could you explain why the suffix restriction is necessary?
>
> JR38. MirrorMetadataManager: Could you explain how mirrors are assigned to
> MirrorMetadataManager instances? Is it based on the _mrror_state topic
> partitions?
>
> JR39. StopMirrorTopics: Do we need TopicId? How does the AdminClient get
> it?
>
> JR40. ReadMirrorStatesResponse: Could you explain how PreviousState is
> being used?
>
> JR41. PartitionChangeRecord.MinLeaderEpoch: Currently, leader epochs are
> implicitly derived from PartitionChangeRecord. By adding this field, the
> controller will sometimes derive the epoch and sometimes use the explicitly
> stored epoch, which can be confusing. Would it be better to always store
> the leader epoch explicitly in PartitionChangeRecord?
>
> JR42. MirrorPartitionStateValue: Why does it use topicName, not topicId?
>
> JR43. Mirror config: What about authentication (sasl, ssl, etc) related
> configs?
>
> JR44. Could you also describe how a partition leader picks up the mirroring
> flow when it changes?
>
> JR45. Broker config: sasl.mechanism.mirror.admin.protocol
> Is that for mirror-related RPCs between brokers and controllers?
>
> JR46. Compatibility
> Since this KIP changes the metadata record format, we need an MV bump. For
> mirror-related records and internal RPCs, should we introduce a
> mirror-specific feature like GV/TV to gate them?
>
> Jun
>
>
> On Tue, Jul 7, 2026 at 6:40 AM Federico Valeri <fe...@gmail.com>
> wrote:
>
> > Hi Michael, thanks for your review.
> >
> > MM1: It was a bit confusing but the coordinator is the scheduler and
> > the manager is the executor. That said, given that the manager already
> > has a scheduler, we moved that logic there and updated the KIP.
> >
> > MM2: Good point. A deleted source topic would now cause the mirror
> > partitions on the destination cluster to move to FAILED state. You can
> > easily identify them using the describe command with the --failed
> > flag.
> >
> > MM3: We have a log message at warn level. I think this is enough.
> >
> > MM4: Correct. KIP updated.
> >
> > MM5: Correct. KIP updated.
> >
> > MM6: The isolation level is hardcoded to READ_UNCOMMITTED. Not sure
> > about the actual use case, but we can certainly make it configurable
> > in a follow-up KIP.
> >
> > MM7: For quota related topic configurations,
> > follower.replication.throttled.replicas and
> > leader.replication.throttled.replicas are excluded by default in MM2
> > and we apply the same logic. mirror.support.unclean.leader.election
> > and mirror.replication.throttled.replicas are included by default
> > because they are mirror related configurations.
> >
> > MM8: Updated the field description to: Topic patterns to remove from
> > mirror.topics.include or add to mirror.topics.exclude.
> >
> > MM9: Not needed. KIP updated.
> >
> > MM10: The config is needed because the listener can have multiple
> > enabled SASL mechanisms and the listener name alone doesn't select
> > one. The naming follows the existing
> > sasl.mechanism.inter.broker.protocol convention, so renaming would be
> > inconsistent with that precedent.
> >
> > MM11: Correct. KIP updated.
> >
> > MM12: No, when mirror.support.unclean.leader.election=true, we’ll wait
> > for all replicas, not just ISR replicas to complete truncation.
> >
> > MM13: Correct. KIP updated.
> >
> > MM14: We missed the “MIRROR_TOPIC_NOT_STOPPED”, so now: start on non
> > stopped throws MIRROR_TOPIC_NOT_STOPPED. Stop on paused throws
> > MIRROR_TOPIC_ALREADY_PAUSED. Pause on stopped throws
> > MIRROR_TOPIC_BEING_STOPPED. Resume on non paused throws
> > MIRROR_TOPIC_NOT_PAUSED. Stop on stopped throws
> > TOPIC_NOT_IN_CLUSTER_MIRROR.
> >
> >
> >
> > On Fri, Jul 3, 2026 at 5:26 PM Mickael Maison <mi...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > Thanks for the KIP!
> > > This will definitively be a great addition to Kafka.
> > >
> > > MM1: In the description of the main components,
> > > ClusterMirrorCoordinator has this responsibility:
> > > "Metadata Refresh Scheduling: The coordinator schedules periodic
> > > metadata refresh operations by invoking a metadata manager every 30
> > > seconds by default. This ensures that configuration updates and group
> > > offset commits in the source cluster are continuously propagated to
> > > the destination cluster. The refresh interval is configurable via
> > > mirror.metadata.refresh.interval.ms."
> > > Is that accurate because MirrorMetadataManager has pretty much the
> > > same responsibility.
> > >
> > > MM2: About topic deletions, the KIP states: "the operator would need
> > > to manually remove the topic from the mirror"
> > > Is there a way to find topics that have been deleted in the source
> > > cluster for an operator to do that action?
> > >
> > > MM3: About group offsets: It states: "Groups with active members on
> > > the destination are skipped."
> > > Should these be logged too?
> > >
> > > MM4: In the security control section, the DeleteClusterMirror RPC
> > > requires Alter on the ClusterMirror resource. Should we use Delete
> > > instead of Alter?
> > > On non-cluster resources, we typically use Delete for deletion. See
> > >
> >
> https://urldefense.com/v3/__https://kafka.apache.org/43/security/authorization-and-acls/*operations-and-resources-on-protocols__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqU3AKr-48$
> > >
> > > MM5: In Destination cluster permissions, I think FindCoordinator
> > > should also need Describe on the Group resources for regular consumer
> > > groups.
> > >
> > > MM6: In Exactly-Once Semantics, it states: "The mirror fetcher thread
> > > uses READ_UNCOMMITTED isolation"
> > > Could this be configurable? If so we can do this in a follow-up KIP
> > >
> > > MM7: It seems quotas are not mirrored between the clusters?
> > > Is that omission intentional?
> > >
> > > MM8: In StopMirrorTopicsRequest I'm not sure I understand what the
> > > Patterns field is?
> > > Is it a pattern for the topics to stop? If so the description seems odd
> > >
> > > MM9: In
> >
> StopMirrorTopicsResponse/StartMirrorTopicsResponse/PauseMirrorTopicsResponse/ResumeMirrorTopicsResponse
> > > do we really need the mirror name?
> > >
> > > MM10: In Configuration / Broker:
> > > Can you clarify why we need sasl.mechanism.mirror.admin.protocol?
> > > Can't we retrieve the protocol from mirror.admin.listener.name or
> > > inter.broker.listener.name?
> > > If we really need it, should it be named
> > > mirror.admin.sasl.mechanism.protocol to match the other names?
> > >
> > > MM11: In Configuration / Mirror
> > > Should it be mirror.acls.include instead of mirror.acl.include to
> > > match mirror.groups.include and mirror.topics.include?
> > >
> > > MM12: Is the description of mirror.support.unclean.leader.election
> > inverted?
> > >
> > > MM13: The description of CLUSTER_MIRROR_NOT_EMPTY states: "The cluster
> > > mirror still has active or non-removed topics"
> > > Should it be "The cluster mirror still has active or non-stopped
> topics"?
> > >
> > > MM14: We have the MIRROR_TOPIC_BEING_STOPPED error when stopping topic
> > > already in stopping state? What happens when pausing topics already in
> > > pausing state?
> > > Related, I assume when stopping a stopped topic, you get
> > > TOPIC_NOT_IN_CLUSTER_MIRROR
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Mon, Jun 22, 2026 at 7:47 PM Andrew Schofield <
> schofielaj@apache.org>
> > wrote:
> > > >
> > > > Hi Fede,
> > > > Thanks for your replies.
> > > >
> > > > AS30: Yes, that's right. Two different end markers for the same
> > transaction, and essentially unbounded duration of the inconsistency
> since
> > KIP-939. I agree that it's a class of transaction state divergence, like
> > KAFKA-20716. For KIP-1279, the most important thing is that consumers of
> > the mirrored topic must not get confused by the markers. For example,
> when
> > the Fetch response contains the list of aborted transactions, provided we
> > are sure that it can be built properly without confusion from the extra
> > marker, then we are good. It came to mind because the way this is
> > represented in the protocol for the responses is somewhat fragile I feel,
> > so I'm a bit paranoid that a mis-alignment could break it.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On 2026/06/19 12:16:52 Federico Valeri wrote:
> > > > > Hi Andrew, me, again.
> > > > >
> > > > > Given that READ_COMMITTED consumers are unaffected because the
> ABORT
> > > > > is already recorded in the transaction index and the dangling
> COMMIT
> > > > > produces no CompletedTxn (no index and LSO updates), the practical
> > > > > impact is limited. However, __transaction_state records the
> > > > > transaction as COMMITTED while the partition log has it as aborted.
> > > > > While analyzing this, we discovered that this type of divergence is
> > > > > not unique to mirroring. KAFKA-20716 documents the same category of
> > > > > inconsistency after unclean leader elections, where lost
> transaction
> > > > > markers cause the LSO to get stuck (much bigger problem). We
> believe
> > > > > that solving the transaction state divergence issue would also
> close
> > > > > this mirroring scenario. Wdyt?
> > > > >
> > > > > On Thu, Jun 18, 2026 at 10:36 PM Federico Valeri <
> > fedevaleri@gmail.com> wrote:
> > > > > >
> > > > > > Hi Andrew, about AS30, I had a closer look at KIP-939, and I
> think
> > > > > > this is what you mean:
> > > > > >
> > > > > > The problem is that a WriteTxnMarker request for a PID unknown to
> > the
> > > > > > ProducerStateManager (cleared by MIRROR_PID_RESET) does not fail.
> > The
> > > > > > end transaction marker is still appended, but transaction index
> and
> > > > > > LSO are unaffected (ProducerStateManager.prepareUpdate,
> > > > > > ProducerAppendInfo.appendEndTxnMarker). This may lead to a
> > situation
> > > > > > where we have two different end txn markers for the same
> > transaction:
> > > > > > ABORT from the stop mirroring operation, followed by a COMMIT
> from
> > the
> > > > > > local Txn Coordinator retrying its WriteTxnMarker after the
> > partition
> > > > > > becomes writable again.
> > > > > >
> > > > > > READ_COMMITTED consumers still see the earlier ABORT in the
> > > > > > transaction index and skip the aborted data. The visible outcome
> > for
> > > > > > consumers is correct. However, __transaction_state on the
> original
> > > > > > source cluster now records the transaction as COMMITTED while the
> > > > > > partition log has it as aborted, creating a silent inconsistency
> > that
> > > > > > tools querying transaction state will not detect. The window for
> > this
> > > > > > race is bounded by transaction.max.timeout.ms (default 15
> > minutes),
> > > > > > but becomes unbounded with KIP-939 because it allows prepared
> > > > > > transactions to sit indefinitely.
> > > > > >
> > > > > > We could move the coordinator state to “complete abort” at the
> > cost of
> > > > > > an RPC. This would prevent any retry, but that would break the
> 2PC
> > > > > > contract (Kafka must hold the prepared state until the external
> > > > > > coordinator decides). Also, before stop there would be a number
> of
> > > > > > retries constantly going on. We definitely need a better
> solution.
> > > > > >
> > > > > > On Thu, Jun 18, 2026 at 11:36 AM Federico Valeri <
> > fedevaleri@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Andrew, thanks for your careful review.
> > > > > > >
> > > > > > > AS28: We reworked that section and added some diagrams to
> > illustrate
> > > > > > > problem and solutions. Let us know if it is better.
> > > > > > >
> > > > > > > AS30: We updated that table, but we are not sure about the
> > problem you
> > > > > > > mention. Can you clarify and send us the sequence of actions
> that
> > > > > > > would lead to that problem?
> > > > > > >
> > > > > > > AS32: You are right. Replaced
> > StartMirrorTopicsRequestData.TopicData
> > > > > > > with a Controller.MirrorTopicMetadata record. Also renamed
> > TopicData
> > > > > > > to TopicMetadata in all schemas.
> > > > > > >
> > > > > > > AS33: Good idea. Added per-topic error messages in the RPC
> > responses
> > > > > > > you mentioned.
> > > > > > >
> > > > > > > AS29, AS31, AS34 (both of them), AS35, AS36: These small issues
> > are
> > > > > > > all fixed now.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 9, 2026 at 9:58 PM Andrew Schofield <
> > schofielaj@apache.org> wrote:
> > > > > > > >
> > > > > > > > Hi Fede and friends,
> > > > > > > > This KIP is shaping up nicely. The change to the way that
> > state management for the mirrors is done is much more elegant. The
> removal
> > of the PID mapping is also a nice improvement.
> > > > > > > >
> > > > > > > > Inevitably, I have some more comments.
> > > > > > > >
> > > > > > > > AS28: I understand KAFKA-18723 and how leader epochs work in
> > that case, but the enhancements that you made to the Fetch RPC and how
> they
> > work escapes me. Please could we have some diagrams to illustrate the
> > various epochs?
> > > > > > > >
> > > > > > > > AS29: I like the addition of share group offset syncing. You
> > need AlterShareGroupOffsets and DescribeShareGroupOffsets in the tables
> of
> > RPCs and permissions.
> > > > > > > >
> > > > > > > > AS30: It is certainly true that EOS is not supported. Using
> > cluster mirroring on transactional data does risk breaking transactional
> > atomicity for transactions in flight at the time that mirroring was
> > stopped. That's a known limitation, and it's an effect of per-topic async
> > replication. Writing ABORT markers when stop mirroring is triggered is
> > interesting. I just wonder whether it's truly safe.
> > > > > > > >
> > > > > > > > Could you enhance the tables of records in the Exactly-Once
> > Semantics section to include leader epochs? I think that the leader epoch
> > for the ABORT markers will have been bumped as the mirroring stopped and
> > the partitions became writable. However, I worry that someone might
> switch
> > the mirroring back in the other direction before the transaction which
> was
> > rudely partially aborted has completed on the original source cluster,
> and
> > that the effect of the transaction coordinator completing the
> transaction,
> > potentially writing duplicate transaction markers after the ABORT markers
> > which will have been replicated back again.
> > > > > > > >
> > > > > > > > This is probably only a realistic possibility when KIP-939 is
> > present, but it is accepted and I'm sure it will land soon enough. I
> would
> > say this KIP should be prepared for the existence of long-running
> > transactions so we don't dig a hole to fall into later.
> > > > > > > >
> > > > > > > > AS31: In the section on the Admin Client, the description for
> > Admin.createClusterMirror(String, Map, CreateClusterMirrorOptions) is
> > duplicated.
> > > > > > > >
> > > > > > > > AS32: The use of StartMirrorTopicsRequestData.TopicData in an
> > external interface looks odd.
> > > > > > > >
> > > > > > > > AS33: I suggest per-topic error messages in the RPC responses
> > such as PauseMirrorTopics so you can pass back more context, such as the
> > state if the request failed due to a state mismatch.
> > > > > > > >
> > > > > > > > AS34: There is an unnamed field in
> > DescribeClusterMirrorsResponse. The same in LastMirrorEpochsValue.
> > > > > > > >
> > > > > > > > AS34: Some of the fields in BumpLeaderEpochsRequest start
> with
> > lower-case letters. Also, there are quite a few fields in the RPCs in
> > general that should be version "0+" which are "0".
> > > > > > > >
> > > > > > > > AS35: Sometimes the partition states in the RPCs are string,
> > but sometimes int8.
> > > > > > > >
> > > > > > > > AS36: The base level for share group support is probably 4.1.
> > All of the RPCs were present in 4.1, although they were only enabled by
> > default in 4.2.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Andrew
> > > > > > > >
> > > > > > > > On 2026/06/04 11:01:36 Federico Valeri wrote:
> > > > > > > > > Hello, additional updates and replies:
> > > > > > > > >
> > > > > > > > > VK12: Consumer data loss risk is a good point. We updated
> > the "Group
> > > > > > > > > Offsets" paragraph.
> > > > > > > > >
> > > > > > > > > Some of you raised the point that using mirror.name
> > suffixes for pause
> > > > > > > > > and stop operations is a bit inelegant and we agreed. Now
> we
> > propose a
> > > > > > > > > better approach:
> > > > > > > > >
> > > > > > > > > We replaced the topic config based approach (mirror.name
> and
> > > > > > > > > .stopped/.paused suffix conventions) with a first class
> > metadata
> > > > > > > > > record for tracking mirror topic state changes. A new
> > > > > > > > > MirrorTopicStateChangeRecord is added to the metadata log
> > with three
> > > > > > > > > fields: TopicId (uuid), MirrorName (string), and
> > DesiredState (int8,
> > > > > > > > > where 0 = start mirroring, 1 = stop, 2 = pause). When a
> user
> > calls
> > > > > > > > > startMirrorTopics, stopMirrorTopics, or pauseMirrorTopics,
> > the
> > > > > > > > > controller writes this record to the metadata log instead
> of
> > > > > > > > > manipulating config suffixes. Brokers receive the record
> via
> > metadata
> > > > > > > > > updates and react accordingly, triggering partition level
> > state
> > > > > > > > > transitions through the existing MirrorPartitionState state
> > machine.
> > > > > > > > > More details in the updated KIP.
> > > > > > > > >
> > > > > > > > > Let us know what you think.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Fede
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jun 2, 2026 at 8:50 AM Luke Chen <
> showuon@gmail.com>
> > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Viquar,
> > > > > > > > > >
> > > > > > > > > > Thanks for the comment.
> > > > > > > > > >
> > > > > > > > > > Regarding VK12, thanks for raising the issue.
> > > > > > > > > > We didn't think about that.
> > > > > > > > > > I agree that compared with data re-processing, it's worse
> > to have data loss.
> > > > > > > > > > The proposed formula makes sense to me.
> > > > > > > > > > *syncedOffset = max(destinationLogStartOffset,
> > min(sourceCommitted,
> > > > > > > > > > destinationLEO))*
> > > > > > > > > >
> > > > > > > > > > Let us have some discussion internally and then update
> the
> > KIP and reply to
> > > > > > > > > > the thread.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Luke
> > > > > > > > > >
> > > > > > > > > > On Wed, May 20, 2026 at 2:39 AM Federico Valeri <
> > fedevaleri@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Rajini,
> > > > > > > > > > >
> > > > > > > > > > > RS14: Done. Missed that, sorry.
> > > > > > > > > > >
> > > > > > > > > > > Thanks again.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 19, 2026 at 7:28 PM Rajini Sivaram <
> > rajinisivaram@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Federico,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the updates. Just one minor point, apart
> > from that, looks
> > > > > > > > > > > good.
> > > > > > > > > > > >
> > > > > > > > > > > > RS14: KIP still shows "bin/kafka-configs.sh
> > --bootstrap-server :9094
> > > > > > > > > > > > --entity-type mirrors".
> > > > > > > > > > > > Will be good to change that to `--entity-type
> > cluster-mirrors`.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Rajini
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 19, 2026 at 5:28 PM vaquar khan <
> > vaquar.khan@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Federico and team,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you for your detailed response on May 11,
> > 2026. I greatly
> > > > > > > > > > > appreciate
> > > > > > > > > > > > > the collaborative effort over the past few months
> to
> > harden the
> > > > > > > > > > > KIP-1279
> > > > > > > > > > > > > architecture.
> > > > > > > > > > > > >
> > > > > > > > > > > > > * 1. Resolved Architectural & Stability
> > Vulnerabilities*
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am pleased to confirm that the KIP has
> > successfully integrated fixes
> > > > > > > > > > > for
> > > > > > > > > > > > > the critical vulnerabilities I flagged, thereby
> > protecting the
> > > > > > > > > > > cluster's
> > > > > > > > > > > > > state machine, memory bounds, and control plane.
> > Consider the following
> > > > > > > > > > > > > items fully resolved on my end:
> > > > > > > > > > > > >
> > > > > > > > > > > > > -
> > > > > > > > > > > > >
> > > > > > > > > > > > > *VK4 / VK8 (Negative PID Bug & State Machine
> > Failure):* I previously
> > > > > > > > > > > > > identified that the proposed -(sourceProducerId
> +
> > 2) PID rewriting
> > > > > > > > > > > > > formula would fundamentally break
> hasProducerId()
> > in
> > > > > > > > > > > > > AbstractRecordBatch.java, causing the broker to
> > bypass
> > > > > > > > > > > > > ProducerStateManager.update() and default the
> > Last Stable Offset
> > > > > > > > > > > (LSO)
> > > > > > > > > > > > > to the High Watermark. I am glad to see the KIP
> > authors abandoned
> > > > > > > > > > > PID
> > > > > > > > > > > > > mapping entirely and adopted the deterministic
> > MIRROR_PID_RESET
> > > > > > > > > > > control
> > > > > > > > > > > > > record barrier. This perfectly protects Kafka's
> > exactly-once
> > > > > > > > > > > semantics.
> > > > > > > > > > > > >
> > > > > > > > > > > > > -
> > > > > > > > > > > > >
> > > > > > > > > > > > > *VK11 (TransactionIndex Rebuild Ambiguity):*
> > Thank you for
> > > > > > > > > > > officially
> > > > > > > > > > > > > confirming that the TransactionIndex is strictly
> > rebuilt locally
> > > > > > > > > > > during
> > > > > > > > > > > > > log appends rather than copied byte-for-byte.
> > This resolves my
> > > > > > > > > > > concern
> > > > > > > > > > > > > regarding PID mismatches inducing consumer read
> > anomalies.
> > > > > > > > > > > > > -
> > > > > > > > > > > > >
> > > > > > > > > > > > > *VK1 (Thundering Herd / OOM Heap Allocation):*
> > Your clarification
> > > > > > > > > > > that
> > > > > > > > > > > > > fetcher threads multiplex partitions meaning the
> > memory footprint is
> > > > > > > > > > > > > strictly bounded by num_fetcher_threads *
> > response_max_bytes rather
> > > > > > > > > > > than
> > > > > > > > > > > > > a concurrent 1MB buffer per partition fully
> > resolves my concern
> > > > > > > > > > > > > regarding
> > > > > > > > > > > > > 50GB broker-wide OOM spikes during mass
> partition
> > wake-ups.
> > > > > > > > > > > > > -
> > > > > > > > > > > > >
> > > > > > > > > > > > > *VK3 (Control Plane Hotspots):* I had flagged
> the
> > severe risk of
> > > > > > > > > > > > > metadata saturation on a single broker during a
> > "link flap" event.
> > > > > > > > > > > Your
> > > > > > > > > > > > > confirmation that the __mirror_state topic
> > utilizes a compound hash
> > > > > > > > > > > > > of (mirrorName,
> > > > > > > > > > > > > topicId, partition number) mathematically
> ensures
> > that the 50,000+
> > > > > > > > > > > state
> > > > > > > > > > > > > transitions will be safely distributed across
> the
> > cluster,
> > > > > > > > > > > neutralizing
> > > > > > > > > > > > > the
> > > > > > > > > > > > > single-node hotspot risk.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. Outstanding Critical Blocker: VK12 (Offset Sync
> > Data Loss)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding VK12, there is a fundamental
> > misunderstanding in your
> > > > > > > > > > > previous
> > > > > > > > > > > > > reply. You stated: *"The scenario described can
> only
> > occur if offsets
> > > > > > > > > > > are
> > > > > > > > > > > > > force-written to an active group, which the design
> > prevents."*
> > > > > > > > > > > > >
> > > > > > > > > > > > > My concern has absolutely nothing to do with
> > overwriting active
> > > > > > > > > > > groups. My
> > > > > > > > > > > > > concern applies strictly to the normal
> > synchronization of inactive/dead
> > > > > > > > > > > > > groups, and is based directly on the race condition
> > currently
> > > > > > > > > > > documented in
> > > > > > > > > > > > > the official KIP-1279 text:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *"During offset synchronization, the committed
> > offset in the
> > > > > > > > > > > destination
> > > > > > > > > > > > > cluster may temporarily exceed the current log end
> > offset (LEO) of the
> > > > > > > > > > > > > mirror topic... consumers attempting to resume from
> > offset 100 will
> > > > > > > > > > > receive
> > > > > > > > > > > > > an OffsetOutOfRangeException. To handle this
> > gracefully, consumers
> > > > > > > > > > > should
> > > > > > > > > > > > > configure auto.offset.reset=latest..."*
> > > > > > > > > > > > >
> > > > > > > > > > > > > If a failover happens during this documented
> > divergence window, a
> > > > > > > > > > > > > reconnecting consumer will hit the
> > OffsetOutOfRangeException. If the
> > > > > > > > > > > > > downstream consumer follows the KIP's official
> > advice and relies on
> > > > > > > > > > > > > auto.offset.reset=latest, the consumer will jump to
> > the absolute newest
> > > > > > > > > > > > > offset on the partition. *This completely skips any
> > newly produced
> > > > > > > > > > > records
> > > > > > > > > > > > > that arrived between the failover and the consumer
> > reconnecting,
> > > > > > > > > > > resulting
> > > > > > > > > > > > > in silent, unrecoverable data loss for the
> > downstream application.*
> > > > > > > > > > > > > *Proposed Resolution: Double-Clamped Offset Safety
> > Invariant* Instead
> > > > > > > > > > > of
> > > > > > > > > > > > > requiring consumers to use auto.offset.reset=latest
> > and endorsing a
> > > > > > > > > > > known
> > > > > > > > > > > > > data loss vector, I propose that the
> > ClusterMirrorCoordinator enforce a
> > > > > > > > > > > > > pre-persist offset validation invariant during
> > offset synchronization.
> > > > > > > > > > > > > Before any translated offset is committed to the
> > destination cluster,
> > > > > > > > > > > the
> > > > > > > > > > > > > coordinator must apply a double-clamp:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *syncedOffset = max(destinationLSO,
> > min(sourceCommitted,
> > > > > > > > > > > destinationLEO))*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Where destinationLSO is the Log Start Offset
> > (earliest readable
> > > > > > > > > > > position
> > > > > > > > > > > > > post-retention) and destinationLEO is the Log End
> > Offset (latest
> > > > > > > > > > > replicated
> > > > > > > > > > > > > position) of the destination partition.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This guarantees that every persisted offset falls
> > within the physically
> > > > > > > > > > > > > valid range eliminating both the
> > OffsetOutOfRangeException caused by
> > > > > > > > > > > > > exceeding the LEO during replication lag, and the
> > expired-offset rewind
> > > > > > > > > > > > > caused by falling below the LSO due to destination
> > retention policies.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The worst-case trade-off under this invariant is
> > bounded re-processing
> > > > > > > > > > > > > proportional to the replication lag at failover
> time
> > not total
> > > > > > > > > > > partition
> > > > > > > > > > > > > depth which is perfectly consistent with Kafka's
> > documented
> > > > > > > > > > > at-least-once
> > > > > > > > > > > > > delivery guarantees. Broad enterprise production
> > evidence from
> > > > > > > > > > > large-scale
> > > > > > > > > > > > > cross-cluster failovers confirms that state checks
> > alone (preventing
> > > > > > > > > > > active
> > > > > > > > > > > > > group overwrites) are insufficient; strict offset
> > bounds clamping is
> > > > > > > > > > > > > required to achieve enterprise-grade data
> integrity.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I look forward to your thoughts on implementing
> this
> > final offset
> > > > > > > > > > > capping
> > > > > > > > > > > > > logic. Once VK12 is patched, I believe this
> > architecture will be
> > > > > > > > > > > > > exceptionally robust and ready for enterprise
> > deployment.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Viquar Khan
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, 18 May 2026 at 14:23, Rajini Sivaram <
> > rajinisivaram@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Federico,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the updates! The KIP is looking good.
> A
> > few more small
> > > > > > > > > > > > > comments.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > RS13: A couple of places still refer to
> > `kafka-mirror.sh` like under
> > > > > > > > > > > > > > `Failover Process`. Can we change them to
> > > > > > > > > > > `*kafka-cluster-mirrors.sh*`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > RS14: Should we change `--entity-type mirrors`
> for
> > kafka-configs to
> > > > > > > > > > > be `
> > > > > > > > > > > > > > --entity-type *cluster-mirrors*` to be
> consistent?
> > Also,
> > > > > > > > > > > > > > CLUSTER_MIRROR((byte)
> > > > > > > > > > > > > > 64, "mirror"); could be `*cluster-mirror*`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > RS15: It may be useful to rename
> > `mirror.topic.num.partitions` and `
> > > > > > > > > > > > > > mirror.topic.replication.factor` since they are
> > very similar to `
> > > > > > > > > > > > > > mirror.topic.properties.exclude`, but the
> > `mirror.topic` prefix
> > > > > > > > > > > refers to
> > > > > > > > > > > > > > different topics (the internal topic for the
> first
> > two and actual
> > > > > > > > > > > mirror
> > > > > > > > > > > > > > topics for the other one).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > RS16: ACL Sync: KIP says "Deletes ACLs that exist
> > in destination but
> > > > > > > > > > > not
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > source using DeleteAcls request."
> > > > > > > > > > > > > > What happens if someone creates an ACL on the
> > destination to deny
> > > > > > > > > > > > > > User:Alice access to all topics?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. If that ACL also existed on the source
> > cluster and then it was
> > > > > > > > > > > > > > removed, will it get removed from the
> > destination?
> > > > > > > > > > > > > > 2. If that ACL never existed on the source
> > cluster, will it get
> > > > > > > > > > > > > removed
> > > > > > > > > > > > > > from the destination?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > RS17: The table in the Source ACLs section says:
> > > > > > > > > > > "DescribeClusterMirrors
> > > > > > > > > > > > > > MC Read Cluster Log truncation"
> > > > > > > > > > > > > > Should that be `ClusterMirror:Read` instead of
> > `Cluster:Read`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Rajini
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, May 15, 2026 at 8:41 AM Federico Valeri <
> > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Rajini, we finally addressed the API and
> tool
> > naming
> > > > > > > > > > > refactoring as
> > > > > > > > > > > > > > > you suggested in RS6. Please take a look when
> > you have time.
> > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:17 PM Federico
> Valeri <
> > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hello all, I want to highlight a couple of
> new
> > paragraphs:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Leader Epoch Invariant: Cluster mirroring
> > enforces the
> > > > > > > > > > > invariant
> > > > > > > > > > > > > > > > that the destination leader epoch must always
> > be greater than or
> > > > > > > > > > > > > equal
> > > > > > > > > > > > > > > > to the source leader epoch (DLE>=SLE).
> Without
> > this, consumers
> > > > > > > > > > > on the
> > > > > > > > > > > > > > > > destination cluster can get stuck in an
> > infinite metadata refresh
> > > > > > > > > > > > > loop
> > > > > > > > > > > > > > > > when they encounter committed offsets
> carrying
> > source epochs
> > > > > > > > > > > higher
> > > > > > > > > > > > > > > > than the local epoch. The invariant is
> > maintained through three
> > > > > > > > > > > > > > > > mechanisms: reactive bumping (epoch fencing
> > triggered when SLE >
> > > > > > > > > > > DLE
> > > > > > > > > > > > > > > > during fetch), proactive bumping (scheduled
> > when SLE approaches
> > > > > > > > > > > DLE
> > > > > > > > > > > > > > > > within a threshold), and periodic bumping
> > (checked during
> > > > > > > > > > > coordinator
> > > > > > > > > > > > > > > > metadata sync).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973*KIP1279:ClusterMirroring-LeaderEpochInvariant__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqUcl_uWkU$
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. Group Offsets: The coordinator
> periodically
> > syncs consumer and
> > > > > > > > > > > > > > > > share group offsets from the source cluster
> to
> > the destination
> > > > > > > > > > > for
> > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > mirrored topics. Groups are filtered by
> > configurable
> > > > > > > > > > > include/exclude
> > > > > > > > > > > > > > > > patterns, and offsets are only synced for
> > groups that are not
> > > > > > > > > > > > > > > > currently active on the destination cluster,
> > preventing
> > > > > > > > > > > overwrites of
> > > > > > > > > > > > > > > > local consumer progress. Because source and
> > destination share the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > topic offsets (no offset translation), synced
> > offsets can be used
> > > > > > > > > > > > > > > > directly without mapping.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973*KIP1279:ClusterMirroring-GroupOffsets__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqU-JubnOA$
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > These new paragraphs directly address some of
> > your questions,
> > > > > > > > > > > but let
> > > > > > > > > > > > > > > > me list them here:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > JR2: Yes, we removed the incorrect phrase and
> > added more details
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > the paragraph.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > JR4: When source cluster topic has tiered
> > storage enabled, CM
> > > > > > > > > > > works
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > mirroring remote and local log into
> > destination cluster. When
> > > > > > > > > > > > > > > > destination cluster topic has tiered storage
> > enabled, CM fails in
> > > > > > > > > > > > > > > > PREPARING state because the LME may be in
> > remote storage, but
> > > > > > > > > > > works
> > > > > > > > > > > > > > > > fine if already MIRRORING because no
> > truncation is needed.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > JR11: See "Leader Epoch Invariant" paragraph
> > mentioned above.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > JR13: We can't support stateful Streams
> > application because
> > > > > > > > > > > > > > > > asynchronous replication cannot preserve the
> > transactional
> > > > > > > > > > > boundaries
> > > > > > > > > > > > > > > > between input offset commits, state store
> > mutations written to
> > > > > > > > > > > > > > > > changelog topics, and intermediate records
> > written to repartition
> > > > > > > > > > > > > > > > topics. The synchronous extension of this
> > design will be able to
> > > > > > > > > > > > > > > > support them. Existing Features Integration
> > paragraph updated.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > JR18: See "Group Offsets" paragraph mentioned
> > above.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > IY1: See "Group Offsets" paragraph mentioned
> > above.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:08 PM Federico
> > Valeri <
> > > > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Vaquar,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > VK4/VK8: We don't do PID mapping anymore.
> > The KIP was updated
> > > > > > > > > > > some
> > > > > > > > > > > > > > > > > time ago with the new approach based on the
> > new PID reset
> > > > > > > > > > > control
> > > > > > > > > > > > > > > > > record.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > VK11: The transaction index is always built
> > locally during log
> > > > > > > > > > > > > > append,
> > > > > > > > > > > > > > > > > never copied.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > VK1: The 50,000 * 1MB = 50GB calculation
> > misunderstands the
> > > > > > > > > > > fetch
> > > > > > > > > > > > > > > > > model. Fetcher threads don't allocate one
> > buffer per partition.
> > > > > > > > > > > > > > Actual
> > > > > > > > > > > > > > > > > peak memory is roughly num_fetcher_threads
> *
> > > > > > > > > > > response_max_bytes,
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > num_partitions * partition_max_bytes. With
> 1
> > fetcher thread
> > > > > > > > > > > and the
> > > > > > > > > > > > > > > > > default response max, the memory footprint
> > is modest
> > > > > > > > > > > regardless of
> > > > > > > > > > > > > > > > > partition count. We are leveraging the same
> > proven pattern
> > > > > > > > > > > used by
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > internal replication.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > VK3: The __mirror_state topic uses
> > hash-based partitioning
> > > > > > > > > > > based on
> > > > > > > > > > > > > > > > > mirrorName, topicId and partition number.
> > With the production
> > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > of 50 partitions, 50,000 partition
> > transitions distribute
> > > > > > > > > > > across
> > > > > > > > > > > > > ~50
> > > > > > > > > > > > > > > > > partition leaders on different brokers, not
> > a single broker.
> > > > > > > > > > > This
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > the same proven pattern as
> > __consumer_offsets, which handles
> > > > > > > > > > > > > millions
> > > > > > > > > > > > > > > > > of commits.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > VK12: The scenario described can only occur
> > if offsets are
> > > > > > > > > > > > > > > > > force-written to an active group, which the
> > design prevents.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> >
> >
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Federico,
Thanks for the updated KIP. A few more comments.
JR20. "Groups that are active on the destination cluster (i.e., in any
state other than EMPTY or DEAD) are skipped to avoid regressing offsets
after failover."
Does MirrorMetadataManager need to send an RPC to the group coordinator to
know the group's state?
JR21. Are mirrored records validated against
log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms
on the destination cluster?
JR22. "Leader Epoch Bump (LEB): The leader epoch in the destination cluster
remains unchanged during replication. That means it is possible that the
fetched batch carries a leader epoch that does not match with the local
leader epoch."
Is that correct? In the Leader Epoch Invariant section, there is logic to
makes sure SLE > DLE can't happen.
JR23. "A new dynamic topic level configuration
mirror.support.unclean.leader.election (boolean, default false) controls
whether LME truncation waits for all assigned replicas to converge, not
just current ISR members. When enabled, the leader will not complete
truncation until every replica has joined the ISR and truncated past the
LME offset. This prevents subsequent unclean leader elections from
introducing undetectable log divergence between clusters."
Is this to prevent issues with ULE on the source or destination?
JR24. "KIP-1327 introduces immutable creation timestamps on both partitions
and consumer groups to establish a happen-before relationship that
determines which offset reset policy applies when a consumer group
encounters a partition with no committed offset."
This needs adjustment since we decided not to pursue KIP-1327.
JR25. IncludePatterns/ExcludePatterns: Which one takes precedence if both
are specified?
JR26. For each record and RPC involving a state field, could you add a
description of the all possible states?
JR27. WriteMirrorStatesRequest
{ "name": "", "type": "int32", "versions": "0+", "default": "-1",
"about": "The last mirror leader epoch, or -1 if not available."
},
The field name is empty.
JR28. Could you describe how the StateValidationOffset field is being used?
JR29. "The ReplicaManager which provides source offset, destination offset,
and lag for each partition."
Which RPC is used to query the ReplicaManager?
JR30. Failover: The original destination b stores LME under the mirror name
a-to-b. During fallback, the original source a creates a new mirror name
b-to-a and uses it in DescribeClusterMirrors to obtain LME. How can the
original destination cluster b find the right LME if the stored and
requested mirror names don't match?
JR31. Start Mirror Topics:
JR31.1 "Controller persists include/exclude patterns as ConfigRecord
entries on the MIRROR resource in the metadata log."
Should we persist topics too? It's possible that the source topic doesn't
exist yet when a mirror is started.
JR31.2 "For each topic, the controller creates it on the destination if it
does not already exist, updates include/exclude topic pattern configs"
What are include/exclude topic pattern configs? In the previous step, the
controller already persists include/exclude patterns as ConfigRecord
entries on the MIRROR resource in the metadata log.
JR31.3 "Brokers receive the metadata update. The MirrorMetadataManager
detects the new mirrorName and DesiredState from the TopicImage and queries
the coordinator for the current partition state.
Partitions transition from UNKNOWN to LOG_TRUNCATION. During this state,
LME truncation runs and waits for all ISR members (or all replicas if ULE
is enabled)."
The MirrorMetadataManager instance responsible for a mirror is not
necessarily the same broker that leads a mirror partition. How can the
MirrorMetadataManager instance perform truncation? Ditto for Stop Mirror
Topics.
JR32. Stop Mirror Topics: "For each topic, the controller writes a
MirrorTopicStateChangeRecord with DesiredState=1 (stop)."
Is this record kept forever in the metadata snapshot?
JR33. Delete Mirror
JR33.1 he controller tombstones => The controller tombstones
JR34. "The mirror state records in __mirror_state internal topic are also
tombstoned."
Could you describe how MirrorTopicStateChangeRecord is tombstoned since it
doesn't have a key?
JR34. AdminClient:
JR34.1 Could you define the public facing classes such as
CreateClusterMirrorResult, StartMirrorTopicsResult,
PauseMirrorTopicsOptions, etc?
JR34.2 describeClusterMirrors: How does it populate the clusterId field in
DescribeClusterMirrorsRequest?
JR35. CreateTopic : "The replicas for the partition assignment are all
active and not in fenced or controlled shutdown. This is to make sure when
a topic gets deleted and re-created with the same topic ID, the stale
offline log dir won’t be treated as the active log dir after it becomes
online"
Could you explain why this is only needed for mirror topics, but not
regular topics?
JR36. Fetch: Could you explain why we need to add the MirrorLeaderEpoch
field? I thought the destination cluster just issues a regular fetch
request like a consumer.
JR37. CreateClusterMirror: "The broker validates that the mirror name is
not already in use, contains only permitted characters, and does not end
with .stopped or .paused suffix."
Could you explain why the suffix restriction is necessary?
JR38. MirrorMetadataManager: Could you explain how mirrors are assigned to
MirrorMetadataManager instances? Is it based on the _mrror_state topic
partitions?
JR39. StopMirrorTopics: Do we need TopicId? How does the AdminClient get it?
JR40. ReadMirrorStatesResponse: Could you explain how PreviousState is
being used?
JR41. PartitionChangeRecord.MinLeaderEpoch: Currently, leader epochs are
implicitly derived from PartitionChangeRecord. By adding this field, the
controller will sometimes derive the epoch and sometimes use the explicitly
stored epoch, which can be confusing. Would it be better to always store
the leader epoch explicitly in PartitionChangeRecord?
JR42. MirrorPartitionStateValue: Why does it use topicName, not topicId?
JR43. Mirror config: What about authentication (sasl, ssl, etc) related
configs?
JR44. Could you also describe how a partition leader picks up the mirroring
flow when it changes?
JR45. Broker config: sasl.mechanism.mirror.admin.protocol
Is that for mirror-related RPCs between brokers and controllers?
JR46. Compatibility
Since this KIP changes the metadata record format, we need an MV bump. For
mirror-related records and internal RPCs, should we introduce a
mirror-specific feature like GV/TV to gate them?
Jun
On Tue, Jul 7, 2026 at 6:40 AM Federico Valeri <fe...@gmail.com> wrote:
> Hi Michael, thanks for your review.
>
> MM1: It was a bit confusing but the coordinator is the scheduler and
> the manager is the executor. That said, given that the manager already
> has a scheduler, we moved that logic there and updated the KIP.
>
> MM2: Good point. A deleted source topic would now cause the mirror
> partitions on the destination cluster to move to FAILED state. You can
> easily identify them using the describe command with the --failed
> flag.
>
> MM3: We have a log message at warn level. I think this is enough.
>
> MM4: Correct. KIP updated.
>
> MM5: Correct. KIP updated.
>
> MM6: The isolation level is hardcoded to READ_UNCOMMITTED. Not sure
> about the actual use case, but we can certainly make it configurable
> in a follow-up KIP.
>
> MM7: For quota related topic configurations,
> follower.replication.throttled.replicas and
> leader.replication.throttled.replicas are excluded by default in MM2
> and we apply the same logic. mirror.support.unclean.leader.election
> and mirror.replication.throttled.replicas are included by default
> because they are mirror related configurations.
>
> MM8: Updated the field description to: Topic patterns to remove from
> mirror.topics.include or add to mirror.topics.exclude.
>
> MM9: Not needed. KIP updated.
>
> MM10: The config is needed because the listener can have multiple
> enabled SASL mechanisms and the listener name alone doesn't select
> one. The naming follows the existing
> sasl.mechanism.inter.broker.protocol convention, so renaming would be
> inconsistent with that precedent.
>
> MM11: Correct. KIP updated.
>
> MM12: No, when mirror.support.unclean.leader.election=true, we’ll wait
> for all replicas, not just ISR replicas to complete truncation.
>
> MM13: Correct. KIP updated.
>
> MM14: We missed the “MIRROR_TOPIC_NOT_STOPPED”, so now: start on non
> stopped throws MIRROR_TOPIC_NOT_STOPPED. Stop on paused throws
> MIRROR_TOPIC_ALREADY_PAUSED. Pause on stopped throws
> MIRROR_TOPIC_BEING_STOPPED. Resume on non paused throws
> MIRROR_TOPIC_NOT_PAUSED. Stop on stopped throws
> TOPIC_NOT_IN_CLUSTER_MIRROR.
>
>
>
> On Fri, Jul 3, 2026 at 5:26 PM Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > Thanks for the KIP!
> > This will definitively be a great addition to Kafka.
> >
> > MM1: In the description of the main components,
> > ClusterMirrorCoordinator has this responsibility:
> > "Metadata Refresh Scheduling: The coordinator schedules periodic
> > metadata refresh operations by invoking a metadata manager every 30
> > seconds by default. This ensures that configuration updates and group
> > offset commits in the source cluster are continuously propagated to
> > the destination cluster. The refresh interval is configurable via
> > mirror.metadata.refresh.interval.ms."
> > Is that accurate because MirrorMetadataManager has pretty much the
> > same responsibility.
> >
> > MM2: About topic deletions, the KIP states: "the operator would need
> > to manually remove the topic from the mirror"
> > Is there a way to find topics that have been deleted in the source
> > cluster for an operator to do that action?
> >
> > MM3: About group offsets: It states: "Groups with active members on
> > the destination are skipped."
> > Should these be logged too?
> >
> > MM4: In the security control section, the DeleteClusterMirror RPC
> > requires Alter on the ClusterMirror resource. Should we use Delete
> > instead of Alter?
> > On non-cluster resources, we typically use Delete for deletion. See
> >
> https://urldefense.com/v3/__https://kafka.apache.org/43/security/authorization-and-acls/*operations-and-resources-on-protocols__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqU3AKr-48$
> >
> > MM5: In Destination cluster permissions, I think FindCoordinator
> > should also need Describe on the Group resources for regular consumer
> > groups.
> >
> > MM6: In Exactly-Once Semantics, it states: "The mirror fetcher thread
> > uses READ_UNCOMMITTED isolation"
> > Could this be configurable? If so we can do this in a follow-up KIP
> >
> > MM7: It seems quotas are not mirrored between the clusters?
> > Is that omission intentional?
> >
> > MM8: In StopMirrorTopicsRequest I'm not sure I understand what the
> > Patterns field is?
> > Is it a pattern for the topics to stop? If so the description seems odd
> >
> > MM9: In
> StopMirrorTopicsResponse/StartMirrorTopicsResponse/PauseMirrorTopicsResponse/ResumeMirrorTopicsResponse
> > do we really need the mirror name?
> >
> > MM10: In Configuration / Broker:
> > Can you clarify why we need sasl.mechanism.mirror.admin.protocol?
> > Can't we retrieve the protocol from mirror.admin.listener.name or
> > inter.broker.listener.name?
> > If we really need it, should it be named
> > mirror.admin.sasl.mechanism.protocol to match the other names?
> >
> > MM11: In Configuration / Mirror
> > Should it be mirror.acls.include instead of mirror.acl.include to
> > match mirror.groups.include and mirror.topics.include?
> >
> > MM12: Is the description of mirror.support.unclean.leader.election
> inverted?
> >
> > MM13: The description of CLUSTER_MIRROR_NOT_EMPTY states: "The cluster
> > mirror still has active or non-removed topics"
> > Should it be "The cluster mirror still has active or non-stopped topics"?
> >
> > MM14: We have the MIRROR_TOPIC_BEING_STOPPED error when stopping topic
> > already in stopping state? What happens when pausing topics already in
> > pausing state?
> > Related, I assume when stopping a stopped topic, you get
> > TOPIC_NOT_IN_CLUSTER_MIRROR
> >
> > Thanks,
> > Mickael
> >
> > On Mon, Jun 22, 2026 at 7:47 PM Andrew Schofield <sc...@apache.org>
> wrote:
> > >
> > > Hi Fede,
> > > Thanks for your replies.
> > >
> > > AS30: Yes, that's right. Two different end markers for the same
> transaction, and essentially unbounded duration of the inconsistency since
> KIP-939. I agree that it's a class of transaction state divergence, like
> KAFKA-20716. For KIP-1279, the most important thing is that consumers of
> the mirrored topic must not get confused by the markers. For example, when
> the Fetch response contains the list of aborted transactions, provided we
> are sure that it can be built properly without confusion from the extra
> marker, then we are good. It came to mind because the way this is
> represented in the protocol for the responses is somewhat fragile I feel,
> so I'm a bit paranoid that a mis-alignment could break it.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On 2026/06/19 12:16:52 Federico Valeri wrote:
> > > > Hi Andrew, me, again.
> > > >
> > > > Given that READ_COMMITTED consumers are unaffected because the ABORT
> > > > is already recorded in the transaction index and the dangling COMMIT
> > > > produces no CompletedTxn (no index and LSO updates), the practical
> > > > impact is limited. However, __transaction_state records the
> > > > transaction as COMMITTED while the partition log has it as aborted.
> > > > While analyzing this, we discovered that this type of divergence is
> > > > not unique to mirroring. KAFKA-20716 documents the same category of
> > > > inconsistency after unclean leader elections, where lost transaction
> > > > markers cause the LSO to get stuck (much bigger problem). We believe
> > > > that solving the transaction state divergence issue would also close
> > > > this mirroring scenario. Wdyt?
> > > >
> > > > On Thu, Jun 18, 2026 at 10:36 PM Federico Valeri <
> fedevaleri@gmail.com> wrote:
> > > > >
> > > > > Hi Andrew, about AS30, I had a closer look at KIP-939, and I think
> > > > > this is what you mean:
> > > > >
> > > > > The problem is that a WriteTxnMarker request for a PID unknown to
> the
> > > > > ProducerStateManager (cleared by MIRROR_PID_RESET) does not fail.
> The
> > > > > end transaction marker is still appended, but transaction index and
> > > > > LSO are unaffected (ProducerStateManager.prepareUpdate,
> > > > > ProducerAppendInfo.appendEndTxnMarker). This may lead to a
> situation
> > > > > where we have two different end txn markers for the same
> transaction:
> > > > > ABORT from the stop mirroring operation, followed by a COMMIT from
> the
> > > > > local Txn Coordinator retrying its WriteTxnMarker after the
> partition
> > > > > becomes writable again.
> > > > >
> > > > > READ_COMMITTED consumers still see the earlier ABORT in the
> > > > > transaction index and skip the aborted data. The visible outcome
> for
> > > > > consumers is correct. However, __transaction_state on the original
> > > > > source cluster now records the transaction as COMMITTED while the
> > > > > partition log has it as aborted, creating a silent inconsistency
> that
> > > > > tools querying transaction state will not detect. The window for
> this
> > > > > race is bounded by transaction.max.timeout.ms (default 15
> minutes),
> > > > > but becomes unbounded with KIP-939 because it allows prepared
> > > > > transactions to sit indefinitely.
> > > > >
> > > > > We could move the coordinator state to “complete abort” at the
> cost of
> > > > > an RPC. This would prevent any retry, but that would break the 2PC
> > > > > contract (Kafka must hold the prepared state until the external
> > > > > coordinator decides). Also, before stop there would be a number of
> > > > > retries constantly going on. We definitely need a better solution.
> > > > >
> > > > > On Thu, Jun 18, 2026 at 11:36 AM Federico Valeri <
> fedevaleri@gmail.com> wrote:
> > > > > >
> > > > > > Hi Andrew, thanks for your careful review.
> > > > > >
> > > > > > AS28: We reworked that section and added some diagrams to
> illustrate
> > > > > > problem and solutions. Let us know if it is better.
> > > > > >
> > > > > > AS30: We updated that table, but we are not sure about the
> problem you
> > > > > > mention. Can you clarify and send us the sequence of actions that
> > > > > > would lead to that problem?
> > > > > >
> > > > > > AS32: You are right. Replaced
> StartMirrorTopicsRequestData.TopicData
> > > > > > with a Controller.MirrorTopicMetadata record. Also renamed
> TopicData
> > > > > > to TopicMetadata in all schemas.
> > > > > >
> > > > > > AS33: Good idea. Added per-topic error messages in the RPC
> responses
> > > > > > you mentioned.
> > > > > >
> > > > > > AS29, AS31, AS34 (both of them), AS35, AS36: These small issues
> are
> > > > > > all fixed now.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 9, 2026 at 9:58 PM Andrew Schofield <
> schofielaj@apache.org> wrote:
> > > > > > >
> > > > > > > Hi Fede and friends,
> > > > > > > This KIP is shaping up nicely. The change to the way that
> state management for the mirrors is done is much more elegant. The removal
> of the PID mapping is also a nice improvement.
> > > > > > >
> > > > > > > Inevitably, I have some more comments.
> > > > > > >
> > > > > > > AS28: I understand KAFKA-18723 and how leader epochs work in
> that case, but the enhancements that you made to the Fetch RPC and how they
> work escapes me. Please could we have some diagrams to illustrate the
> various epochs?
> > > > > > >
> > > > > > > AS29: I like the addition of share group offset syncing. You
> need AlterShareGroupOffsets and DescribeShareGroupOffsets in the tables of
> RPCs and permissions.
> > > > > > >
> > > > > > > AS30: It is certainly true that EOS is not supported. Using
> cluster mirroring on transactional data does risk breaking transactional
> atomicity for transactions in flight at the time that mirroring was
> stopped. That's a known limitation, and it's an effect of per-topic async
> replication. Writing ABORT markers when stop mirroring is triggered is
> interesting. I just wonder whether it's truly safe.
> > > > > > >
> > > > > > > Could you enhance the tables of records in the Exactly-Once
> Semantics section to include leader epochs? I think that the leader epoch
> for the ABORT markers will have been bumped as the mirroring stopped and
> the partitions became writable. However, I worry that someone might switch
> the mirroring back in the other direction before the transaction which was
> rudely partially aborted has completed on the original source cluster, and
> that the effect of the transaction coordinator completing the transaction,
> potentially writing duplicate transaction markers after the ABORT markers
> which will have been replicated back again.
> > > > > > >
> > > > > > > This is probably only a realistic possibility when KIP-939 is
> present, but it is accepted and I'm sure it will land soon enough. I would
> say this KIP should be prepared for the existence of long-running
> transactions so we don't dig a hole to fall into later.
> > > > > > >
> > > > > > > AS31: In the section on the Admin Client, the description for
> Admin.createClusterMirror(String, Map, CreateClusterMirrorOptions) is
> duplicated.
> > > > > > >
> > > > > > > AS32: The use of StartMirrorTopicsRequestData.TopicData in an
> external interface looks odd.
> > > > > > >
> > > > > > > AS33: I suggest per-topic error messages in the RPC responses
> such as PauseMirrorTopics so you can pass back more context, such as the
> state if the request failed due to a state mismatch.
> > > > > > >
> > > > > > > AS34: There is an unnamed field in
> DescribeClusterMirrorsResponse. The same in LastMirrorEpochsValue.
> > > > > > >
> > > > > > > AS34: Some of the fields in BumpLeaderEpochsRequest start with
> lower-case letters. Also, there are quite a few fields in the RPCs in
> general that should be version "0+" which are "0".
> > > > > > >
> > > > > > > AS35: Sometimes the partition states in the RPCs are string,
> but sometimes int8.
> > > > > > >
> > > > > > > AS36: The base level for share group support is probably 4.1.
> All of the RPCs were present in 4.1, although they were only enabled by
> default in 4.2.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Andrew
> > > > > > >
> > > > > > > On 2026/06/04 11:01:36 Federico Valeri wrote:
> > > > > > > > Hello, additional updates and replies:
> > > > > > > >
> > > > > > > > VK12: Consumer data loss risk is a good point. We updated
> the "Group
> > > > > > > > Offsets" paragraph.
> > > > > > > >
> > > > > > > > Some of you raised the point that using mirror.name
> suffixes for pause
> > > > > > > > and stop operations is a bit inelegant and we agreed. Now we
> propose a
> > > > > > > > better approach:
> > > > > > > >
> > > > > > > > We replaced the topic config based approach (mirror.name and
> > > > > > > > .stopped/.paused suffix conventions) with a first class
> metadata
> > > > > > > > record for tracking mirror topic state changes. A new
> > > > > > > > MirrorTopicStateChangeRecord is added to the metadata log
> with three
> > > > > > > > fields: TopicId (uuid), MirrorName (string), and
> DesiredState (int8,
> > > > > > > > where 0 = start mirroring, 1 = stop, 2 = pause). When a user
> calls
> > > > > > > > startMirrorTopics, stopMirrorTopics, or pauseMirrorTopics,
> the
> > > > > > > > controller writes this record to the metadata log instead of
> > > > > > > > manipulating config suffixes. Brokers receive the record via
> metadata
> > > > > > > > updates and react accordingly, triggering partition level
> state
> > > > > > > > transitions through the existing MirrorPartitionState state
> machine.
> > > > > > > > More details in the updated KIP.
> > > > > > > >
> > > > > > > > Let us know what you think.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Fede
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 2, 2026 at 8:50 AM Luke Chen <sh...@gmail.com>
> wrote:
> > > > > > > > >
> > > > > > > > > Hi Viquar,
> > > > > > > > >
> > > > > > > > > Thanks for the comment.
> > > > > > > > >
> > > > > > > > > Regarding VK12, thanks for raising the issue.
> > > > > > > > > We didn't think about that.
> > > > > > > > > I agree that compared with data re-processing, it's worse
> to have data loss.
> > > > > > > > > The proposed formula makes sense to me.
> > > > > > > > > *syncedOffset = max(destinationLogStartOffset,
> min(sourceCommitted,
> > > > > > > > > destinationLEO))*
> > > > > > > > >
> > > > > > > > > Let us have some discussion internally and then update the
> KIP and reply to
> > > > > > > > > the thread.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Luke
> > > > > > > > >
> > > > > > > > > On Wed, May 20, 2026 at 2:39 AM Federico Valeri <
> fedevaleri@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Rajini,
> > > > > > > > > >
> > > > > > > > > > RS14: Done. Missed that, sorry.
> > > > > > > > > >
> > > > > > > > > > Thanks again.
> > > > > > > > > >
> > > > > > > > > > On Tue, May 19, 2026 at 7:28 PM Rajini Sivaram <
> rajinisivaram@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Federico,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the updates. Just one minor point, apart
> from that, looks
> > > > > > > > > > good.
> > > > > > > > > > >
> > > > > > > > > > > RS14: KIP still shows "bin/kafka-configs.sh
> --bootstrap-server :9094
> > > > > > > > > > > --entity-type mirrors".
> > > > > > > > > > > Will be good to change that to `--entity-type
> cluster-mirrors`.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Rajini
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 19, 2026 at 5:28 PM vaquar khan <
> vaquar.khan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Federico and team,
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you for your detailed response on May 11,
> 2026. I greatly
> > > > > > > > > > appreciate
> > > > > > > > > > > > the collaborative effort over the past few months to
> harden the
> > > > > > > > > > KIP-1279
> > > > > > > > > > > > architecture.
> > > > > > > > > > > >
> > > > > > > > > > > > * 1. Resolved Architectural & Stability
> Vulnerabilities*
> > > > > > > > > > > >
> > > > > > > > > > > > I am pleased to confirm that the KIP has
> successfully integrated fixes
> > > > > > > > > > for
> > > > > > > > > > > > the critical vulnerabilities I flagged, thereby
> protecting the
> > > > > > > > > > cluster's
> > > > > > > > > > > > state machine, memory bounds, and control plane.
> Consider the following
> > > > > > > > > > > > items fully resolved on my end:
> > > > > > > > > > > >
> > > > > > > > > > > > -
> > > > > > > > > > > >
> > > > > > > > > > > > *VK4 / VK8 (Negative PID Bug & State Machine
> Failure):* I previously
> > > > > > > > > > > > identified that the proposed -(sourceProducerId +
> 2) PID rewriting
> > > > > > > > > > > > formula would fundamentally break hasProducerId()
> in
> > > > > > > > > > > > AbstractRecordBatch.java, causing the broker to
> bypass
> > > > > > > > > > > > ProducerStateManager.update() and default the
> Last Stable Offset
> > > > > > > > > > (LSO)
> > > > > > > > > > > > to the High Watermark. I am glad to see the KIP
> authors abandoned
> > > > > > > > > > PID
> > > > > > > > > > > > mapping entirely and adopted the deterministic
> MIRROR_PID_RESET
> > > > > > > > > > control
> > > > > > > > > > > > record barrier. This perfectly protects Kafka's
> exactly-once
> > > > > > > > > > semantics.
> > > > > > > > > > > >
> > > > > > > > > > > > -
> > > > > > > > > > > >
> > > > > > > > > > > > *VK11 (TransactionIndex Rebuild Ambiguity):*
> Thank you for
> > > > > > > > > > officially
> > > > > > > > > > > > confirming that the TransactionIndex is strictly
> rebuilt locally
> > > > > > > > > > during
> > > > > > > > > > > > log appends rather than copied byte-for-byte.
> This resolves my
> > > > > > > > > > concern
> > > > > > > > > > > > regarding PID mismatches inducing consumer read
> anomalies.
> > > > > > > > > > > > -
> > > > > > > > > > > >
> > > > > > > > > > > > *VK1 (Thundering Herd / OOM Heap Allocation):*
> Your clarification
> > > > > > > > > > that
> > > > > > > > > > > > fetcher threads multiplex partitions meaning the
> memory footprint is
> > > > > > > > > > > > strictly bounded by num_fetcher_threads *
> response_max_bytes rather
> > > > > > > > > > than
> > > > > > > > > > > > a concurrent 1MB buffer per partition fully
> resolves my concern
> > > > > > > > > > > > regarding
> > > > > > > > > > > > 50GB broker-wide OOM spikes during mass partition
> wake-ups.
> > > > > > > > > > > > -
> > > > > > > > > > > >
> > > > > > > > > > > > *VK3 (Control Plane Hotspots):* I had flagged the
> severe risk of
> > > > > > > > > > > > metadata saturation on a single broker during a
> "link flap" event.
> > > > > > > > > > Your
> > > > > > > > > > > > confirmation that the __mirror_state topic
> utilizes a compound hash
> > > > > > > > > > > > of (mirrorName,
> > > > > > > > > > > > topicId, partition number) mathematically ensures
> that the 50,000+
> > > > > > > > > > state
> > > > > > > > > > > > transitions will be safely distributed across the
> cluster,
> > > > > > > > > > neutralizing
> > > > > > > > > > > > the
> > > > > > > > > > > > single-node hotspot risk.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. Outstanding Critical Blocker: VK12 (Offset Sync
> Data Loss)
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding VK12, there is a fundamental
> misunderstanding in your
> > > > > > > > > > previous
> > > > > > > > > > > > reply. You stated: *"The scenario described can only
> occur if offsets
> > > > > > > > > > are
> > > > > > > > > > > > force-written to an active group, which the design
> prevents."*
> > > > > > > > > > > >
> > > > > > > > > > > > My concern has absolutely nothing to do with
> overwriting active
> > > > > > > > > > groups. My
> > > > > > > > > > > > concern applies strictly to the normal
> synchronization of inactive/dead
> > > > > > > > > > > > groups, and is based directly on the race condition
> currently
> > > > > > > > > > documented in
> > > > > > > > > > > > the official KIP-1279 text:
> > > > > > > > > > > >
> > > > > > > > > > > > *"During offset synchronization, the committed
> offset in the
> > > > > > > > > > destination
> > > > > > > > > > > > cluster may temporarily exceed the current log end
> offset (LEO) of the
> > > > > > > > > > > > mirror topic... consumers attempting to resume from
> offset 100 will
> > > > > > > > > > receive
> > > > > > > > > > > > an OffsetOutOfRangeException. To handle this
> gracefully, consumers
> > > > > > > > > > should
> > > > > > > > > > > > configure auto.offset.reset=latest..."*
> > > > > > > > > > > >
> > > > > > > > > > > > If a failover happens during this documented
> divergence window, a
> > > > > > > > > > > > reconnecting consumer will hit the
> OffsetOutOfRangeException. If the
> > > > > > > > > > > > downstream consumer follows the KIP's official
> advice and relies on
> > > > > > > > > > > > auto.offset.reset=latest, the consumer will jump to
> the absolute newest
> > > > > > > > > > > > offset on the partition. *This completely skips any
> newly produced
> > > > > > > > > > records
> > > > > > > > > > > > that arrived between the failover and the consumer
> reconnecting,
> > > > > > > > > > resulting
> > > > > > > > > > > > in silent, unrecoverable data loss for the
> downstream application.*
> > > > > > > > > > > > *Proposed Resolution: Double-Clamped Offset Safety
> Invariant* Instead
> > > > > > > > > > of
> > > > > > > > > > > > requiring consumers to use auto.offset.reset=latest
> and endorsing a
> > > > > > > > > > known
> > > > > > > > > > > > data loss vector, I propose that the
> ClusterMirrorCoordinator enforce a
> > > > > > > > > > > > pre-persist offset validation invariant during
> offset synchronization.
> > > > > > > > > > > > Before any translated offset is committed to the
> destination cluster,
> > > > > > > > > > the
> > > > > > > > > > > > coordinator must apply a double-clamp:
> > > > > > > > > > > >
> > > > > > > > > > > > *syncedOffset = max(destinationLSO,
> min(sourceCommitted,
> > > > > > > > > > destinationLEO))*
> > > > > > > > > > > >
> > > > > > > > > > > > Where destinationLSO is the Log Start Offset
> (earliest readable
> > > > > > > > > > position
> > > > > > > > > > > > post-retention) and destinationLEO is the Log End
> Offset (latest
> > > > > > > > > > replicated
> > > > > > > > > > > > position) of the destination partition.
> > > > > > > > > > > >
> > > > > > > > > > > > This guarantees that every persisted offset falls
> within the physically
> > > > > > > > > > > > valid range eliminating both the
> OffsetOutOfRangeException caused by
> > > > > > > > > > > > exceeding the LEO during replication lag, and the
> expired-offset rewind
> > > > > > > > > > > > caused by falling below the LSO due to destination
> retention policies.
> > > > > > > > > > > >
> > > > > > > > > > > > The worst-case trade-off under this invariant is
> bounded re-processing
> > > > > > > > > > > > proportional to the replication lag at failover time
> not total
> > > > > > > > > > partition
> > > > > > > > > > > > depth which is perfectly consistent with Kafka's
> documented
> > > > > > > > > > at-least-once
> > > > > > > > > > > > delivery guarantees. Broad enterprise production
> evidence from
> > > > > > > > > > large-scale
> > > > > > > > > > > > cross-cluster failovers confirms that state checks
> alone (preventing
> > > > > > > > > > active
> > > > > > > > > > > > group overwrites) are insufficient; strict offset
> bounds clamping is
> > > > > > > > > > > > required to achieve enterprise-grade data integrity.
> > > > > > > > > > > >
> > > > > > > > > > > > I look forward to your thoughts on implementing this
> final offset
> > > > > > > > > > capping
> > > > > > > > > > > > logic. Once VK12 is patched, I believe this
> architecture will be
> > > > > > > > > > > > exceptionally robust and ready for enterprise
> deployment.
> > > > > > > > > > > >
> > > > > > > > > > > > Best Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Viquar Khan
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, 18 May 2026 at 14:23, Rajini Sivaram <
> rajinisivaram@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Federico,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the updates! The KIP is looking good. A
> few more small
> > > > > > > > > > > > comments.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > RS13: A couple of places still refer to
> `kafka-mirror.sh` like under
> > > > > > > > > > > > > `Failover Process`. Can we change them to
> > > > > > > > > > `*kafka-cluster-mirrors.sh*`?
> > > > > > > > > > > > >
> > > > > > > > > > > > > RS14: Should we change `--entity-type mirrors` for
> kafka-configs to
> > > > > > > > > > be `
> > > > > > > > > > > > > --entity-type *cluster-mirrors*` to be consistent?
> Also,
> > > > > > > > > > > > > CLUSTER_MIRROR((byte)
> > > > > > > > > > > > > 64, "mirror"); could be `*cluster-mirror*`?
> > > > > > > > > > > > >
> > > > > > > > > > > > > RS15: It may be useful to rename
> `mirror.topic.num.partitions` and `
> > > > > > > > > > > > > mirror.topic.replication.factor` since they are
> very similar to `
> > > > > > > > > > > > > mirror.topic.properties.exclude`, but the
> `mirror.topic` prefix
> > > > > > > > > > refers to
> > > > > > > > > > > > > different topics (the internal topic for the first
> two and actual
> > > > > > > > > > mirror
> > > > > > > > > > > > > topics for the other one).
> > > > > > > > > > > > >
> > > > > > > > > > > > > RS16: ACL Sync: KIP says "Deletes ACLs that exist
> in destination but
> > > > > > > > > > not
> > > > > > > > > > > > in
> > > > > > > > > > > > > source using DeleteAcls request."
> > > > > > > > > > > > > What happens if someone creates an ACL on the
> destination to deny
> > > > > > > > > > > > > User:Alice access to all topics?
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. If that ACL also existed on the source
> cluster and then it was
> > > > > > > > > > > > > removed, will it get removed from the
> destination?
> > > > > > > > > > > > > 2. If that ACL never existed on the source
> cluster, will it get
> > > > > > > > > > > > removed
> > > > > > > > > > > > > from the destination?
> > > > > > > > > > > > >
> > > > > > > > > > > > > RS17: The table in the Source ACLs section says:
> > > > > > > > > > "DescribeClusterMirrors
> > > > > > > > > > > > > MC Read Cluster Log truncation"
> > > > > > > > > > > > > Should that be `ClusterMirror:Read` instead of
> `Cluster:Read`?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rajini
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, May 15, 2026 at 8:41 AM Federico Valeri <
> > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Rajini, we finally addressed the API and tool
> naming
> > > > > > > > > > refactoring as
> > > > > > > > > > > > > > you suggested in RS6. Please take a look when
> you have time.
> > > > > > > > > > Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:17 PM Federico Valeri <
> > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hello all, I want to highlight a couple of new
> paragraphs:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Leader Epoch Invariant: Cluster mirroring
> enforces the
> > > > > > > > > > invariant
> > > > > > > > > > > > > > > that the destination leader epoch must always
> be greater than or
> > > > > > > > > > > > equal
> > > > > > > > > > > > > > > to the source leader epoch (DLE>=SLE). Without
> this, consumers
> > > > > > > > > > on the
> > > > > > > > > > > > > > > destination cluster can get stuck in an
> infinite metadata refresh
> > > > > > > > > > > > loop
> > > > > > > > > > > > > > > when they encounter committed offsets carrying
> source epochs
> > > > > > > > > > higher
> > > > > > > > > > > > > > > than the local epoch. The invariant is
> maintained through three
> > > > > > > > > > > > > > > mechanisms: reactive bumping (epoch fencing
> triggered when SLE >
> > > > > > > > > > DLE
> > > > > > > > > > > > > > > during fetch), proactive bumping (scheduled
> when SLE approaches
> > > > > > > > > > DLE
> > > > > > > > > > > > > > > within a threshold), and periodic bumping
> (checked during
> > > > > > > > > > coordinator
> > > > > > > > > > > > > > > metadata sync).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973*KIP1279:ClusterMirroring-LeaderEpochInvariant__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqUcl_uWkU$
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. Group Offsets: The coordinator periodically
> syncs consumer and
> > > > > > > > > > > > > > > share group offsets from the source cluster to
> the destination
> > > > > > > > > > for
> > > > > > > > > > > > all
> > > > > > > > > > > > > > > mirrored topics. Groups are filtered by
> configurable
> > > > > > > > > > include/exclude
> > > > > > > > > > > > > > > patterns, and offsets are only synced for
> groups that are not
> > > > > > > > > > > > > > > currently active on the destination cluster,
> preventing
> > > > > > > > > > overwrites of
> > > > > > > > > > > > > > > local consumer progress. Because source and
> destination share the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > > topic offsets (no offset translation), synced
> offsets can be used
> > > > > > > > > > > > > > > directly without mapping.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973*KIP1279:ClusterMirroring-GroupOffsets__;Iw!!Ayb5sqE7!tihx6ElDOI77AtiHliFlPPSZqZJwpnkf9ZzYWGySbnmOir-2Y0uKuum9iDrYrvt-HCTR3A8oaxqU-JubnOA$
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > These new paragraphs directly address some of
> your questions,
> > > > > > > > > > but let
> > > > > > > > > > > > > > > me list them here:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > JR2: Yes, we removed the incorrect phrase and
> added more details
> > > > > > > > > > to
> > > > > > > > > > > > > > > the paragraph.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > JR4: When source cluster topic has tiered
> storage enabled, CM
> > > > > > > > > > works
> > > > > > > > > > > > by
> > > > > > > > > > > > > > > mirroring remote and local log into
> destination cluster. When
> > > > > > > > > > > > > > > destination cluster topic has tiered storage
> enabled, CM fails in
> > > > > > > > > > > > > > > PREPARING state because the LME may be in
> remote storage, but
> > > > > > > > > > works
> > > > > > > > > > > > > > > fine if already MIRRORING because no
> truncation is needed.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > JR11: See "Leader Epoch Invariant" paragraph
> mentioned above.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > JR13: We can't support stateful Streams
> application because
> > > > > > > > > > > > > > > asynchronous replication cannot preserve the
> transactional
> > > > > > > > > > boundaries
> > > > > > > > > > > > > > > between input offset commits, state store
> mutations written to
> > > > > > > > > > > > > > > changelog topics, and intermediate records
> written to repartition
> > > > > > > > > > > > > > > topics. The synchronous extension of this
> design will be able to
> > > > > > > > > > > > > > > support them. Existing Features Integration
> paragraph updated.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > JR18: See "Group Offsets" paragraph mentioned
> above.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > IY1: See "Group Offsets" paragraph mentioned
> above.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:08 PM Federico
> Valeri <
> > > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Vaquar,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > VK4/VK8: We don't do PID mapping anymore.
> The KIP was updated
> > > > > > > > > > some
> > > > > > > > > > > > > > > > time ago with the new approach based on the
> new PID reset
> > > > > > > > > > control
> > > > > > > > > > > > > > > > record.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > VK11: The transaction index is always built
> locally during log
> > > > > > > > > > > > > append,
> > > > > > > > > > > > > > > > never copied.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > VK1: The 50,000 * 1MB = 50GB calculation
> misunderstands the
> > > > > > > > > > fetch
> > > > > > > > > > > > > > > > model. Fetcher threads don't allocate one
> buffer per partition.
> > > > > > > > > > > > > Actual
> > > > > > > > > > > > > > > > peak memory is roughly num_fetcher_threads *
> > > > > > > > > > response_max_bytes,
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > num_partitions * partition_max_bytes. With 1
> fetcher thread
> > > > > > > > > > and the
> > > > > > > > > > > > > > > > default response max, the memory footprint
> is modest
> > > > > > > > > > regardless of
> > > > > > > > > > > > > > > > partition count. We are leveraging the same
> proven pattern
> > > > > > > > > > used by
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > internal replication.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > VK3: The __mirror_state topic uses
> hash-based partitioning
> > > > > > > > > > based on
> > > > > > > > > > > > > > > > mirrorName, topicId and partition number.
> With the production
> > > > > > > > > > > > default
> > > > > > > > > > > > > > > > of 50 partitions, 50,000 partition
> transitions distribute
> > > > > > > > > > across
> > > > > > > > > > > > ~50
> > > > > > > > > > > > > > > > partition leaders on different brokers, not
> a single broker.
> > > > > > > > > > This
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > the same proven pattern as
> __consumer_offsets, which handles
> > > > > > > > > > > > millions
> > > > > > > > > > > > > > > > of commits.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > VK12: The scenario described can only occur
> if offsets are
> > > > > > > > > > > > > > > > force-written to an active group, which the
> design prevents.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > >
>
>
Re: [DISCUSS] KIP-1279: Cluster Mirroring
Posted by Federico Valeri <fe...@gmail.com>.
Hi Michael, thanks for your review.
MM1: It was a bit confusing but the coordinator is the scheduler and
the manager is the executor. That said, given that the manager already
has a scheduler, we moved that logic there and updated the KIP.
MM2: Good point. A deleted source topic would now cause the mirror
partitions on the destination cluster to move to FAILED state. You can
easily identify them using the describe command with the --failed
flag.
MM3: We have a log message at warn level. I think this is enough.
MM4: Correct. KIP updated.
MM5: Correct. KIP updated.
MM6: The isolation level is hardcoded to READ_UNCOMMITTED. Not sure
about the actual use case, but we can certainly make it configurable
in a follow-up KIP.
MM7: For quota related topic configurations,
follower.replication.throttled.replicas and
leader.replication.throttled.replicas are excluded by default in MM2
and we apply the same logic. mirror.support.unclean.leader.election
and mirror.replication.throttled.replicas are included by default
because they are mirror related configurations.
MM8: Updated the field description to: Topic patterns to remove from
mirror.topics.include or add to mirror.topics.exclude.
MM9: Not needed. KIP updated.
MM10: The config is needed because the listener can have multiple
enabled SASL mechanisms and the listener name alone doesn't select
one. The naming follows the existing
sasl.mechanism.inter.broker.protocol convention, so renaming would be
inconsistent with that precedent.
MM11: Correct. KIP updated.
MM12: No, when mirror.support.unclean.leader.election=true, we’ll wait
for all replicas, not just ISR replicas to complete truncation.
MM13: Correct. KIP updated.
MM14: We missed the “MIRROR_TOPIC_NOT_STOPPED”, so now: start on non
stopped throws MIRROR_TOPIC_NOT_STOPPED. Stop on paused throws
MIRROR_TOPIC_ALREADY_PAUSED. Pause on stopped throws
MIRROR_TOPIC_BEING_STOPPED. Resume on non paused throws
MIRROR_TOPIC_NOT_PAUSED. Stop on stopped throws
TOPIC_NOT_IN_CLUSTER_MIRROR.
On Fri, Jul 3, 2026 at 5:26 PM Mickael Maison <mi...@gmail.com> wrote:
>
> Hi,
>
> Thanks for the KIP!
> This will definitively be a great addition to Kafka.
>
> MM1: In the description of the main components,
> ClusterMirrorCoordinator has this responsibility:
> "Metadata Refresh Scheduling: The coordinator schedules periodic
> metadata refresh operations by invoking a metadata manager every 30
> seconds by default. This ensures that configuration updates and group
> offset commits in the source cluster are continuously propagated to
> the destination cluster. The refresh interval is configurable via
> mirror.metadata.refresh.interval.ms."
> Is that accurate because MirrorMetadataManager has pretty much the
> same responsibility.
>
> MM2: About topic deletions, the KIP states: "the operator would need
> to manually remove the topic from the mirror"
> Is there a way to find topics that have been deleted in the source
> cluster for an operator to do that action?
>
> MM3: About group offsets: It states: "Groups with active members on
> the destination are skipped."
> Should these be logged too?
>
> MM4: In the security control section, the DeleteClusterMirror RPC
> requires Alter on the ClusterMirror resource. Should we use Delete
> instead of Alter?
> On non-cluster resources, we typically use Delete for deletion. See
> https://kafka.apache.org/43/security/authorization-and-acls/#operations-and-resources-on-protocols
>
> MM5: In Destination cluster permissions, I think FindCoordinator
> should also need Describe on the Group resources for regular consumer
> groups.
>
> MM6: In Exactly-Once Semantics, it states: "The mirror fetcher thread
> uses READ_UNCOMMITTED isolation"
> Could this be configurable? If so we can do this in a follow-up KIP
>
> MM7: It seems quotas are not mirrored between the clusters?
> Is that omission intentional?
>
> MM8: In StopMirrorTopicsRequest I'm not sure I understand what the
> Patterns field is?
> Is it a pattern for the topics to stop? If so the description seems odd
>
> MM9: In StopMirrorTopicsResponse/StartMirrorTopicsResponse/PauseMirrorTopicsResponse/ResumeMirrorTopicsResponse
> do we really need the mirror name?
>
> MM10: In Configuration / Broker:
> Can you clarify why we need sasl.mechanism.mirror.admin.protocol?
> Can't we retrieve the protocol from mirror.admin.listener.name or
> inter.broker.listener.name?
> If we really need it, should it be named
> mirror.admin.sasl.mechanism.protocol to match the other names?
>
> MM11: In Configuration / Mirror
> Should it be mirror.acls.include instead of mirror.acl.include to
> match mirror.groups.include and mirror.topics.include?
>
> MM12: Is the description of mirror.support.unclean.leader.election inverted?
>
> MM13: The description of CLUSTER_MIRROR_NOT_EMPTY states: "The cluster
> mirror still has active or non-removed topics"
> Should it be "The cluster mirror still has active or non-stopped topics"?
>
> MM14: We have the MIRROR_TOPIC_BEING_STOPPED error when stopping topic
> already in stopping state? What happens when pausing topics already in
> pausing state?
> Related, I assume when stopping a stopped topic, you get
> TOPIC_NOT_IN_CLUSTER_MIRROR
>
> Thanks,
> Mickael
>
> On Mon, Jun 22, 2026 at 7:47 PM Andrew Schofield <sc...@apache.org> wrote:
> >
> > Hi Fede,
> > Thanks for your replies.
> >
> > AS30: Yes, that's right. Two different end markers for the same transaction, and essentially unbounded duration of the inconsistency since KIP-939. I agree that it's a class of transaction state divergence, like KAFKA-20716. For KIP-1279, the most important thing is that consumers of the mirrored topic must not get confused by the markers. For example, when the Fetch response contains the list of aborted transactions, provided we are sure that it can be built properly without confusion from the extra marker, then we are good. It came to mind because the way this is represented in the protocol for the responses is somewhat fragile I feel, so I'm a bit paranoid that a mis-alignment could break it.
> >
> > Thanks,
> > Andrew
> >
> > On 2026/06/19 12:16:52 Federico Valeri wrote:
> > > Hi Andrew, me, again.
> > >
> > > Given that READ_COMMITTED consumers are unaffected because the ABORT
> > > is already recorded in the transaction index and the dangling COMMIT
> > > produces no CompletedTxn (no index and LSO updates), the practical
> > > impact is limited. However, __transaction_state records the
> > > transaction as COMMITTED while the partition log has it as aborted.
> > > While analyzing this, we discovered that this type of divergence is
> > > not unique to mirroring. KAFKA-20716 documents the same category of
> > > inconsistency after unclean leader elections, where lost transaction
> > > markers cause the LSO to get stuck (much bigger problem). We believe
> > > that solving the transaction state divergence issue would also close
> > > this mirroring scenario. Wdyt?
> > >
> > > On Thu, Jun 18, 2026 at 10:36 PM Federico Valeri <fe...@gmail.com> wrote:
> > > >
> > > > Hi Andrew, about AS30, I had a closer look at KIP-939, and I think
> > > > this is what you mean:
> > > >
> > > > The problem is that a WriteTxnMarker request for a PID unknown to the
> > > > ProducerStateManager (cleared by MIRROR_PID_RESET) does not fail. The
> > > > end transaction marker is still appended, but transaction index and
> > > > LSO are unaffected (ProducerStateManager.prepareUpdate,
> > > > ProducerAppendInfo.appendEndTxnMarker). This may lead to a situation
> > > > where we have two different end txn markers for the same transaction:
> > > > ABORT from the stop mirroring operation, followed by a COMMIT from the
> > > > local Txn Coordinator retrying its WriteTxnMarker after the partition
> > > > becomes writable again.
> > > >
> > > > READ_COMMITTED consumers still see the earlier ABORT in the
> > > > transaction index and skip the aborted data. The visible outcome for
> > > > consumers is correct. However, __transaction_state on the original
> > > > source cluster now records the transaction as COMMITTED while the
> > > > partition log has it as aborted, creating a silent inconsistency that
> > > > tools querying transaction state will not detect. The window for this
> > > > race is bounded by transaction.max.timeout.ms (default 15 minutes),
> > > > but becomes unbounded with KIP-939 because it allows prepared
> > > > transactions to sit indefinitely.
> > > >
> > > > We could move the coordinator state to “complete abort” at the cost of
> > > > an RPC. This would prevent any retry, but that would break the 2PC
> > > > contract (Kafka must hold the prepared state until the external
> > > > coordinator decides). Also, before stop there would be a number of
> > > > retries constantly going on. We definitely need a better solution.
> > > >
> > > > On Thu, Jun 18, 2026 at 11:36 AM Federico Valeri <fe...@gmail.com> wrote:
> > > > >
> > > > > Hi Andrew, thanks for your careful review.
> > > > >
> > > > > AS28: We reworked that section and added some diagrams to illustrate
> > > > > problem and solutions. Let us know if it is better.
> > > > >
> > > > > AS30: We updated that table, but we are not sure about the problem you
> > > > > mention. Can you clarify and send us the sequence of actions that
> > > > > would lead to that problem?
> > > > >
> > > > > AS32: You are right. Replaced StartMirrorTopicsRequestData.TopicData
> > > > > with a Controller.MirrorTopicMetadata record. Also renamed TopicData
> > > > > to TopicMetadata in all schemas.
> > > > >
> > > > > AS33: Good idea. Added per-topic error messages in the RPC responses
> > > > > you mentioned.
> > > > >
> > > > > AS29, AS31, AS34 (both of them), AS35, AS36: These small issues are
> > > > > all fixed now.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jun 9, 2026 at 9:58 PM Andrew Schofield <sc...@apache.org> wrote:
> > > > > >
> > > > > > Hi Fede and friends,
> > > > > > This KIP is shaping up nicely. The change to the way that state management for the mirrors is done is much more elegant. The removal of the PID mapping is also a nice improvement.
> > > > > >
> > > > > > Inevitably, I have some more comments.
> > > > > >
> > > > > > AS28: I understand KAFKA-18723 and how leader epochs work in that case, but the enhancements that you made to the Fetch RPC and how they work escapes me. Please could we have some diagrams to illustrate the various epochs?
> > > > > >
> > > > > > AS29: I like the addition of share group offset syncing. You need AlterShareGroupOffsets and DescribeShareGroupOffsets in the tables of RPCs and permissions.
> > > > > >
> > > > > > AS30: It is certainly true that EOS is not supported. Using cluster mirroring on transactional data does risk breaking transactional atomicity for transactions in flight at the time that mirroring was stopped. That's a known limitation, and it's an effect of per-topic async replication. Writing ABORT markers when stop mirroring is triggered is interesting. I just wonder whether it's truly safe.
> > > > > >
> > > > > > Could you enhance the tables of records in the Exactly-Once Semantics section to include leader epochs? I think that the leader epoch for the ABORT markers will have been bumped as the mirroring stopped and the partitions became writable. However, I worry that someone might switch the mirroring back in the other direction before the transaction which was rudely partially aborted has completed on the original source cluster, and that the effect of the transaction coordinator completing the transaction, potentially writing duplicate transaction markers after the ABORT markers which will have been replicated back again.
> > > > > >
> > > > > > This is probably only a realistic possibility when KIP-939 is present, but it is accepted and I'm sure it will land soon enough. I would say this KIP should be prepared for the existence of long-running transactions so we don't dig a hole to fall into later.
> > > > > >
> > > > > > AS31: In the section on the Admin Client, the description for Admin.createClusterMirror(String, Map, CreateClusterMirrorOptions) is duplicated.
> > > > > >
> > > > > > AS32: The use of StartMirrorTopicsRequestData.TopicData in an external interface looks odd.
> > > > > >
> > > > > > AS33: I suggest per-topic error messages in the RPC responses such as PauseMirrorTopics so you can pass back more context, such as the state if the request failed due to a state mismatch.
> > > > > >
> > > > > > AS34: There is an unnamed field in DescribeClusterMirrorsResponse. The same in LastMirrorEpochsValue.
> > > > > >
> > > > > > AS34: Some of the fields in BumpLeaderEpochsRequest start with lower-case letters. Also, there are quite a few fields in the RPCs in general that should be version "0+" which are "0".
> > > > > >
> > > > > > AS35: Sometimes the partition states in the RPCs are string, but sometimes int8.
> > > > > >
> > > > > > AS36: The base level for share group support is probably 4.1. All of the RPCs were present in 4.1, although they were only enabled by default in 4.2.
> > > > > >
> > > > > > Thanks,
> > > > > > Andrew
> > > > > >
> > > > > > On 2026/06/04 11:01:36 Federico Valeri wrote:
> > > > > > > Hello, additional updates and replies:
> > > > > > >
> > > > > > > VK12: Consumer data loss risk is a good point. We updated the "Group
> > > > > > > Offsets" paragraph.
> > > > > > >
> > > > > > > Some of you raised the point that using mirror.name suffixes for pause
> > > > > > > and stop operations is a bit inelegant and we agreed. Now we propose a
> > > > > > > better approach:
> > > > > > >
> > > > > > > We replaced the topic config based approach (mirror.name and
> > > > > > > .stopped/.paused suffix conventions) with a first class metadata
> > > > > > > record for tracking mirror topic state changes. A new
> > > > > > > MirrorTopicStateChangeRecord is added to the metadata log with three
> > > > > > > fields: TopicId (uuid), MirrorName (string), and DesiredState (int8,
> > > > > > > where 0 = start mirroring, 1 = stop, 2 = pause). When a user calls
> > > > > > > startMirrorTopics, stopMirrorTopics, or pauseMirrorTopics, the
> > > > > > > controller writes this record to the metadata log instead of
> > > > > > > manipulating config suffixes. Brokers receive the record via metadata
> > > > > > > updates and react accordingly, triggering partition level state
> > > > > > > transitions through the existing MirrorPartitionState state machine.
> > > > > > > More details in the updated KIP.
> > > > > > >
> > > > > > > Let us know what you think.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Fede
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 2, 2026 at 8:50 AM Luke Chen <sh...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi Viquar,
> > > > > > > >
> > > > > > > > Thanks for the comment.
> > > > > > > >
> > > > > > > > Regarding VK12, thanks for raising the issue.
> > > > > > > > We didn't think about that.
> > > > > > > > I agree that compared with data re-processing, it's worse to have data loss.
> > > > > > > > The proposed formula makes sense to me.
> > > > > > > > *syncedOffset = max(destinationLogStartOffset, min(sourceCommitted,
> > > > > > > > destinationLEO))*
> > > > > > > >
> > > > > > > > Let us have some discussion internally and then update the KIP and reply to
> > > > > > > > the thread.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Luke
> > > > > > > >
> > > > > > > > On Wed, May 20, 2026 at 2:39 AM Federico Valeri <fe...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Rajini,
> > > > > > > > >
> > > > > > > > > RS14: Done. Missed that, sorry.
> > > > > > > > >
> > > > > > > > > Thanks again.
> > > > > > > > >
> > > > > > > > > On Tue, May 19, 2026 at 7:28 PM Rajini Sivaram <ra...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Federico,
> > > > > > > > > >
> > > > > > > > > > Thanks for the updates. Just one minor point, apart from that, looks
> > > > > > > > > good.
> > > > > > > > > >
> > > > > > > > > > RS14: KIP still shows "bin/kafka-configs.sh --bootstrap-server :9094
> > > > > > > > > > --entity-type mirrors".
> > > > > > > > > > Will be good to change that to `--entity-type cluster-mirrors`.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, May 19, 2026 at 5:28 PM vaquar khan <va...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Federico and team,
> > > > > > > > > > >
> > > > > > > > > > > Thank you for your detailed response on May 11, 2026. I greatly
> > > > > > > > > appreciate
> > > > > > > > > > > the collaborative effort over the past few months to harden the
> > > > > > > > > KIP-1279
> > > > > > > > > > > architecture.
> > > > > > > > > > >
> > > > > > > > > > > * 1. Resolved Architectural & Stability Vulnerabilities*
> > > > > > > > > > >
> > > > > > > > > > > I am pleased to confirm that the KIP has successfully integrated fixes
> > > > > > > > > for
> > > > > > > > > > > the critical vulnerabilities I flagged, thereby protecting the
> > > > > > > > > cluster's
> > > > > > > > > > > state machine, memory bounds, and control plane. Consider the following
> > > > > > > > > > > items fully resolved on my end:
> > > > > > > > > > >
> > > > > > > > > > > -
> > > > > > > > > > >
> > > > > > > > > > > *VK4 / VK8 (Negative PID Bug & State Machine Failure):* I previously
> > > > > > > > > > > identified that the proposed -(sourceProducerId + 2) PID rewriting
> > > > > > > > > > > formula would fundamentally break hasProducerId() in
> > > > > > > > > > > AbstractRecordBatch.java, causing the broker to bypass
> > > > > > > > > > > ProducerStateManager.update() and default the Last Stable Offset
> > > > > > > > > (LSO)
> > > > > > > > > > > to the High Watermark. I am glad to see the KIP authors abandoned
> > > > > > > > > PID
> > > > > > > > > > > mapping entirely and adopted the deterministic MIRROR_PID_RESET
> > > > > > > > > control
> > > > > > > > > > > record barrier. This perfectly protects Kafka's exactly-once
> > > > > > > > > semantics.
> > > > > > > > > > >
> > > > > > > > > > > -
> > > > > > > > > > >
> > > > > > > > > > > *VK11 (TransactionIndex Rebuild Ambiguity):* Thank you for
> > > > > > > > > officially
> > > > > > > > > > > confirming that the TransactionIndex is strictly rebuilt locally
> > > > > > > > > during
> > > > > > > > > > > log appends rather than copied byte-for-byte. This resolves my
> > > > > > > > > concern
> > > > > > > > > > > regarding PID mismatches inducing consumer read anomalies.
> > > > > > > > > > > -
> > > > > > > > > > >
> > > > > > > > > > > *VK1 (Thundering Herd / OOM Heap Allocation):* Your clarification
> > > > > > > > > that
> > > > > > > > > > > fetcher threads multiplex partitions meaning the memory footprint is
> > > > > > > > > > > strictly bounded by num_fetcher_threads * response_max_bytes rather
> > > > > > > > > than
> > > > > > > > > > > a concurrent 1MB buffer per partition fully resolves my concern
> > > > > > > > > > > regarding
> > > > > > > > > > > 50GB broker-wide OOM spikes during mass partition wake-ups.
> > > > > > > > > > > -
> > > > > > > > > > >
> > > > > > > > > > > *VK3 (Control Plane Hotspots):* I had flagged the severe risk of
> > > > > > > > > > > metadata saturation on a single broker during a "link flap" event.
> > > > > > > > > Your
> > > > > > > > > > > confirmation that the __mirror_state topic utilizes a compound hash
> > > > > > > > > > > of (mirrorName,
> > > > > > > > > > > topicId, partition number) mathematically ensures that the 50,000+
> > > > > > > > > state
> > > > > > > > > > > transitions will be safely distributed across the cluster,
> > > > > > > > > neutralizing
> > > > > > > > > > > the
> > > > > > > > > > > single-node hotspot risk.
> > > > > > > > > > >
> > > > > > > > > > > 2. Outstanding Critical Blocker: VK12 (Offset Sync Data Loss)
> > > > > > > > > > >
> > > > > > > > > > > Regarding VK12, there is a fundamental misunderstanding in your
> > > > > > > > > previous
> > > > > > > > > > > reply. You stated: *"The scenario described can only occur if offsets
> > > > > > > > > are
> > > > > > > > > > > force-written to an active group, which the design prevents."*
> > > > > > > > > > >
> > > > > > > > > > > My concern has absolutely nothing to do with overwriting active
> > > > > > > > > groups. My
> > > > > > > > > > > concern applies strictly to the normal synchronization of inactive/dead
> > > > > > > > > > > groups, and is based directly on the race condition currently
> > > > > > > > > documented in
> > > > > > > > > > > the official KIP-1279 text:
> > > > > > > > > > >
> > > > > > > > > > > *"During offset synchronization, the committed offset in the
> > > > > > > > > destination
> > > > > > > > > > > cluster may temporarily exceed the current log end offset (LEO) of the
> > > > > > > > > > > mirror topic... consumers attempting to resume from offset 100 will
> > > > > > > > > receive
> > > > > > > > > > > an OffsetOutOfRangeException. To handle this gracefully, consumers
> > > > > > > > > should
> > > > > > > > > > > configure auto.offset.reset=latest..."*
> > > > > > > > > > >
> > > > > > > > > > > If a failover happens during this documented divergence window, a
> > > > > > > > > > > reconnecting consumer will hit the OffsetOutOfRangeException. If the
> > > > > > > > > > > downstream consumer follows the KIP's official advice and relies on
> > > > > > > > > > > auto.offset.reset=latest, the consumer will jump to the absolute newest
> > > > > > > > > > > offset on the partition. *This completely skips any newly produced
> > > > > > > > > records
> > > > > > > > > > > that arrived between the failover and the consumer reconnecting,
> > > > > > > > > resulting
> > > > > > > > > > > in silent, unrecoverable data loss for the downstream application.*
> > > > > > > > > > > *Proposed Resolution: Double-Clamped Offset Safety Invariant* Instead
> > > > > > > > > of
> > > > > > > > > > > requiring consumers to use auto.offset.reset=latest and endorsing a
> > > > > > > > > known
> > > > > > > > > > > data loss vector, I propose that the ClusterMirrorCoordinator enforce a
> > > > > > > > > > > pre-persist offset validation invariant during offset synchronization.
> > > > > > > > > > > Before any translated offset is committed to the destination cluster,
> > > > > > > > > the
> > > > > > > > > > > coordinator must apply a double-clamp:
> > > > > > > > > > >
> > > > > > > > > > > *syncedOffset = max(destinationLSO, min(sourceCommitted,
> > > > > > > > > destinationLEO))*
> > > > > > > > > > >
> > > > > > > > > > > Where destinationLSO is the Log Start Offset (earliest readable
> > > > > > > > > position
> > > > > > > > > > > post-retention) and destinationLEO is the Log End Offset (latest
> > > > > > > > > replicated
> > > > > > > > > > > position) of the destination partition.
> > > > > > > > > > >
> > > > > > > > > > > This guarantees that every persisted offset falls within the physically
> > > > > > > > > > > valid range eliminating both the OffsetOutOfRangeException caused by
> > > > > > > > > > > exceeding the LEO during replication lag, and the expired-offset rewind
> > > > > > > > > > > caused by falling below the LSO due to destination retention policies.
> > > > > > > > > > >
> > > > > > > > > > > The worst-case trade-off under this invariant is bounded re-processing
> > > > > > > > > > > proportional to the replication lag at failover time not total
> > > > > > > > > partition
> > > > > > > > > > > depth which is perfectly consistent with Kafka's documented
> > > > > > > > > at-least-once
> > > > > > > > > > > delivery guarantees. Broad enterprise production evidence from
> > > > > > > > > large-scale
> > > > > > > > > > > cross-cluster failovers confirms that state checks alone (preventing
> > > > > > > > > active
> > > > > > > > > > > group overwrites) are insufficient; strict offset bounds clamping is
> > > > > > > > > > > required to achieve enterprise-grade data integrity.
> > > > > > > > > > >
> > > > > > > > > > > I look forward to your thoughts on implementing this final offset
> > > > > > > > > capping
> > > > > > > > > > > logic. Once VK12 is patched, I believe this architecture will be
> > > > > > > > > > > exceptionally robust and ready for enterprise deployment.
> > > > > > > > > > >
> > > > > > > > > > > Best Regards,
> > > > > > > > > > >
> > > > > > > > > > > Viquar Khan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, 18 May 2026 at 14:23, Rajini Sivaram <ra...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Federico,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the updates! The KIP is looking good. A few more small
> > > > > > > > > > > comments.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > RS13: A couple of places still refer to `kafka-mirror.sh` like under
> > > > > > > > > > > > `Failover Process`. Can we change them to
> > > > > > > > > `*kafka-cluster-mirrors.sh*`?
> > > > > > > > > > > >
> > > > > > > > > > > > RS14: Should we change `--entity-type mirrors` for kafka-configs to
> > > > > > > > > be `
> > > > > > > > > > > > --entity-type *cluster-mirrors*` to be consistent? Also,
> > > > > > > > > > > > CLUSTER_MIRROR((byte)
> > > > > > > > > > > > 64, "mirror"); could be `*cluster-mirror*`?
> > > > > > > > > > > >
> > > > > > > > > > > > RS15: It may be useful to rename `mirror.topic.num.partitions` and `
> > > > > > > > > > > > mirror.topic.replication.factor` since they are very similar to `
> > > > > > > > > > > > mirror.topic.properties.exclude`, but the `mirror.topic` prefix
> > > > > > > > > refers to
> > > > > > > > > > > > different topics (the internal topic for the first two and actual
> > > > > > > > > mirror
> > > > > > > > > > > > topics for the other one).
> > > > > > > > > > > >
> > > > > > > > > > > > RS16: ACL Sync: KIP says "Deletes ACLs that exist in destination but
> > > > > > > > > not
> > > > > > > > > > > in
> > > > > > > > > > > > source using DeleteAcls request."
> > > > > > > > > > > > What happens if someone creates an ACL on the destination to deny
> > > > > > > > > > > > User:Alice access to all topics?
> > > > > > > > > > > >
> > > > > > > > > > > > 1. If that ACL also existed on the source cluster and then it was
> > > > > > > > > > > > removed, will it get removed from the destination?
> > > > > > > > > > > > 2. If that ACL never existed on the source cluster, will it get
> > > > > > > > > > > removed
> > > > > > > > > > > > from the destination?
> > > > > > > > > > > >
> > > > > > > > > > > > RS17: The table in the Source ACLs section says:
> > > > > > > > > "DescribeClusterMirrors
> > > > > > > > > > > > MC Read Cluster Log truncation"
> > > > > > > > > > > > Should that be `ClusterMirror:Read` instead of `Cluster:Read`?
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Rajini
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, May 15, 2026 at 8:41 AM Federico Valeri <
> > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Rajini, we finally addressed the API and tool naming
> > > > > > > > > refactoring as
> > > > > > > > > > > > > you suggested in RS6. Please take a look when you have time.
> > > > > > > > > Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, May 11, 2026 at 6:17 PM Federico Valeri <
> > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hello all, I want to highlight a couple of new paragraphs:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Leader Epoch Invariant: Cluster mirroring enforces the
> > > > > > > > > invariant
> > > > > > > > > > > > > > that the destination leader epoch must always be greater than or
> > > > > > > > > > > equal
> > > > > > > > > > > > > > to the source leader epoch (DLE>=SLE). Without this, consumers
> > > > > > > > > on the
> > > > > > > > > > > > > > destination cluster can get stuck in an infinite metadata refresh
> > > > > > > > > > > loop
> > > > > > > > > > > > > > when they encounter committed offsets carrying source epochs
> > > > > > > > > higher
> > > > > > > > > > > > > > than the local epoch. The invariant is maintained through three
> > > > > > > > > > > > > > mechanisms: reactive bumping (epoch fencing triggered when SLE >
> > > > > > > > > DLE
> > > > > > > > > > > > > > during fetch), proactive bumping (scheduled when SLE approaches
> > > > > > > > > DLE
> > > > > > > > > > > > > > within a threshold), and periodic bumping (checked during
> > > > > > > > > coordinator
> > > > > > > > > > > > > > metadata sync).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973#KIP1279:ClusterMirroring-LeaderEpochInvariant
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. Group Offsets: The coordinator periodically syncs consumer and
> > > > > > > > > > > > > > share group offsets from the source cluster to the destination
> > > > > > > > > for
> > > > > > > > > > > all
> > > > > > > > > > > > > > mirrored topics. Groups are filtered by configurable
> > > > > > > > > include/exclude
> > > > > > > > > > > > > > patterns, and offsets are only synced for groups that are not
> > > > > > > > > > > > > > currently active on the destination cluster, preventing
> > > > > > > > > overwrites of
> > > > > > > > > > > > > > local consumer progress. Because source and destination share the
> > > > > > > > > > > same
> > > > > > > > > > > > > > topic offsets (no offset translation), synced offsets can be used
> > > > > > > > > > > > > > directly without mapping.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406620973#KIP1279:ClusterMirroring-GroupOffsets
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > These new paragraphs directly address some of your questions,
> > > > > > > > > but let
> > > > > > > > > > > > > > me list them here:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > JR2: Yes, we removed the incorrect phrase and added more details
> > > > > > > > > to
> > > > > > > > > > > > > > the paragraph.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > JR4: When source cluster topic has tiered storage enabled, CM
> > > > > > > > > works
> > > > > > > > > > > by
> > > > > > > > > > > > > > mirroring remote and local log into destination cluster. When
> > > > > > > > > > > > > > destination cluster topic has tiered storage enabled, CM fails in
> > > > > > > > > > > > > > PREPARING state because the LME may be in remote storage, but
> > > > > > > > > works
> > > > > > > > > > > > > > fine if already MIRRORING because no truncation is needed.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > JR11: See "Leader Epoch Invariant" paragraph mentioned above.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > JR13: We can't support stateful Streams application because
> > > > > > > > > > > > > > asynchronous replication cannot preserve the transactional
> > > > > > > > > boundaries
> > > > > > > > > > > > > > between input offset commits, state store mutations written to
> > > > > > > > > > > > > > changelog topics, and intermediate records written to repartition
> > > > > > > > > > > > > > topics. The synchronous extension of this design will be able to
> > > > > > > > > > > > > > support them. Existing Features Integration paragraph updated.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > JR18: See "Group Offsets" paragraph mentioned above.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > IY1: See "Group Offsets" paragraph mentioned above.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > Fede
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, May 11, 2026 at 6:08 PM Federico Valeri <
> > > > > > > > > > > fedevaleri@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Vaquar,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > VK4/VK8: We don't do PID mapping anymore. The KIP was updated
> > > > > > > > > some
> > > > > > > > > > > > > > > time ago with the new approach based on the new PID reset
> > > > > > > > > control
> > > > > > > > > > > > > > > record.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > VK11: The transaction index is always built locally during log
> > > > > > > > > > > > append,
> > > > > > > > > > > > > > > never copied.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > VK1: The 50,000 * 1MB = 50GB calculation misunderstands the
> > > > > > > > > fetch
> > > > > > > > > > > > > > > model. Fetcher threads don't allocate one buffer per partition.
> > > > > > > > > > > > Actual
> > > > > > > > > > > > > > > peak memory is roughly num_fetcher_threads *
> > > > > > > > > response_max_bytes,
> > > > > > > > > > > not
> > > > > > > > > > > > > > > num_partitions * partition_max_bytes. With 1 fetcher thread
> > > > > > > > > and the
> > > > > > > > > > > > > > > default response max, the memory footprint is modest
> > > > > > > > > regardless of
> > > > > > > > > > > > > > > partition count. We are leveraging the same proven pattern
> > > > > > > > > used by
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > internal replication.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > VK3: The __mirror_state topic uses hash-based partitioning
> > > > > > > > > based on
> > > > > > > > > > > > > > > mirrorName, topicId and partition number. With the production
> > > > > > > > > > > default
> > > > > > > > > > > > > > > of 50 partitions, 50,000 partition transitions distribute
> > > > > > > > > across
> > > > > > > > > > > ~50
> > > > > > > > > > > > > > > partition leaders on different brokers, not a single broker.
> > > > > > > > > This
> > > > > > > > > > > is
> > > > > > > > > > > > > > > the same proven pattern as __consumer_offsets, which handles
> > > > > > > > > > > millions
> > > > > > > > > > > > > > > of commits.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > VK12: The scenario described can only occur if offsets are
> > > > > > > > > > > > > > > force-written to an active group, which the design prevents.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > > > Fede
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > >