You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Jayteeps (via GitHub)" <gi...@apache.org> on 2026/10/27 10:32:55 UTC

[PR] KAFKA-20186: [WIP] Add mirror coordinator record schemas and serde [kafka]

Jayteeps opened a new pull request, #23979:
URL: https://github.com/apache/kafka/pull/23979

   Add the version 0 `MirrorPartitionState` and `LastMirrorEpochs` key/value schemas for `__mirror_state`, as specified by [KIP-1279](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406620973/KIP-1279+Cluster+Mirroring).
   
   Generate the message classes, record type dispatch, and JSON converters in `mirror-coordinator`. Add `MirrorCoordinatorRecordSerde` using Kafka's shared coordinator record format, together with the `mirrorName` generator entity type. Unit tests pin the binary layout and cover defaults, tombstones, unsupported types and versions, truncated input, and unknown tagged fields. Integration tests verify that records and tombstones remain readable after a real Kafka broker restarts.
   
   This is a contribution toward [KAFKA-20186](https://issues.apache.org/jira/browse/KAFKA-20186). This original implementation builds on Kafka's existing coordinator serde infrastructure and is contributed under the Apache License, Version 2.0.
   
   Generated-by: OpenAI Codex (GPT-6)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20186: [WIP] Add mirror coordinator record schemas and serde [kafka]

Posted by "Jayteeps (via GitHub)" <gi...@apache.org>.
Jayteeps commented on PR #23979:
URL: https://github.com/apache/kafka/pull/23979#issuecomment-6439631563

   This PR is a draft while I review the contribution.
   
   Validation passed locally on Java 17:
   
   - `:mirror-coordinator:test`: 26 test cases passed.
   - `:generator:test`: all 166 tests passed.
   - `:server:integrationTest --tests org.apache.kafka.server.mirror.MirrorCoordinatorRecordSerdeIntegrationTest`: both real-broker restart tests passed, including tombstones for both record types.
   - `spotlessApply` followed by `spotlessCheck`, `checkstyleMain`, and `checkstyleTest` for the mirror coordinator, generator, and server modules.
   - Apache RAT license audit and `git diff --check`.
   
   The integration tests use a compacted `__mirror_state` topic and exercise the serialized record format through actual broker storage and consumer reads across a broker restart. Coordinator lifecycle and cross-cluster replication behavior are covered by future KIP-1279 work.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21379: Add mirror coordinator record schemas and serde [kafka]

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #23979:
URL: https://github.com/apache/kafka/pull/23979#discussion_r4415414399


##########
mirror-coordinator/src/main/resources/common/message/LastMirrorEpochsValue.json:
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+{
+  "apiKey": 1,
+  "type": "coordinator-value",
+  "name": "LastMirrorEpochsValue",
+  "validVersions": "0",
+  "flexibleVersions": "0+",
+  "fields": [
+    { "name": "LastMirrorEpoch", "type": "int32", "versions": "0+",
+      "about": "The last mirror leader epoch for this partition." },

Review Comment:
   I agree. We forgot to add the default=-1 to represent an uninitialized epoch. Epoch -1 with offset -1 actually means replicate from scratch.



##########
mirror-coordinator/src/main/resources/common/message/LastMirrorEpochsValue.json:
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+{
+  "apiKey": 1,
+  "type": "coordinator-value",
+  "name": "LastMirrorEpochsValue",
+  "validVersions": "0",
+  "flexibleVersions": "0+",
+  "fields": [
+    { "name": "LastMirrorEpoch", "type": "int32", "versions": "0+",
+      "about": "The last mirror leader epoch for this partition." },

Review Comment:
   I agree. We forgot to add the default=-1 to represent an uninitialized epoch. Epoch -1 with offset -1 means replicate from scratch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21379: Add mirror coordinator record schemas and serde [kafka]

Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov merged PR #23979:
URL: https://github.com/apache/kafka/pull/23979


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21379: Add mirror coordinator record schemas and serde [kafka]

Posted by "Jayteeps (via GitHub)" <gi...@apache.org>.
Jayteeps commented on PR #23979:
URL: https://github.com/apache/kafka/pull/23979#issuecomment-6518175853

   > @Jayteeps thanks for your interest in Cluster Mirroring.
   > 
   > Related to this patch, it would be good to add the `--mirror-state-decoder` flag and logic to the log dump tool, and create a `ClusterMirrorStateMessageFormatter` class. They are also [mentioned in the KIP](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406620973/KIP-1279+Cluster+Mirroring#KIP1279%3AClusterMirroring-Debugging). If you are keen to work on them, I can log dedicated Jiras.
   
   Thanks @fvaleri , yes please create the JIRA's and assign to me. Ty!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21379: Add mirror coordinator record schemas and serde [kafka]

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #23979:
URL: https://github.com/apache/kafka/pull/23979#discussion_r4415414399


##########
mirror-coordinator/src/main/resources/common/message/LastMirrorEpochsValue.json:
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+{
+  "apiKey": 1,
+  "type": "coordinator-value",
+  "name": "LastMirrorEpochsValue",
+  "validVersions": "0",
+  "flexibleVersions": "0+",
+  "fields": [
+    { "name": "LastMirrorEpoch", "type": "int32", "versions": "0+",
+      "about": "The last mirror leader epoch for this partition." },

Review Comment:
   I agree. We forgot to add the default=-1 to represent an uninitialized epoch. Epoch -1 and offset -1 actually means replicate from scratch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21379: Add mirror coordinator record schemas and serde [kafka]

Posted by "clolov (via GitHub)" <gi...@apache.org>.
clolov commented on code in PR #23979:
URL: https://github.com/apache/kafka/pull/23979#discussion_r4412885012


##########
mirror-coordinator/src/main/resources/common/message/LastMirrorEpochsValue.json:
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+{
+  "apiKey": 1,
+  "type": "coordinator-value",
+  "name": "LastMirrorEpochsValue",
+  "validVersions": "0",
+  "flexibleVersions": "0+",
+  "fields": [
+    { "name": "LastMirrorEpoch", "type": "int32", "versions": "0+",
+      "about": "The last mirror leader epoch for this partition." },

Review Comment:
   I have a suspicion that the default value here ought to be -1 rather than 0. @showuon what is your opinion on this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20186: [WIP] Add mirror coordinator record schemas and serde [kafka]

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #23979:
URL: https://github.com/apache/kafka/pull/23979#discussion_r4396661530


##########
mirror-coordinator/src/main/resources/common/message/MirrorPartitionStateKey.json:
##########
@@ -0,0 +1,30 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+{
+  "apiKey": 1,

Review Comment:
   Let's try to start the apiKey with 0, not 1. I know KIP is written with it, we should update it.



##########
mirror-coordinator/src/test/java/org/apache/kafka/coordinator/mirror/MirrorCoordinatorRecordSerdeTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.mirror;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.protocol.types.RawTaggedField;
+import org.apache.kafka.coordinator.common.runtime.CoordinatorRecord;
+import org.apache.kafka.coordinator.common.runtime.Deserializer;
+import org.apache.kafka.coordinator.mirror.generated.LastMirrorEpochsKey;
+import org.apache.kafka.coordinator.mirror.generated.LastMirrorEpochsValue;
+import org.apache.kafka.coordinator.mirror.generated.MirrorPartitionStateKey;
+import org.apache.kafka.coordinator.mirror.generated.MirrorPartitionStateValue;
+import org.apache.kafka.server.common.ApiMessageAndVersion;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.stream.Stream;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+class MirrorCoordinatorRecordSerdeTest {
+    private static final Uuid TOPIC_ID = new Uuid(1L, 2L);
+    private final MirrorCoordinatorRecordSerde serde = new MirrorCoordinatorRecordSerde();
+
+    @Test
+    void testMirrorPartitionStateWireFormat() {
+        CoordinatorRecord record = stateRecord("error");
+        // Version 0, fixed-width fields, compact nullable error string, and no tagged fields.
+        byte[] value = ByteBuffer.allocate(21)
+            .putShort((short) 0)
+            .put((byte) 8)
+            .putInt(42)
+            .putInt(7)
+            .put((byte) 6)
+            .put("error".getBytes(UTF_8))
+            .putShort((short) 3)
+            .put((byte) 3)
+            .put((byte) 0)
+            .array();

Review Comment:
   Could you add comments for each put to explain what value it means? Same comments apply to below. Otherwise, it's unreadable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21379: Add mirror coordinator record schemas and serde [kafka]

Posted by "Jayteeps (via GitHub)" <gi...@apache.org>.
Jayteeps commented on code in PR #23979:
URL: https://github.com/apache/kafka/pull/23979#discussion_r4421681087


##########
mirror-coordinator/src/main/resources/common/message/LastMirrorEpochsValue.json:
##########
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+{
+  "apiKey": 1,
+  "type": "coordinator-value",
+  "name": "LastMirrorEpochsValue",
+  "validVersions": "0",
+  "flexibleVersions": "0+",
+  "fields": [
+    { "name": "LastMirrorEpoch", "type": "int32", "versions": "0+",
+      "about": "The last mirror leader epoch for this partition." },

Review Comment:
   Good spot! Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org