You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "smjn (via GitHub)" <gi...@apache.org> on 2024/11/26 09:13:09 UTC
[PR] KAFKA-18057: Add share partition redundant offset manager. [kafka]
smjn opened a new pull request, #17943:
URL: https://github.com/apache/kafka/pull/17943
* The share coordinator writes state to the internal topic `__share_group_state`. This topic is not compacted.
* To prevent the size of partitions of the topic to become unwieldy, we would want to prune this topic periodically.
* Since the disk log can only supports prefix based delete, we require the knowledge of the last redundant offset below which the partition can be truncated. This PR introduces a class which manages the state to calculate the said offset.
* Tests have also been added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18057: Add share partition redundant offset manager. [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn closed pull request #17943: KAFKA-18057: Add share partition redundant offset manager.
URL: https://github.com/apache/kafka/pull/17943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18057: Add share partition redundant offset manager. [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17943:
URL: https://github.com/apache/kafka/pull/17943#discussion_r1858475955
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class SharePartitionOffsetManager {
+
+ // map to store share partition key => current partition offset
+ // being written
+ private final TimelineHashMap<SharePartitionKey, Long> offsets;
+
+ // minimum offset representing the smallest necessary offset (non-redundant)
+ // across the internal partition
+ private long minOffset = Long.MAX_VALUE;
+
+ public SharePartitionOffsetManager(SnapshotRegistry snapshotRegistry) {
+ Objects.requireNonNull(snapshotRegistry);
+ offsets = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Method updates internal state with the supplied offset for the provided
+ * share partition key. It then calculates the minimum offset, if possible,
+ * below which all offsets are redundant. This value is then returned as
+ * an optional.
+ * <p>
+ * The value returned is exclusive, in that all offsets below it but non including
+ * it are redundant.
+ *
+ * @param key - represents {@link SharePartitionKey} whose offset needs updating
+ * @param offset - represents the latest partition offset for provided key
+ * @return Optional of last redundant offset, exclusive.
+ */
+ public Optional<Long> updateState(SharePartitionKey key, long offset) {
+ minOffset = Math.min(minOffset, offset);
+ offsets.put(key, offset);
+
+ Optional<Long> deleteTillOffset = lastRedundantOffset();
+ deleteTillOffset.ifPresent(off -> minOffset = off);
+ return deleteTillOffset;
+ }
+
+ private Optional<Long> lastRedundantOffset() {
+ long soFar = Long.MAX_VALUE;
Review Comment:
I will add but having no offsets in map is not possible in the current state as this is a private method which is called by `updateState` which will for sure have some value in the map before making a call to this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18057: Add share partition redundant offset manager. [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17943:
URL: https://github.com/apache/kafka/pull/17943#discussion_r1858294427
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class SharePartitionOffsetManager {
+
+ // map to store share partition key => current partition offset
+ // being written
+ private final TimelineHashMap<SharePartitionKey, Long> offsets;
+
+ // minimum offset representing the smallest necessary offset (non-redundant)
+ // across the internal partition
+ private long minOffset = Long.MAX_VALUE;
+
+ public SharePartitionOffsetManager(SnapshotRegistry snapshotRegistry) {
+ Objects.requireNonNull(snapshotRegistry);
+ offsets = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Method updates internal state with the supplied offset for the provided
+ * share partition key. It then calculates the minimum offset, if possible,
+ * below which all offsets are redundant. This value is then returned as
+ * an optional.
+ * <p>
+ * The value returned is exclusive, in that all offsets below it but non including
Review Comment:
nit: "not including"
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class SharePartitionOffsetManager {
Review Comment:
This needs a class-level comment saying what its purpose is.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class SharePartitionOffsetManager {
+
+ // map to store share partition key => current partition offset
+ // being written
+ private final TimelineHashMap<SharePartitionKey, Long> offsets;
+
+ // minimum offset representing the smallest necessary offset (non-redundant)
+ // across the internal partition
+ private long minOffset = Long.MAX_VALUE;
+
+ public SharePartitionOffsetManager(SnapshotRegistry snapshotRegistry) {
+ Objects.requireNonNull(snapshotRegistry);
+ offsets = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Method updates internal state with the supplied offset for the provided
+ * share partition key. It then calculates the minimum offset, if possible,
+ * below which all offsets are redundant. This value is then returned as
+ * an optional.
+ * <p>
+ * The value returned is exclusive, in that all offsets below it but non including
+ * it are redundant.
+ *
+ * @param key - represents {@link SharePartitionKey} whose offset needs updating
+ * @param offset - represents the latest partition offset for provided key
+ * @return Optional of last redundant offset, exclusive.
+ */
+ public Optional<Long> updateState(SharePartitionKey key, long offset) {
+ minOffset = Math.min(minOffset, offset);
+ offsets.put(key, offset);
+
+ Optional<Long> deleteTillOffset = lastRedundantOffset();
+ deleteTillOffset.ifPresent(off -> minOffset = off);
+ return deleteTillOffset;
+ }
+
+ private Optional<Long> lastRedundantOffset() {
+ long soFar = Long.MAX_VALUE;
+
+ for (long offset : offsets.values()) {
+ // get min offset among latest offsets
+ // for all share keys in the internal partition
+ soFar = Math.min(soFar, offset);
+
+ // minOffset represents the smallest necessary offset
+ // and if soFar equals it, we cannot proceed. This can happen
+ // if a share partition key hasn't had records written for a while
+ // For example,
+ // <p>
+ // key1:1
+ // key2:2 4 6
+ // key2:3 5 7
Review Comment:
nit: Should this line be "key3"?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class SharePartitionOffsetManager {
+
+ // map to store share partition key => current partition offset
+ // being written
+ private final TimelineHashMap<SharePartitionKey, Long> offsets;
+
+ // minimum offset representing the smallest necessary offset (non-redundant)
+ // across the internal partition
+ private long minOffset = Long.MAX_VALUE;
+
+ public SharePartitionOffsetManager(SnapshotRegistry snapshotRegistry) {
+ Objects.requireNonNull(snapshotRegistry);
+ offsets = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Method updates internal state with the supplied offset for the provided
+ * share partition key. It then calculates the minimum offset, if possible,
+ * below which all offsets are redundant. This value is then returned as
+ * an optional.
+ * <p>
+ * The value returned is exclusive, in that all offsets below it but non including
+ * it are redundant.
+ *
+ * @param key - represents {@link SharePartitionKey} whose offset needs updating
+ * @param offset - represents the latest partition offset for provided key
+ * @return Optional of last redundant offset, exclusive.
+ */
+ public Optional<Long> updateState(SharePartitionKey key, long offset) {
+ minOffset = Math.min(minOffset, offset);
+ offsets.put(key, offset);
+
+ Optional<Long> deleteTillOffset = lastRedundantOffset();
+ deleteTillOffset.ifPresent(off -> minOffset = off);
+ return deleteTillOffset;
+ }
+
+ private Optional<Long> lastRedundantOffset() {
+ long soFar = Long.MAX_VALUE;
Review Comment:
I would expect this method to return `Optional.empty()` if there are no offsets in the map.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class SharePartitionOffsetManager {
+
+ // map to store share partition key => current partition offset
+ // being written
+ private final TimelineHashMap<SharePartitionKey, Long> offsets;
+
+ // minimum offset representing the smallest necessary offset (non-redundant)
+ // across the internal partition
+ private long minOffset = Long.MAX_VALUE;
Review Comment:
I'm surprised that 0 is not a better starting value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18057: Add share partition redundant offset manager. [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17943:
URL: https://github.com/apache/kafka/pull/17943#discussion_r1858650224
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManagerTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class SharePartitionOffsetManagerTest {
+
+ private SharePartitionOffsetManager manager;
+ private static final SharePartitionKey KEY1 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 0);
+ private static final SharePartitionKey KEY2 = SharePartitionKey.getInstance("gs2", Uuid.randomUuid(), 0);
+ private static final SharePartitionKey KEY3 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 1);
+ private static final SharePartitionKey KEY4 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 7);
+
+ @BeforeEach
+ public void setUp() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext(SharePartitionOffsetManagerTest.class.getName()));
+ manager = new SharePartitionOffsetManager(snapshotRegistry);
+ }
+
+
+ @Test
+ public void testNullRegistryArgInvocation() {
+ assertThrows(NullPointerException.class, () -> new SharePartitionOffsetManager(null));
+ }
+
+ @Test
+ public void testUpdateStateAddsToInternalState() {
+ assertEquals(Optional.empty(), manager.updateState(KEY1, 0L));
+ assertEquals(Optional.of(10L), manager.updateState(KEY1, 10L));
+ assertEquals(Optional.empty(), manager.updateState(KEY2, 15L));
+
+ assertEquals(10L, manager.offsets().get(KEY1));
+ assertEquals(15L, manager.offsets().get(KEY2));
+ }
+
+ private static class ShareOffsetTestHolder {
+ static class TestTuple {
+ final SharePartitionKey key;
+ final long offset;
+ final Optional<Long> expectedOffset;
+
+ private TestTuple(SharePartitionKey key, long offset, Optional<Long> expectedOffset) {
+ this.key = key;
+ this.offset = offset;
+ this.expectedOffset = expectedOffset;
+ }
+
+ static TestTuple instance(SharePartitionKey key, long offset, Optional<Long> expectedOffset) {
+ return new TestTuple(key, offset, expectedOffset);
+ }
+ }
+
+ private final String testName;
+ private final List<TestTuple> tuples;
+ private final boolean shouldRun;
+
+ ShareOffsetTestHolder(String testName, List<TestTuple> tuples) {
+ this(testName, tuples, true);
+ }
+
+ ShareOffsetTestHolder(String testName, List<TestTuple> tuples, boolean shouldRun) {
+ this.testName = testName;
+ this.tuples = tuples;
+ this.shouldRun = shouldRun;
+ }
+ }
+
+ static Stream<ShareOffsetTestHolder> generateNoRedundantStateCases() {
+ return Stream.of(
+ new ShareOffsetTestHolder(
+ "no redundant state single key",
+ Collections.singletonList(
+ ShareOffsetTestHolder.TestTuple.instance(KEY1, 10L, Optional.empty())
+ )
+ ),
+
+ new ShareOffsetTestHolder(
+ "no redundant state multiple keys",
+ Arrays.asList(
Review Comment:
List.of?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Util class to track the offsets written into the internal topic
+ * per share partition key.
+ * It calculates the minimum offset globally up to which the records
+ * in the internal partition are redundant i.e. they have been overridden
+ * by newer records.
+ */
+public class SharePartitionOffsetManager {
+
+ // map to store share partition key => current partition offset
+ // being written
+ private final TimelineHashMap<SharePartitionKey, Long> offsets;
Review Comment:
It's not evident to me why this needs to be a TimelineHashMap yet, but this code is still just the first piece of the puzzle.
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManagerTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class SharePartitionOffsetManagerTest {
+
+ private SharePartitionOffsetManager manager;
+ private static final SharePartitionKey KEY1 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 0);
+ private static final SharePartitionKey KEY2 = SharePartitionKey.getInstance("gs2", Uuid.randomUuid(), 0);
+ private static final SharePartitionKey KEY3 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 1);
+ private static final SharePartitionKey KEY4 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 7);
+
+ @BeforeEach
+ public void setUp() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext(SharePartitionOffsetManagerTest.class.getName()));
+ manager = new SharePartitionOffsetManager(snapshotRegistry);
+ }
+
+
+ @Test
+ public void testNullRegistryArgInvocation() {
+ assertThrows(NullPointerException.class, () -> new SharePartitionOffsetManager(null));
+ }
+
+ @Test
+ public void testUpdateStateAddsToInternalState() {
+ assertEquals(Optional.empty(), manager.updateState(KEY1, 0L));
+ assertEquals(Optional.of(10L), manager.updateState(KEY1, 10L));
+ assertEquals(Optional.empty(), manager.updateState(KEY2, 15L));
+
+ assertEquals(10L, manager.offsets().get(KEY1));
+ assertEquals(15L, manager.offsets().get(KEY2));
+ }
+
+ private static class ShareOffsetTestHolder {
+ static class TestTuple {
+ final SharePartitionKey key;
+ final long offset;
+ final Optional<Long> expectedOffset;
+
+ private TestTuple(SharePartitionKey key, long offset, Optional<Long> expectedOffset) {
+ this.key = key;
+ this.offset = offset;
+ this.expectedOffset = expectedOffset;
+ }
+
+ static TestTuple instance(SharePartitionKey key, long offset, Optional<Long> expectedOffset) {
+ return new TestTuple(key, offset, expectedOffset);
+ }
+ }
+
+ private final String testName;
+ private final List<TestTuple> tuples;
+ private final boolean shouldRun;
+
+ ShareOffsetTestHolder(String testName, List<TestTuple> tuples) {
+ this(testName, tuples, true);
+ }
+
+ ShareOffsetTestHolder(String testName, List<TestTuple> tuples, boolean shouldRun) {
+ this.testName = testName;
+ this.tuples = tuples;
+ this.shouldRun = shouldRun;
+ }
+ }
+
+ static Stream<ShareOffsetTestHolder> generateNoRedundantStateCases() {
+ return Stream.of(
+ new ShareOffsetTestHolder(
+ "no redundant state single key",
+ Collections.singletonList(
Review Comment:
List.of?
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManagerTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class SharePartitionOffsetManagerTest {
+
+ private SharePartitionOffsetManager manager;
+ private static final SharePartitionKey KEY1 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 0);
+ private static final SharePartitionKey KEY2 = SharePartitionKey.getInstance("gs2", Uuid.randomUuid(), 0);
+ private static final SharePartitionKey KEY3 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 1);
+ private static final SharePartitionKey KEY4 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 7);
+
+ @BeforeEach
+ public void setUp() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext(SharePartitionOffsetManagerTest.class.getName()));
+ manager = new SharePartitionOffsetManager(snapshotRegistry);
+ }
+
+
+ @Test
+ public void testNullRegistryArgInvocation() {
+ assertThrows(NullPointerException.class, () -> new SharePartitionOffsetManager(null));
+ }
+
+ @Test
+ public void testUpdateStateAddsToInternalState() {
+ assertEquals(Optional.empty(), manager.updateState(KEY1, 0L));
+ assertEquals(Optional.of(10L), manager.updateState(KEY1, 10L));
+ assertEquals(Optional.empty(), manager.updateState(KEY2, 15L));
+
+ assertEquals(10L, manager.offsets().get(KEY1));
+ assertEquals(15L, manager.offsets().get(KEY2));
+ }
+
+ private static class ShareOffsetTestHolder {
+ static class TestTuple {
+ final SharePartitionKey key;
+ final long offset;
+ final Optional<Long> expectedOffset;
+
+ private TestTuple(SharePartitionKey key, long offset, Optional<Long> expectedOffset) {
+ this.key = key;
+ this.offset = offset;
+ this.expectedOffset = expectedOffset;
+ }
+
+ static TestTuple instance(SharePartitionKey key, long offset, Optional<Long> expectedOffset) {
+ return new TestTuple(key, offset, expectedOffset);
+ }
+ }
+
+ private final String testName;
+ private final List<TestTuple> tuples;
+ private final boolean shouldRun;
Review Comment:
This seems unnecessary at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18057: Add share partition redundant offset manager. [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17943:
URL: https://github.com/apache/kafka/pull/17943#discussion_r1858323955
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class SharePartitionOffsetManager {
+
+ // map to store share partition key => current partition offset
+ // being written
+ private final TimelineHashMap<SharePartitionKey, Long> offsets;
+
+ // minimum offset representing the smallest necessary offset (non-redundant)
+ // across the internal partition
+ private long minOffset = Long.MAX_VALUE;
Review Comment:
This variable is used as arg in call to `Math.min` - 0 might not be suitable for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18057: Add share partition redundant offset manager. [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17943:
URL: https://github.com/apache/kafka/pull/17943#discussion_r1858624278
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManager.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class SharePartitionOffsetManager {
+
+ // map to store share partition key => current partition offset
+ // being written
+ private final TimelineHashMap<SharePartitionKey, Long> offsets;
+
+ // minimum offset representing the smallest necessary offset (non-redundant)
+ // across the internal partition
+ private long minOffset = Long.MAX_VALUE;
Review Comment:
Of course. Long.MAX_VALUE would be a catastrophic offset to delete from 😨
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18057: Add share partition redundant offset manager. [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17943:
URL: https://github.com/apache/kafka/pull/17943#discussion_r1863011946
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/SharePartitionOffsetManagerTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.server.share.SharePartitionKey;
+import org.apache.kafka.timeline.SnapshotRegistry;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class SharePartitionOffsetManagerTest {
+
+ private SharePartitionOffsetManager manager;
+ private static final SharePartitionKey KEY1 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 0);
+ private static final SharePartitionKey KEY2 = SharePartitionKey.getInstance("gs2", Uuid.randomUuid(), 0);
+ private static final SharePartitionKey KEY3 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 1);
+ private static final SharePartitionKey KEY4 = SharePartitionKey.getInstance("gs1", Uuid.randomUuid(), 7);
+
+ @BeforeEach
+ public void setUp() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext(SharePartitionOffsetManagerTest.class.getName()));
+ manager = new SharePartitionOffsetManager(snapshotRegistry);
+ }
+
+
+ @Test
+ public void testNullRegistryArgInvocation() {
+ assertThrows(NullPointerException.class, () -> new SharePartitionOffsetManager(null));
+ }
+
+ @Test
+ public void testUpdateStateAddsToInternalState() {
+ assertEquals(Optional.empty(), manager.updateState(KEY1, 0L));
+ assertEquals(Optional.of(10L), manager.updateState(KEY1, 10L));
+ assertEquals(Optional.empty(), manager.updateState(KEY2, 15L));
+
+ assertEquals(10L, manager.offsets().get(KEY1));
+ assertEquals(15L, manager.offsets().get(KEY2));
+ }
+
+ private static class ShareOffsetTestHolder {
+ static class TestTuple {
+ final SharePartitionKey key;
+ final long offset;
+ final Optional<Long> expectedOffset;
+
+ private TestTuple(SharePartitionKey key, long offset, Optional<Long> expectedOffset) {
+ this.key = key;
+ this.offset = offset;
+ this.expectedOffset = expectedOffset;
+ }
+
+ static TestTuple instance(SharePartitionKey key, long offset, Optional<Long> expectedOffset) {
+ return new TestTuple(key, offset, expectedOffset);
+ }
+ }
+
+ private final String testName;
+ private final List<TestTuple> tuples;
+ private final boolean shouldRun;
Review Comment:
It makes very easy to run a single test among the stream. Helpful in debugging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org