You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "smjn (via GitHub)" <gi...@apache.org> on 2024/09/10 06:44:24 UTC
[PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
smjn opened a new pull request, #17149:
URL: https://github.com/apache/kafka/pull/17149
* Tightened the signature of `ShareCoordinatorShard.combineStateBatches`.
* Added comprehensive tests for the above method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1775461259
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/StateBatchUtilTest.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class StateBatchUtilTest {
+ static class BatchTestHolder {
+ final String testName;
+ final List<PersisterStateBatch> curList;
+ final List<PersisterStateBatch> newList;
+ final List<PersisterStateBatch> expectedResult;
+ final long startOffset;
+ final boolean shouldRun;
Review Comment:
Why do we need this boolean?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
+ int deltaCount = Short.compare(b1.deliveryCount(), b2.deliveryCount());
+
+ // Delivery state could be:
+ // 0 - AVAILABLE (non-terminal)
+ // 1 - ACQUIRED - should not be persisted yet
+ // 2 - ACKNOWLEDGED (terminal)
+ // 3 - ARCHIVING - not implemented in KIP-932 - non-terminal - leads only to ARCHIVED
+ // 4 - ARCHIVED (terminal)
+
+ if (deltaCount == 0) { // same delivery count
+ return Byte.compare(b1.deliveryState(), b2.deliveryState());
+ }
+ return deltaCount;
+ }
+
+ /**
+ * Accepts a list of {@link PersisterStateBatch} and removes/prunes ones which lie
+ * before the start offset.
+ * @param batches - List of {@link PersisterStateBatch}
+ * @param startOffset - long representing the start offset
+ * @return List of pruned {@link PersisterStateBatch}
+ */
+ public static List<PersisterStateBatch> pruneBatches(List<PersisterStateBatch> batches, long startOffset) {
+ if (startOffset != -1) {
+ List<PersisterStateBatch> prunedList = new ArrayList<>(batches.size());
+ batches.forEach(batch -> {
+ if (batch.firstOffset() >= startOffset) {
Review Comment:
The javadoc states: "prunes ones which lie before the start offset." but here we are pruning the batch whose first offset is after the start offset. The logic seems inverted from the description.
Is it the case that the `prunedList` contains batches which were kept (rather than pruned)? If that's true, we should consider a name like `retainBatchesAfterOffset` or `pruneBatchesBeforeOffset` to be more clear.
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/StateBatchUtilTest.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class StateBatchUtilTest {
+ static class BatchTestHolder {
+ final String testName;
+ final List<PersisterStateBatch> curList;
+ final List<PersisterStateBatch> newList;
+ final List<PersisterStateBatch> expectedResult;
+ final long startOffset;
+ final boolean shouldRun;
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset) {
+ this(testName, curList, newList, expectedResult, startOffset, true);
+ }
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset,
+ boolean shouldRun) {
+ this.testName = testName;
+ this.curList = curList;
+ this.newList = newList;
+ this.expectedResult = expectedResult;
+ this.startOffset = startOffset;
+ this.shouldRun = shouldRun;
+ }
+
+ @Override
+ public String toString() {
+ return this.testName;
+ }
+ }
+
+ @SuppressWarnings({"MethodLength"})
+ private static Stream<BatchTestHolder> generator() {
+ return Stream.of(
+ new BatchTestHolder(
+ "Current batches with start offset midway are pruned.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "New batches with start offset midway are pruned.",
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "Both current and new batches empty.",
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ 120
+ ),
+
+ // same state
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first and last offset.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first offset, candidate last offset strictly smaller.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first offset, candidate last offset strictly larger.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 108, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset equal to last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset is last first offset + 1 (contiguous).",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ // different states
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset and last offset match last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset and last offset match last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset same and last offset smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 2),
+ new PersisterStateBatch(106, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset same and last offset smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset same and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset same and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 1, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 110, (byte) 1, (short) 1),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
Review Comment:
Continuing the above suggestion, for these you can do a build thing
```
BatchTestHolder.multiBatch()
.addBatch(100, 104, 0, 1)
.addBatch(105, 110, 1, 1)
.build() // returns List<PersisterStateBatch>
```
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
Review Comment:
In the overlap case, do we need to consider the other state of the batches? (delivery count and delivery state)
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
Review Comment:
naming: how about `prev` rather than `last`. `last` implies the last element in the iterator.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
Review Comment:
I like the visual examples here. Could you select a different character that don't form a ligature? Maybe a `-` instead?
(same for the other examples in this class)
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
Review Comment:
I think we should consider making this into a class. As written, we have a utility function that has quite a bit of state and a large while loop with many branches.
Perhaps a PersisterStateBatchCombiner class? If I understand this workflow, we are doing:
1) sort the batches given
2) merge the batches
3) return merged results
Steps 2 and 3 can be done in a streaming/iterable fashion which can potentially reduce our memory usage.
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/StateBatchUtilTest.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class StateBatchUtilTest {
+ static class BatchTestHolder {
+ final String testName;
+ final List<PersisterStateBatch> curList;
+ final List<PersisterStateBatch> newList;
+ final List<PersisterStateBatch> expectedResult;
+ final long startOffset;
+ final boolean shouldRun;
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset) {
+ this(testName, curList, newList, expectedResult, startOffset, true);
+ }
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset,
+ boolean shouldRun) {
+ this.testName = testName;
+ this.curList = curList;
+ this.newList = newList;
+ this.expectedResult = expectedResult;
+ this.startOffset = startOffset;
+ this.shouldRun = shouldRun;
+ }
+
+ @Override
+ public String toString() {
+ return this.testName;
+ }
+ }
+
+ @SuppressWarnings({"MethodLength"})
+ private static Stream<BatchTestHolder> generator() {
+ return Stream.of(
+ new BatchTestHolder(
+ "Current batches with start offset midway are pruned.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "New batches with start offset midway are pruned.",
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "Both current and new batches empty.",
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ 120
+ ),
+
+ // same state
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first and last offset.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
Review Comment:
Since you have so many of these, I would suggest a factory that lets you pass in ints.
E.g.,
```
new BatchTestHolder(
"Same state. Last and candidate have same first and last offset.",
BatchTestHolder.singleBatch(100, 110, 0, 1),
BatchTestHolder.singleBatch(100, 110, 0, 1),
...
);
```
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
Review Comment:
It seems like the fall-through case here will always copy everything from `sortedBatches` into the List of non-overlapping batches. Instead of this copy, could we return an Optional here where the absence of a value indicates fall-through, and the presence of a value indicates some overlapping state was found?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
+ int deltaCount = Short.compare(b1.deliveryCount(), b2.deliveryCount());
+
+ // Delivery state could be:
+ // 0 - AVAILABLE (non-terminal)
+ // 1 - ACQUIRED - should not be persisted yet
+ // 2 - ACKNOWLEDGED (terminal)
+ // 3 - ARCHIVING - not implemented in KIP-932 - non-terminal - leads only to ARCHIVED
+ // 4 - ARCHIVED (terminal)
+
+ if (deltaCount == 0) { // same delivery count
+ return Byte.compare(b1.deliveryState(), b2.deliveryState());
+ }
+ return deltaCount;
+ }
+
+ /**
+ * Accepts a list of {@link PersisterStateBatch} and removes/prunes ones which lie
+ * before the start offset.
+ * @param batches - List of {@link PersisterStateBatch}
+ * @param startOffset - long representing the start offset
+ * @return List of pruned {@link PersisterStateBatch}
+ */
+ public static List<PersisterStateBatch> pruneBatches(List<PersisterStateBatch> batches, long startOffset) {
+ if (startOffset != -1) {
+ List<PersisterStateBatch> prunedList = new ArrayList<>(batches.size());
+ batches.forEach(batch -> {
+ if (batch.firstOffset() >= startOffset) {
+ // covers:
+ // ______
+ // | -> start offset
+ prunedList.add(batch);
+ } else if (batch.lastOffset() >= startOffset) {
+ // covers:
+ // ________
+ // | -> start offset
+ prunedList.add(new PersisterStateBatch(startOffset, batch.lastOffset(), batch.deliveryState(), batch.deliveryCount()));
+ }
+ // in all other cases, the batch is completely expired.
+ });
+ return prunedList;
+ }
+ return batches;
Review Comment:
Move the short-circuit case to the top
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
Review Comment:
nit: formatting in the java doc (add blank line between paragraphs)
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
This comparison is not considering the offsets. In what cases are we comparing batches without the offsets?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
Review Comment:
Can you add a short comment here explaining how we get to this point (and why the `null`)
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/StateBatchUtilTest.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class StateBatchUtilTest {
+ static class BatchTestHolder {
+ final String testName;
+ final List<PersisterStateBatch> curList;
+ final List<PersisterStateBatch> newList;
+ final List<PersisterStateBatch> expectedResult;
+ final long startOffset;
+ final boolean shouldRun;
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset) {
+ this(testName, curList, newList, expectedResult, startOffset, true);
+ }
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset,
+ boolean shouldRun) {
+ this.testName = testName;
+ this.curList = curList;
+ this.newList = newList;
+ this.expectedResult = expectedResult;
+ this.startOffset = startOffset;
+ this.shouldRun = shouldRun;
+ }
+
+ @Override
+ public String toString() {
+ return this.testName;
+ }
+ }
+
+ @SuppressWarnings({"MethodLength"})
+ private static Stream<BatchTestHolder> generator() {
+ return Stream.of(
+ new BatchTestHolder(
+ "Current batches with start offset midway are pruned.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "New batches with start offset midway are pruned.",
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "Both current and new batches empty.",
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ 120
+ ),
+
+ // same state
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first and last offset.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first offset, candidate last offset strictly smaller.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first offset, candidate last offset strictly larger.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 108, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset equal to last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset is last first offset + 1 (contiguous).",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ // different states
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset and last offset match last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset and last offset match last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset same and last offset smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 2),
+ new PersisterStateBatch(106, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset same and last offset smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset same and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset same and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 1, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 110, (byte) 1, (short) 1),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 1, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 1, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset strictly larger and last offset same as last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset strictly larger and last offset same as last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first and last offsets strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first and last offsets strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ // random cases
+ new BatchTestHolder(
+ "Handle overlapping batches within each list.",
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(121, 130, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1), // overlap with 1st batch
+ new PersisterStateBatch(123, 125, (byte) 0, (short) 1) // overlap with 2nd batch
+
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 119, (byte) 2, (short) 2),
+ new PersisterStateBatch(116, 123, (byte) 2, (short) 2) // overlap with first batch
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(111, 123, (byte) 2, (short) 2),
+ new PersisterStateBatch(124, 130, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Handle overlapping batches with different priority.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(101, 105, (byte) 1, (short) 2),
+ new PersisterStateBatch(101, 115, (byte) 2, (short) 2),
+ new PersisterStateBatch(101, 120, (byte) 3, (short) 2)
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(100, 100, (byte) 0, (short) 1),
+ new PersisterStateBatch(101, 120, (byte) 3, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Handle overlapping batches within each list with pruning.",
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(121, 130, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1) // overlap with 1st batch
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 119, (byte) 2, (short) 2),
+ new PersisterStateBatch(116, 123, (byte) 2, (short) 2) // overlap with first batch
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(120, 123, (byte) 2, (short) 2),
+ new PersisterStateBatch(124, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ // complex tests
+ new BatchTestHolder(
+ "Multiple higher state batch updates.",
+ Collections.singletonList(
+ new PersisterStateBatch(111, 120, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(111, 113, (byte) 0, (short) 2),
+ new PersisterStateBatch(114, 114, (byte) 2, (short) 1),
+ new PersisterStateBatch(115, 119, (byte) 0, (short) 2)
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 113, (byte) 0, (short) 2),
+ new PersisterStateBatch(114, 114, (byte) 2, (short) 1),
+ new PersisterStateBatch(115, 119, (byte) 0, (short) 2),
+ new PersisterStateBatch(120, 120, (byte) 0, (short) 1)
+ ),
+ -1
+ )
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("generator")
+ public void testStateBatchCombine(BatchTestHolder test) {
Review Comment:
I would suggest breaking this up into a few different test cases, organized in whatever way makes sense.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
+ int deltaCount = Short.compare(b1.deliveryCount(), b2.deliveryCount());
+
+ // Delivery state could be:
+ // 0 - AVAILABLE (non-terminal)
+ // 1 - ACQUIRED - should not be persisted yet
+ // 2 - ACKNOWLEDGED (terminal)
+ // 3 - ARCHIVING - not implemented in KIP-932 - non-terminal - leads only to ARCHIVED
+ // 4 - ARCHIVED (terminal)
+
+ if (deltaCount == 0) { // same delivery count
+ return Byte.compare(b1.deliveryState(), b2.deliveryState());
+ }
+ return deltaCount;
+ }
+
+ /**
+ * Accepts a list of {@link PersisterStateBatch} and removes/prunes ones which lie
+ * before the start offset.
+ * @param batches - List of {@link PersisterStateBatch}
+ * @param startOffset - long representing the start offset
+ * @return List of pruned {@link PersisterStateBatch}
+ */
+ public static List<PersisterStateBatch> pruneBatches(List<PersisterStateBatch> batches, long startOffset) {
+ if (startOffset != -1) {
+ List<PersisterStateBatch> prunedList = new ArrayList<>(batches.size());
+ batches.forEach(batch -> {
+ if (batch.firstOffset() >= startOffset) {
+ // covers:
+ // ______
+ // | -> start offset
+ prunedList.add(batch);
+ } else if (batch.lastOffset() >= startOffset) {
+ // covers:
+ // ________
+ // | -> start offset
+ prunedList.add(new PersisterStateBatch(startOffset, batch.lastOffset(), batch.deliveryState(), batch.deliveryCount()));
+ }
+ // in all other cases, the batch is completely expired.
+ });
+ return prunedList;
+ }
+ return batches;
+ }
+
+ /**
+ * Converts a {@link ShareUpdateValue.StateBatch} type state batch to {@link PersisterStateBatch}
+ * @param batch - The object representing {@link ShareUpdateValue.StateBatch}
+ * @return {@link PersisterStateBatch}
+ */
+ public static PersisterStateBatch toPersisterStateBatch(ShareUpdateValue.StateBatch batch) {
+ return new PersisterStateBatch(
+ batch.firstOffset(),
+ batch.lastOffset(),
+ batch.deliveryState(),
+ batch.deliveryCount()
+ );
+ }
+
+ /**
+ * Holder class for intermediate state
+ * used in the batch merge algorithm.
+ */
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
Review Comment:
While this is indeed a sentinel object, we usually give them a name like `EMPTY`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1756503735
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,195 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ // will take care of overlapping batches
+ Queue<PersisterStateBatch> batchQueue = new LinkedList<>(
Review Comment:
For my understanding: Why it simply cannot be a List of type LinkedList, what queue operations we need here?
##########
server-common/src/main/java/org/apache/kafka/server/group/share/PersisterStateBatch.java:
##########
@@ -95,4 +95,18 @@ public String toString() {
"deliveryCount=" + deliveryCount +
")";
}
+
+ @Override
+ public int compareTo(Object o) {
+ PersisterStateBatch that = (PersisterStateBatch) o;
+ int deltaFirst = Long.compare(this.firstOffset(), that.firstOffset());
+ if (deltaFirst == 0) {
+ int deltaLast = Long.compare(this.lastOffset(), that.lastOffset());
+ if (deltaLast == 0) {
+ return this.deliveryCount() - that.deliveryCount();
+ }
+ return deltaLast;
+ }
+ return deltaFirst;
+ }
Review Comment:
What about `deliveryState` is that not important to compare for this class? If not then can we please write a comment regarding why it's skipped.
##########
checkstyle/suppressions.xml:
##########
@@ -340,6 +340,14 @@
<suppress checks="NPathComplexity"
files="CoordinatorRuntime.java"/>
+ <!-- share coordinator -->
+ <suppress checks="MethodLength"
+ files="ShareCoordinatorShardTest.java"/>
+ <suppress checks="NPathComplexity"
+ files="ShareCoordinatorShard.java"/>
+ <suppress checks="CyclomaticComplexity"
+ files="ShareCoordinatorShard.java"/>
Review Comment:
nit: While it's fine to avoid suppressions but avoid if we have better way to handle the situation in code. For `MethodLength` you might want to have a method to generate `new TestAttributes` or something similar to avoid method length issue.
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/ShareCoordinatorShardTest.java:
##########
@@ -791,6 +793,304 @@ public void testNonSequentialBatchUpdates() {
verify(shard.getMetricsShard(), times(3)).record(ShareCoordinatorMetrics.SHARE_COORDINATOR_WRITE_SENSOR_NAME);
}
+ @Test
+ public void testStateBatchCombine() {
Review Comment:
Might be better if you parametrize the test with thest input you created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1751793586
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/ShareCoordinatorShardTest.java:
##########
@@ -791,6 +793,147 @@ public void testNonSequentialBatchUpdates() {
verify(shard.getMetricsShard(), times(3)).record(ShareCoordinatorMetrics.SHARE_COORDINATOR_WRITE_SENSOR_NAME);
}
+ @Test
+ public void testStateBatchCombine() {
+ class TestAttributes {
+ final String testName;
+ final LinkedHashSet<PersisterOffsetsStateBatch> curSet;
+ final LinkedHashSet<PersisterOffsetsStateBatch> newSet;
+ final List<PersisterOffsetsStateBatch> expectedResult;
+ final long startOffset;
+
+ TestAttributes(String testName,
+ LinkedHashSet<PersisterOffsetsStateBatch> curSet,
+ LinkedHashSet<PersisterOffsetsStateBatch> newSet,
+ List<PersisterOffsetsStateBatch> expectedResult,
+ long startOffset) {
+ this.testName = testName;
+ this.curSet = curSet;
+ this.newSet = newSet;
+ this.expectedResult = expectedResult;
+ this.startOffset = startOffset;
+ }
+ }
+
+ List<TestAttributes> tests = Arrays.asList(
+ new TestAttributes(
+ "Both cur and new set empty => empty result list.",
+ new LinkedHashSet<>(),
+ new LinkedHashSet<>(),
+ new LinkedList<>(),
+ -1
+ ),
+
+ new TestAttributes(
+ "New set is empty => result list only contains cur set batches.",
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)
+ )),
+ new LinkedHashSet<>(),
+ Collections.singletonList(new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)),
+ -1
+ ),
+
+ new TestAttributes(
+ "Cur set is empty => result list only contains new set batches.",
+ new LinkedHashSet<>(),
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)
+ )),
+ Collections.singletonList(new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)),
+ -1
+ ),
+
+ new TestAttributes(
+ "Sets have no overlap => result list contains batches from both sets.",
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)
+ )),
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(120, 130, (byte) 0, (short) 1)
+ )),
+ Arrays.asList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterOffsetsStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new TestAttributes(
+ "Sets have partial overlap => result list contains batches from both sets (we do not add gaps).",
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)
+ )),
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(105, 130, (byte) 0, (short) 1)
+ )),
+ Arrays.asList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterOffsetsStateBatch(105, 130, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new TestAttributes(
+ "Sets have overlapping batches => batches in new set are preferred. " +
+ "Overlap means cur.firstOffset == new.firstOffset && cur.lastOffset == new.lastOffset",
+ new LinkedHashSet<>(Arrays.asList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterOffsetsStateBatch(111, 120, (byte) 0, (short) 1)
+ )),
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(111, 120, (byte) 2, (short) 2) // overlap
+ )),
+ Arrays.asList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterOffsetsStateBatch(111, 120, (byte) 2, (short) 2) // batch from new set
+ ),
+ -1
+ ),
+
+ new TestAttributes(
+ "Batches which are older than start offset are removed. " +
+ "Old means batch.lastOffset < startOffset",
+ new LinkedHashSet<>(Arrays.asList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1), // should be removed
+ new PersisterOffsetsStateBatch(111, 120, (byte) 0, (short) 1),
+ new PersisterOffsetsStateBatch(121, 130, (byte) 0, (short) 1)
+ )),
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(111, 120, (byte) 2, (short) 2) // overlap
+ )),
+ Arrays.asList(
+ new PersisterOffsetsStateBatch(121, 130, (byte) 0, (short) 1), // offset values are higher but this arrived first - order is valid
+ new PersisterOffsetsStateBatch(111, 120, (byte) 2, (short) 2) // batch from new set
+ ),
+ 111
+ ),
+
+ new TestAttributes(
+ "StartOffset is -1 => no batches are considered old but overlaps are still removed preferring the new set.",
+ new LinkedHashSet<>(Arrays.asList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1), // should be removed
Review Comment:
This comment `// should be removed` is incorrect.
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/ShareCoordinatorShardTest.java:
##########
@@ -791,6 +793,147 @@ public void testNonSequentialBatchUpdates() {
verify(shard.getMetricsShard(), times(3)).record(ShareCoordinatorMetrics.SHARE_COORDINATOR_WRITE_SENSOR_NAME);
}
+ @Test
+ public void testStateBatchCombine() {
+ class TestAttributes {
+ final String testName;
+ final LinkedHashSet<PersisterOffsetsStateBatch> curSet;
+ final LinkedHashSet<PersisterOffsetsStateBatch> newSet;
+ final List<PersisterOffsetsStateBatch> expectedResult;
+ final long startOffset;
+
+ TestAttributes(String testName,
+ LinkedHashSet<PersisterOffsetsStateBatch> curSet,
+ LinkedHashSet<PersisterOffsetsStateBatch> newSet,
+ List<PersisterOffsetsStateBatch> expectedResult,
+ long startOffset) {
+ this.testName = testName;
+ this.curSet = curSet;
+ this.newSet = newSet;
+ this.expectedResult = expectedResult;
+ this.startOffset = startOffset;
+ }
+ }
+
+ List<TestAttributes> tests = Arrays.asList(
+ new TestAttributes(
+ "Both cur and new set empty => empty result list.",
+ new LinkedHashSet<>(),
+ new LinkedHashSet<>(),
+ new LinkedList<>(),
+ -1
+ ),
+
+ new TestAttributes(
+ "New set is empty => result list only contains cur set batches.",
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)
+ )),
+ new LinkedHashSet<>(),
+ Collections.singletonList(new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)),
+ -1
+ ),
+
+ new TestAttributes(
+ "Cur set is empty => result list only contains new set batches.",
+ new LinkedHashSet<>(),
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)
+ )),
+ Collections.singletonList(new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)),
+ -1
+ ),
+
+ new TestAttributes(
+ "Sets have no overlap => result list contains batches from both sets.",
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)
+ )),
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(120, 130, (byte) 0, (short) 1)
+ )),
+ Arrays.asList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterOffsetsStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new TestAttributes(
+ "Sets have partial overlap => result list contains batches from both sets (we do not add gaps).",
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1)
+ )),
+ new LinkedHashSet<>(Collections.singletonList(
+ new PersisterOffsetsStateBatch(105, 130, (byte) 0, (short) 1)
+ )),
+ Arrays.asList(
+ new PersisterOffsetsStateBatch(100, 110, (byte) 0, (short) 1),
Review Comment:
This result is correct, but I would expect `(100, 130, (byte) 0, (short) 1))`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1759221583
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ if (compareBatchState(candidate, last) < 0) { // candidate is lower priority
+ sortedBatches.add(last);
Review Comment:
actually on second look, there is no state comparison needed. By nature of the compareTo method we defined, conditate can never have lower priority if last.firstOffset == candidate.firstOffset and last.lastOffset == candidate.lastOffset
will simplify the condition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1759247979
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
Review Comment:
then we'll make it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1779348983
##########
share/src/main/java/org/apache/kafka/server/share/PersisterStateBatch.java:
##########
@@ -77,22 +77,40 @@ public boolean equals(Object o) {
PersisterStateBatch that = (PersisterStateBatch) o;
return firstOffset == that.firstOffset &&
lastOffset == that.lastOffset &&
- deliveryState == that.deliveryState &&
- deliveryCount == that.deliveryCount;
+ deliveryCount == that.deliveryCount &&
+ deliveryState == that.deliveryState;
}
@Override
public int hashCode() {
- return Objects.hash(firstOffset, lastOffset, deliveryState, deliveryCount);
+ return Objects.hash(firstOffset, lastOffset, deliveryCount, deliveryState);
}
@Override
public String toString() {
return "PersisterStateBatch(" +
"firstOffset=" + firstOffset + "," +
"lastOffset=" + lastOffset + "," +
- "deliveryState=" + deliveryState + "," +
- "deliveryCount=" + deliveryCount +
+ "deliveryCount=" + deliveryCount + "," +
+ "deliveryState=" + deliveryState +
")";
}
+
+ @Override
+ public int compareTo(Object o) {
+ PersisterStateBatch that = (PersisterStateBatch) o;
+ int deltaFirst = Long.compare(this.firstOffset(), that.firstOffset());
+ if (deltaFirst == 0) {
+ int deltaLast = Long.compare(this.lastOffset(), that.lastOffset());
+ if (deltaLast == 0) {
+ int deltaCount = this.deliveryCount() - that.deliveryCount();
Review Comment:
No, this was a conscious decision:
Plz check discussion https://github.com/apache/kafka/pull/17149#discussion_r1777003384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776452137
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
Review Comment:
No the logic is not that simple. A single iteration over the sorted batches might not be enough.
Consider:
```
-------- A [1,10,0,1]
--- B [5,7,0,2]
-------------- C [5,15,0,3]
```
A and B will combine to
```
---- [1,4,0,1]
---- [5,7,0,2]
--- [8,10,0,1]
```
Now when combining with C, we have 2 previous batches to consider.
Secondly,
```
-------- A [1,10,0,1]
--- B [5,7,0,2]
--- C [5,7,0,3]
```
A and B will combine to
```
---- [1,4,0,1]
---- [5,7,0,2]
--- [8,10,0,1]
--- <- C - we broke invariant for being sorted by batches
```
In the current impl, these situations are implicitly handled by virtue of the treeset. Any newly generated batches are pushed back into the treeset and the `getOverlappingState` method finds the first overlapping pair as well as returns the non-overlapping prefix.
The non-overlapping prefix is then REMOVED from the treeset hence, once a batch is no longer overlapping, it is only looked at once guaranteeing running time efficiency.
@AndrewJSchofield
@mumrah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1779349102
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/PersisterStateBatchCombiner.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class PersisterStateBatchCombiner {
+ private List<PersisterStateBatch> combinedBatchList; // link between pruning and merging
+ private final long startOffset;
+ private TreeSet<PersisterStateBatch> sortedBatches;
+ private List<PersisterStateBatch> finalBatchList; // final list is built here
+
+ public PersisterStateBatchCombiner(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ initializeCombinedList(batchesSoFar, newBatches);
+ finalBatchList = new ArrayList<>(combinedBatchList.size() * 2); // heuristic size
+ this.startOffset = startOffset;
+ }
+
+ private void initializeCombinedList(List<PersisterStateBatch> batchesSoFar, List<PersisterStateBatch> newBatches) {
+ boolean soFarEmpty = batchesSoFar == null || batchesSoFar.isEmpty();
+ boolean newBatchesEmpty = newBatches == null || newBatches.isEmpty();
+
+ if (soFarEmpty && newBatchesEmpty) {
+ combinedBatchList = new ArrayList<>();
+ } else if (soFarEmpty) {
+ combinedBatchList = newBatches;
+ } else if (newBatchesEmpty) {
+ combinedBatchList = batchesSoFar;
+ } else {
+ combinedBatchList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedBatchList.addAll(batchesSoFar);
+ combinedBatchList.addAll(newBatches);
+ }
+ }
+
+ /**
+ * Algorithm: Merge current state batches and new batches into a single non-overlapping batch list.
+ * Input: batchesSoFar, newBatches, startOffset
+ * Output: combined list with non-overlapping batches (finalBatchList)
+ * <p>
+ * - Add both currentBatches and newBatches into a single list combinedBatchList
+ * - Remove/prune any batches from the combinedBatchList:
+ * - if batch.lastOffset < startOffset then remove batch from combinedBatchList
+ * - else if batch.firstOffset > startOffset then we will keep the batch
+ * - else if batch.firstOffset <= startOffset <= batch.lastOffset then keep [startOffset, batch.lastOffset] part only and discard rest.
+ * - create a treeset sortedBatches using pruned combinedBatchList
+ * - do repeat until all batches in sortedSet are non-overlapping:
+ * - find first 2 overlapping batches in sortedBatches set, say, prev and candidate.
+ * - remove any non-overlapping batches from sortedBatches encountered during the find operation and add them to a finalBatchList
+ * - based on various conditions of offset overlap and batch state differences combine the batches or
+ * create new batches, if required, and add to the sortedSet.
+ * - done
+ * - return the finalBatchList
+ *
+ * @return list of {@link PersisterStateBatch} representing non-overlapping combined batches
+ */
+ public List<PersisterStateBatch> combineStateBatches() {
+ pruneBatches();
+ mergeBatches();
+ return finalBatchList;
+ }
+
+ private void mergeBatches() {
+ if (combinedBatchList.size() < 2) {
+ finalBatchList = combinedBatchList;
+ return;
+ }
+
+ sortedBatches = new TreeSet<>(combinedBatchList);
Review Comment:
This was done here as the treeset might not be needed at all if intervals are less than 2, hence a small optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1756707238
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,195 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ // will take care of overlapping batches
+ Queue<PersisterStateBatch> batchQueue = new LinkedList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(batchesSoFar),
+ startOffset
+ )
+ ));
+
+ // will take care of overlapping batches
+ List<PersisterStateBatch> modifiedNewBatches = new ArrayList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(newBatches),
+ startOffset
+ )
+ ));
+
+ for (PersisterStateBatch batch : modifiedNewBatches) {
+ for (int i = 0; i < batchQueue.size(); i++) {
Review Comment:
We will re-evaluate `batchQueue.size()` on each loop iteration, and it could in theory mutate quite significantly. I don't believe this is a safe loop condition.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,195 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ // will take care of overlapping batches
+ Queue<PersisterStateBatch> batchQueue = new LinkedList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(batchesSoFar),
+ startOffset
+ )
+ ));
+
+ // will take care of overlapping batches
+ List<PersisterStateBatch> modifiedNewBatches = new ArrayList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(newBatches),
+ startOffset
+ )
+ ));
+
+ for (PersisterStateBatch batch : modifiedNewBatches) {
+ for (int i = 0; i < batchQueue.size(); i++) {
+ PersisterStateBatch cur = batchQueue.poll();
+ // cur batch under inspection has no overlap with the new one
+ // we will need to add to our result
+ if (batch.lastOffset() < cur.firstOffset() || batch.firstOffset() > cur.lastOffset()) {
+ batchQueue.add(cur);
+ continue;
+ }
+
+ // Covers cases where we need to create a new interval
+ // from the current one such that they do not
+ // overlap with the new one.
+ // Following cases will not produce any new records so need not be handled.
+ // cur: ____ ______ ______ _____
+ // new: ________ ______ _________ _________
+
+
+ // covers
+ // cur: ______ _____ _____ ______
+ // batch: ___ _____ ___ ___
+ if (batch.firstOffset() >= cur.firstOffset() && batch.lastOffset() <= cur.lastOffset()) {
+ // extra batch needs to be created
+ if (batch.firstOffset() > cur.firstOffset()) {
+ batchQueue.add(
+ new PersisterStateBatch(
+ cur.firstOffset(),
+ batch.firstOffset() - 1,
+ cur.deliveryState(),
+ cur.deliveryCount()
+ )
+ );
+ }
+
+ // extra batch needs to be created
+ if (batch.lastOffset() < cur.lastOffset()) {
+ batchQueue.add(
+ new PersisterStateBatch(
+ batch.lastOffset() + 1,
+ cur.lastOffset(),
+ cur.deliveryState(),
+ cur.deliveryCount()
+ )
+ );
+ }
+ } else if (batch.firstOffset() < cur.firstOffset() && batch.lastOffset() < cur.lastOffset()) {
+ // covers
+ // ______
+ //____
+ batchQueue.add(new PersisterStateBatch(
+ batch.lastOffset() + 1,
+ cur.lastOffset(),
+ cur.deliveryState(),
+ cur.deliveryCount()
+ ));
+ } else if (batch.firstOffset() > cur.firstOffset() && batch.lastOffset() > cur.lastOffset()) {
+ // covers
+ // ______
+ // _______
+ batchQueue.add(new PersisterStateBatch(
+ cur.firstOffset(),
+ batch.firstOffset() - 1,
+ cur.deliveryState(),
+ cur.deliveryCount()
+ ));
+ }
+ }
+ batchQueue.add(batch);
+ }
+
+ // sort, prune and merge intervals
+ return mergeBatches(
+ pruneBatches(
+ getSortedList(batchQueue),
+ startOffset
+ )
+ );
+ }
+
+ private static List<PersisterStateBatch> getSortedList(Collection<PersisterStateBatch> collection) {
+ List<PersisterStateBatch> finalList = new ArrayList<>(collection);
+ finalList.sort(PersisterStateBatch::compareTo);
+ return finalList;
+ }
+
+ // assumes sorted input
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ Stack<PersisterStateBatch> stack = new Stack<>();
+ stack.add(batches.get(0));
Review Comment:
Can't this be `batches.remove(0)`? Otherwise, the first element will be added to the stack and also the initial candidate in the loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776430833
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
We use the offsets to determine the relative positioning and then use just that state to determine how to break the batches. Based on state, overlapping batches could be merged or broken down.
example:
1.
------ [1,10,0,1]
------- [5,15,0,1]
=>
---------- [1,15,0,1]
2.
------ [1,10,0,1]
------- [5,15,2,1]
=>
---- [1,4,0,1]
------- [5,15,2,1]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1771822574
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,322 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
Review Comment:
There's no need to assign to `last` here. The pattern you've followed in the later cases simply adds the constructed object.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,322 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 (NP) 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
Review Comment:
So I suppose that case 2 would actually be case 3 because of the sorting order of the key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1779350202
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/PersisterStateBatchCombiner.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class PersisterStateBatchCombiner {
+ private List<PersisterStateBatch> combinedBatchList; // link between pruning and merging
+ private final long startOffset;
+ private TreeSet<PersisterStateBatch> sortedBatches;
+ private List<PersisterStateBatch> finalBatchList; // final list is built here
+
+ public PersisterStateBatchCombiner(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ initializeCombinedList(batchesSoFar, newBatches);
+ finalBatchList = new ArrayList<>(combinedBatchList.size() * 2); // heuristic size
+ this.startOffset = startOffset;
+ }
+
+ private void initializeCombinedList(List<PersisterStateBatch> batchesSoFar, List<PersisterStateBatch> newBatches) {
+ boolean soFarEmpty = batchesSoFar == null || batchesSoFar.isEmpty();
+ boolean newBatchesEmpty = newBatches == null || newBatches.isEmpty();
+
+ if (soFarEmpty && newBatchesEmpty) {
+ combinedBatchList = new ArrayList<>();
+ } else if (soFarEmpty) {
+ combinedBatchList = newBatches;
+ } else if (newBatchesEmpty) {
+ combinedBatchList = batchesSoFar;
+ } else {
+ combinedBatchList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedBatchList.addAll(batchesSoFar);
+ combinedBatchList.addAll(newBatches);
+ }
+ }
+
+ /**
+ * Algorithm: Merge current state batches and new batches into a single non-overlapping batch list.
+ * Input: batchesSoFar, newBatches, startOffset
+ * Output: combined list with non-overlapping batches (finalBatchList)
+ * <p>
+ * - Add both currentBatches and newBatches into a single list combinedBatchList
+ * - Remove/prune any batches from the combinedBatchList:
+ * - if batch.lastOffset < startOffset then remove batch from combinedBatchList
+ * - else if batch.firstOffset > startOffset then we will keep the batch
+ * - else if batch.firstOffset <= startOffset <= batch.lastOffset then keep [startOffset, batch.lastOffset] part only and discard rest.
+ * - create a treeset sortedBatches using pruned combinedBatchList
+ * - do repeat until all batches in sortedSet are non-overlapping:
+ * - find first 2 overlapping batches in sortedBatches set, say, prev and candidate.
+ * - remove any non-overlapping batches from sortedBatches encountered during the find operation and add them to a finalBatchList
+ * - based on various conditions of offset overlap and batch state differences combine the batches or
+ * create new batches, if required, and add to the sortedSet.
+ * - done
+ * - return the finalBatchList
+ *
+ * @return list of {@link PersisterStateBatch} representing non-overlapping combined batches
+ */
+ public List<PersisterStateBatch> combineStateBatches() {
+ pruneBatches();
+ mergeBatches();
+ return finalBatchList;
+ }
+
+ private void mergeBatches() {
+ if (combinedBatchList.size() < 2) {
+ finalBatchList = combinedBatchList;
+ return;
+ }
+
+ sortedBatches = new TreeSet<>(combinedBatchList);
+
+ BatchOverlapState overlapState = getOverlappingState();
+
+ while (overlapState != BatchOverlapState.EMPTY) {
+ PersisterStateBatch prev = overlapState.prev();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove both previous and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(prev);
+ sortedBatches.remove(candidate);
+
+ if (compareBatchState(candidate, prev) == 0) { // same state and overlap or contiguous
+ // overlap and same state (prev.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // prev: ------ ------- ------- ------- ------- -------- -------
+ // candidate: ------ ---- ---------- --- ---- ------- -------
+ handleSameStateOverlap(prev, candidate);
+ } else { // diff state and non-contiguous overlap
+ // overlap and diff state
+ // covers:
+ // case: 1 2* 3 4 5 6 7*
+ // prev: ------ ------- ------- ------- ------- -------- ------
+ // candidate: ------ ---- --------- ---- ---- ------- -------
+ // max batches: 1 2 2 3 2 2 2
+ // min batches: 1 1 1 1 1 2 1
+ // * not possible with treeset
+ handleDiffStateOverlap(prev, candidate);
+ }
+ overlapState = getOverlappingState();
+ }
+ finalBatchList.addAll(sortedBatches); // some non overlapping batches might have remained
+ }
+
+ /**
+ * Compares the non-offset state of 2 batches i.e. the deliveryCount and deliverState.
+ * <p>
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ *
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
No we explicitly want to compare the 2 state parameters only.
We decide whether the pair can be combined based on offsets but when actually merging we can 1, 2 or 3 resultant batches. This is purely determined by the delivery count and state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776452137
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
Review Comment:
No the logic is not that simple. A single iteration over the sorted batches might not be enough.
The invariant is that the batches remain sorted even after manipulation
Consider:
```
-------- A [1,10,0,1]
--- B [5,7,0,2]
-------------- C [5,15,0,3]
```
A and B will combine to
```
---- [1,4,0,1]
---- [5,7,0,2]
--- [8,10,0,1]
```
Now when combining with C, we have 2 previous batches to consider.
Secondly,
```
-------- A [1,10,0,1]
--- B [5,7,0,2]
--- C [5,7,0,3]
```
A and B will combine to
```
---- [1,4,0,1]
---- [5,7,0,2]
--- [8,10,0,1]
--- <- C - we broke invariant for being sorted by batches
```
In the current impl, these situations are implicitly handled by virtue of the treeset. Any newly generated batches are pushed back into the treeset and the `getOverlappingState` method finds the first overlapping pair as well as returns the non-overlapping prefix.
The non-overlapping prefix is then REMOVED from the treeset hence, once a batch is no longer overlapping, it is only looked at once guaranteeing running time efficiency.
@AndrewJSchofield
@mumrah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1757017297
##########
checkstyle/suppressions.xml:
##########
@@ -340,6 +340,14 @@
<suppress checks="NPathComplexity"
files="CoordinatorRuntime.java"/>
+ <!-- share coordinator -->
+ <suppress checks="MethodLength"
+ files="ShareCoordinatorShardTest.java"/>
+ <suppress checks="NPathComplexity"
+ files="ShareCoordinatorShard.java"/>
+ <suppress checks="CyclomaticComplexity"
+ files="ShareCoordinatorShard.java"/>
Review Comment:
The problem will still remain, since the test cases are curated to create various scenarios. We cannot generate them deterministically. If I add a new private method to return the tests, it will still have MethodLength issue.
Best I can do is to suppress warning for this specific method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1757168070
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/ShareCoordinatorShardTest.java:
##########
@@ -791,6 +793,304 @@ public void testNonSequentialBatchUpdates() {
verify(shard.getMetricsShard(), times(3)).record(ShareCoordinatorMetrics.SHARE_COORDINATOR_WRITE_SENSOR_NAME);
}
+ @Test
+ public void testStateBatchCombine() {
Review Comment:
Ok, will add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1759079182
##########
server-common/src/main/java/org/apache/kafka/server/group/share/PersisterStateBatch.java:
##########
@@ -25,7 +25,7 @@
/**
* This class contains the information for a single batch of state information for use by the {@link Persister}.
*/
-public class PersisterStateBatch {
+public class PersisterStateBatch implements Comparable {
private final long firstOffset;
private final long lastOffset;
private final byte deliveryState;
Review Comment:
In sorting terms, this is <firstOffset, lastOffset, deliveryCount, deliveryState>. I suggest putting the member variables in the same order.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ if (compareBatchState(candidate, last) < 0) { // candidate is lower priority
+ sortedBatches.add(last);
+ } else { // last is lower priority
+ sortedBatches.add(candidate);
Review Comment:
And doesn't this duplicate `candidate`?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
Review Comment:
If I understand correctly, at line 676, both of `last` and `candidate` are members of `sortedBatches`.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ if (compareBatchState(candidate, last) < 0) { // candidate is lower priority
+ sortedBatches.add(last);
+ } else { // last is lower priority
+ sortedBatches.add(candidate);
+ }
+ } else if (candidate.lastOffset() < last.lastOffset()) { // case 2
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.firstOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else { // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.lastOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> batchSet) {
+ Iterator<PersisterStateBatch> iter = batchSet.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(batchSet.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
Could do with a comment. This is approximately following the contract for the methods like `Short.compare(short x, short y)`. If x > y then +ve, if x < y then -ve, if x == y then 0.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ if (compareBatchState(candidate, last) < 0) { // candidate is lower priority
+ sortedBatches.add(last);
+ } else { // last is lower priority
+ sortedBatches.add(candidate);
+ }
+ } else if (candidate.lastOffset() < last.lastOffset()) { // case 2
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.firstOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else { // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.lastOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> batchSet) {
Review Comment:
This definitely needs a comment. For example, if the batch set is empty, it will throw an exception, so don't call it with an empty set.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
Review Comment:
I don't believe that overlapState will ever be SENTINEL.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
Review Comment:
tiny nit: trailing spaces
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ if (compareBatchState(candidate, last) < 0) { // candidate is lower priority
+ sortedBatches.add(last);
+ } else { // last is lower priority
+ sortedBatches.add(candidate);
+ }
+ } else if (candidate.lastOffset() < last.lastOffset()) { // case 2
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.firstOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else { // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.lastOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> batchSet) {
Review Comment:
I would tend to call this argument `sortedBatches` for consistency with the caller.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ if (compareBatchState(candidate, last) < 0) { // candidate is lower priority
+ sortedBatches.add(last);
Review Comment:
`candidate` has not been removed, but shouldn't it be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1759227756
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ if (compareBatchState(candidate, last) < 0) { // candidate is lower priority
+ sortedBatches.add(last);
+ } else { // last is lower priority
+ sortedBatches.add(candidate);
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776423900
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/StateBatchUtilTest.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class StateBatchUtilTest {
+ static class BatchTestHolder {
+ final String testName;
+ final List<PersisterStateBatch> curList;
+ final List<PersisterStateBatch> newList;
+ final List<PersisterStateBatch> expectedResult;
+ final long startOffset;
+ final boolean shouldRun;
Review Comment:
Helps in running a single test or a subset of tests. On by default but can be turned off in class and switched on for specific test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776535621
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/StateBatchUtilTest.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class StateBatchUtilTest {
+ static class BatchTestHolder {
+ final String testName;
+ final List<PersisterStateBatch> curList;
+ final List<PersisterStateBatch> newList;
+ final List<PersisterStateBatch> expectedResult;
+ final long startOffset;
+ final boolean shouldRun;
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset) {
+ this(testName, curList, newList, expectedResult, startOffset, true);
+ }
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset,
+ boolean shouldRun) {
+ this.testName = testName;
+ this.curList = curList;
+ this.newList = newList;
+ this.expectedResult = expectedResult;
+ this.startOffset = startOffset;
+ this.shouldRun = shouldRun;
+ }
+
+ @Override
+ public String toString() {
+ return this.testName;
+ }
+ }
+
+ @SuppressWarnings({"MethodLength"})
+ private static Stream<BatchTestHolder> generator() {
+ return Stream.of(
+ new BatchTestHolder(
+ "Current batches with start offset midway are pruned.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "New batches with start offset midway are pruned.",
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "Both current and new batches empty.",
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ 120
+ ),
+
+ // same state
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first and last offset.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first offset, candidate last offset strictly smaller.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first offset, candidate last offset strictly larger.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 108, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset equal to last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset is last first offset + 1 (contiguous).",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ // different states
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset and last offset match last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset and last offset match last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset same and last offset smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 2),
+ new PersisterStateBatch(106, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset same and last offset smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset same and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset same and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 1, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 110, (byte) 1, (short) 1),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 1, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 1, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset strictly larger and last offset same as last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset strictly larger and last offset same as last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first and last offsets strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first and last offsets strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ // random cases
+ new BatchTestHolder(
+ "Handle overlapping batches within each list.",
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(121, 130, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1), // overlap with 1st batch
+ new PersisterStateBatch(123, 125, (byte) 0, (short) 1) // overlap with 2nd batch
+
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 119, (byte) 2, (short) 2),
+ new PersisterStateBatch(116, 123, (byte) 2, (short) 2) // overlap with first batch
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(111, 123, (byte) 2, (short) 2),
+ new PersisterStateBatch(124, 130, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Handle overlapping batches with different priority.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(101, 105, (byte) 1, (short) 2),
+ new PersisterStateBatch(101, 115, (byte) 2, (short) 2),
+ new PersisterStateBatch(101, 120, (byte) 3, (short) 2)
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(100, 100, (byte) 0, (short) 1),
+ new PersisterStateBatch(101, 120, (byte) 3, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Handle overlapping batches within each list with pruning.",
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(121, 130, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1) // overlap with 1st batch
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 119, (byte) 2, (short) 2),
+ new PersisterStateBatch(116, 123, (byte) 2, (short) 2) // overlap with first batch
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(120, 123, (byte) 2, (short) 2),
+ new PersisterStateBatch(124, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ // complex tests
+ new BatchTestHolder(
+ "Multiple higher state batch updates.",
+ Collections.singletonList(
+ new PersisterStateBatch(111, 120, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(111, 113, (byte) 0, (short) 2),
+ new PersisterStateBatch(114, 114, (byte) 2, (short) 1),
+ new PersisterStateBatch(115, 119, (byte) 0, (short) 2)
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 113, (byte) 0, (short) 2),
+ new PersisterStateBatch(114, 114, (byte) 2, (short) 1),
+ new PersisterStateBatch(115, 119, (byte) 0, (short) 2),
+ new PersisterStateBatch(120, 120, (byte) 0, (short) 1)
+ ),
+ -1
+ )
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("generator")
+ public void testStateBatchCombine(BatchTestHolder test) {
Review Comment:
will break generator - tests will exactly be the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1779239837
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/PersisterStateBatchCombiner.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class PersisterStateBatchCombiner {
+ private List<PersisterStateBatch> combinedBatchList; // link between pruning and merging
+ private final long startOffset;
+ private TreeSet<PersisterStateBatch> sortedBatches;
+ private List<PersisterStateBatch> finalBatchList; // final list is built here
+
+ public PersisterStateBatchCombiner(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ initializeCombinedList(batchesSoFar, newBatches);
+ finalBatchList = new ArrayList<>(combinedBatchList.size() * 2); // heuristic size
+ this.startOffset = startOffset;
+ }
+
+ private void initializeCombinedList(List<PersisterStateBatch> batchesSoFar, List<PersisterStateBatch> newBatches) {
+ boolean soFarEmpty = batchesSoFar == null || batchesSoFar.isEmpty();
+ boolean newBatchesEmpty = newBatches == null || newBatches.isEmpty();
+
+ if (soFarEmpty && newBatchesEmpty) {
+ combinedBatchList = new ArrayList<>();
+ } else if (soFarEmpty) {
+ combinedBatchList = newBatches;
+ } else if (newBatchesEmpty) {
+ combinedBatchList = batchesSoFar;
+ } else {
+ combinedBatchList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedBatchList.addAll(batchesSoFar);
+ combinedBatchList.addAll(newBatches);
+ }
+ }
+
+ /**
+ * Algorithm: Merge current state batches and new batches into a single non-overlapping batch list.
+ * Input: batchesSoFar, newBatches, startOffset
+ * Output: combined list with non-overlapping batches (finalBatchList)
+ * <p>
+ * - Add both currentBatches and newBatches into a single list combinedBatchList
+ * - Remove/prune any batches from the combinedBatchList:
+ * - if batch.lastOffset < startOffset then remove batch from combinedBatchList
+ * - else if batch.firstOffset > startOffset then we will keep the batch
+ * - else if batch.firstOffset <= startOffset <= batch.lastOffset then keep [startOffset, batch.lastOffset] part only and discard rest.
+ * - create a treeset sortedBatches using pruned combinedBatchList
+ * - do repeat until all batches in sortedSet are non-overlapping:
+ * - find first 2 overlapping batches in sortedBatches set, say, prev and candidate.
+ * - remove any non-overlapping batches from sortedBatches encountered during the find operation and add them to a finalBatchList
+ * - based on various conditions of offset overlap and batch state differences combine the batches or
+ * create new batches, if required, and add to the sortedSet.
+ * - done
+ * - return the finalBatchList
+ *
+ * @return list of {@link PersisterStateBatch} representing non-overlapping combined batches
+ */
+ public List<PersisterStateBatch> combineStateBatches() {
+ pruneBatches();
+ mergeBatches();
+ return finalBatchList;
+ }
+
+ private void mergeBatches() {
+ if (combinedBatchList.size() < 2) {
+ finalBatchList = combinedBatchList;
+ return;
+ }
+
+ sortedBatches = new TreeSet<>(combinedBatchList);
+
+ BatchOverlapState overlapState = getOverlappingState();
+
+ while (overlapState != BatchOverlapState.EMPTY) {
+ PersisterStateBatch prev = overlapState.prev();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove both previous and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(prev);
+ sortedBatches.remove(candidate);
+
+ if (compareBatchState(candidate, prev) == 0) { // same state and overlap or contiguous
+ // overlap and same state (prev.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // prev: ------ ------- ------- ------- ------- -------- -------
+ // candidate: ------ ---- ---------- --- ---- ------- -------
+ handleSameStateOverlap(prev, candidate);
+ } else { // diff state and non-contiguous overlap
Review Comment:
Hmm, if we reach here, it's possible for prev and candidate to be contiguous, right?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/PersisterStateBatchCombiner.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class PersisterStateBatchCombiner {
+ private List<PersisterStateBatch> combinedBatchList; // link between pruning and merging
+ private final long startOffset;
+ private TreeSet<PersisterStateBatch> sortedBatches;
+ private List<PersisterStateBatch> finalBatchList; // final list is built here
+
+ public PersisterStateBatchCombiner(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ initializeCombinedList(batchesSoFar, newBatches);
+ finalBatchList = new ArrayList<>(combinedBatchList.size() * 2); // heuristic size
+ this.startOffset = startOffset;
+ }
+
+ private void initializeCombinedList(List<PersisterStateBatch> batchesSoFar, List<PersisterStateBatch> newBatches) {
+ boolean soFarEmpty = batchesSoFar == null || batchesSoFar.isEmpty();
+ boolean newBatchesEmpty = newBatches == null || newBatches.isEmpty();
+
+ if (soFarEmpty && newBatchesEmpty) {
+ combinedBatchList = new ArrayList<>();
+ } else if (soFarEmpty) {
+ combinedBatchList = newBatches;
+ } else if (newBatchesEmpty) {
+ combinedBatchList = batchesSoFar;
+ } else {
+ combinedBatchList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedBatchList.addAll(batchesSoFar);
+ combinedBatchList.addAll(newBatches);
+ }
+ }
+
+ /**
+ * Algorithm: Merge current state batches and new batches into a single non-overlapping batch list.
+ * Input: batchesSoFar, newBatches, startOffset
+ * Output: combined list with non-overlapping batches (finalBatchList)
+ * <p>
+ * - Add both currentBatches and newBatches into a single list combinedBatchList
+ * - Remove/prune any batches from the combinedBatchList:
+ * - if batch.lastOffset < startOffset then remove batch from combinedBatchList
+ * - else if batch.firstOffset > startOffset then we will keep the batch
+ * - else if batch.firstOffset <= startOffset <= batch.lastOffset then keep [startOffset, batch.lastOffset] part only and discard rest.
+ * - create a treeset sortedBatches using pruned combinedBatchList
+ * - do repeat until all batches in sortedSet are non-overlapping:
+ * - find first 2 overlapping batches in sortedBatches set, say, prev and candidate.
+ * - remove any non-overlapping batches from sortedBatches encountered during the find operation and add them to a finalBatchList
+ * - based on various conditions of offset overlap and batch state differences combine the batches or
+ * create new batches, if required, and add to the sortedSet.
+ * - done
+ * - return the finalBatchList
+ *
+ * @return list of {@link PersisterStateBatch} representing non-overlapping combined batches
+ */
+ public List<PersisterStateBatch> combineStateBatches() {
+ pruneBatches();
+ mergeBatches();
+ return finalBatchList;
+ }
+
+ private void mergeBatches() {
+ if (combinedBatchList.size() < 2) {
+ finalBatchList = combinedBatchList;
+ return;
+ }
+
+ sortedBatches = new TreeSet<>(combinedBatchList);
+
+ BatchOverlapState overlapState = getOverlappingState();
+
+ while (overlapState != BatchOverlapState.EMPTY) {
+ PersisterStateBatch prev = overlapState.prev();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove both previous and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(prev);
+ sortedBatches.remove(candidate);
+
+ if (compareBatchState(candidate, prev) == 0) { // same state and overlap or contiguous
+ // overlap and same state (prev.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // prev: ------ ------- ------- ------- ------- -------- -------
+ // candidate: ------ ---- ---------- --- ---- ------- -------
+ handleSameStateOverlap(prev, candidate);
+ } else { // diff state and non-contiguous overlap
+ // overlap and diff state
+ // covers:
+ // case: 1 2* 3 4 5 6 7*
+ // prev: ------ ------- ------- ------- ------- -------- ------
+ // candidate: ------ ---- --------- ---- ---- ------- -------
+ // max batches: 1 2 2 3 2 2 2
+ // min batches: 1 1 1 1 1 2 1
+ // * not possible with treeset
+ handleDiffStateOverlap(prev, candidate);
+ }
+ overlapState = getOverlappingState();
+ }
+ finalBatchList.addAll(sortedBatches); // some non overlapping batches might have remained
+ }
+
+ /**
+ * Compares the non-offset state of 2 batches i.e. the deliveryCount and deliverState.
+ * <p>
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ *
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
+ int deltaCount = Short.compare(b1.deliveryCount(), b2.deliveryCount());
+
+ // Delivery state could be:
+ // 0 - AVAILABLE (non-terminal)
+ // 1 - ACQUIRED - should not be persisted yet
+ // 2 - ACKNOWLEDGED (terminal)
+ // 3 - ARCHIVING - not implemented in KIP-932 - non-terminal - leads only to ARCHIVED
+ // 4 - ARCHIVED (terminal)
+
+ if (deltaCount == 0) { // same delivery count
+ return Byte.compare(b1.deliveryState(), b2.deliveryState());
+ }
+ return deltaCount;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * <p>
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example:
+ * ----- ---- ----- ----- -----
+ * ------
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @return object representing the overlap state
+ */
+ private BatchOverlapState getOverlappingState() {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.EMPTY;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch prev = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= prev.lastOffset() || // overlap
+ prev.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(prev, candidate) == 0) { // contiguous
Review Comment:
It seems unintuitive to return continuous batches as overlap batches.
##########
share/src/main/java/org/apache/kafka/server/share/PersisterStateBatch.java:
##########
@@ -77,22 +77,40 @@ public boolean equals(Object o) {
PersisterStateBatch that = (PersisterStateBatch) o;
return firstOffset == that.firstOffset &&
lastOffset == that.lastOffset &&
- deliveryState == that.deliveryState &&
- deliveryCount == that.deliveryCount;
+ deliveryCount == that.deliveryCount &&
+ deliveryState == that.deliveryState;
}
@Override
public int hashCode() {
- return Objects.hash(firstOffset, lastOffset, deliveryState, deliveryCount);
+ return Objects.hash(firstOffset, lastOffset, deliveryCount, deliveryState);
}
@Override
public String toString() {
return "PersisterStateBatch(" +
"firstOffset=" + firstOffset + "," +
"lastOffset=" + lastOffset + "," +
- "deliveryState=" + deliveryState + "," +
- "deliveryCount=" + deliveryCount +
+ "deliveryCount=" + deliveryCount + "," +
+ "deliveryState=" + deliveryState +
")";
}
+
+ @Override
+ public int compareTo(Object o) {
+ PersisterStateBatch that = (PersisterStateBatch) o;
+ int deltaFirst = Long.compare(this.firstOffset(), that.firstOffset());
+ if (deltaFirst == 0) {
+ int deltaLast = Long.compare(this.lastOffset(), that.lastOffset());
+ if (deltaLast == 0) {
+ int deltaCount = this.deliveryCount() - that.deliveryCount();
Review Comment:
Should we check the deliverStates are the same before comparing deliveryCount?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/PersisterStateBatchCombiner.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class PersisterStateBatchCombiner {
+ private List<PersisterStateBatch> combinedBatchList; // link between pruning and merging
+ private final long startOffset;
+ private TreeSet<PersisterStateBatch> sortedBatches;
+ private List<PersisterStateBatch> finalBatchList; // final list is built here
+
+ public PersisterStateBatchCombiner(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ initializeCombinedList(batchesSoFar, newBatches);
+ finalBatchList = new ArrayList<>(combinedBatchList.size() * 2); // heuristic size
+ this.startOffset = startOffset;
+ }
+
+ private void initializeCombinedList(List<PersisterStateBatch> batchesSoFar, List<PersisterStateBatch> newBatches) {
+ boolean soFarEmpty = batchesSoFar == null || batchesSoFar.isEmpty();
+ boolean newBatchesEmpty = newBatches == null || newBatches.isEmpty();
+
+ if (soFarEmpty && newBatchesEmpty) {
+ combinedBatchList = new ArrayList<>();
+ } else if (soFarEmpty) {
+ combinedBatchList = newBatches;
+ } else if (newBatchesEmpty) {
+ combinedBatchList = batchesSoFar;
+ } else {
+ combinedBatchList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedBatchList.addAll(batchesSoFar);
+ combinedBatchList.addAll(newBatches);
+ }
+ }
+
+ /**
+ * Algorithm: Merge current state batches and new batches into a single non-overlapping batch list.
+ * Input: batchesSoFar, newBatches, startOffset
+ * Output: combined list with non-overlapping batches (finalBatchList)
+ * <p>
+ * - Add both currentBatches and newBatches into a single list combinedBatchList
+ * - Remove/prune any batches from the combinedBatchList:
+ * - if batch.lastOffset < startOffset then remove batch from combinedBatchList
+ * - else if batch.firstOffset > startOffset then we will keep the batch
+ * - else if batch.firstOffset <= startOffset <= batch.lastOffset then keep [startOffset, batch.lastOffset] part only and discard rest.
+ * - create a treeset sortedBatches using pruned combinedBatchList
+ * - do repeat until all batches in sortedSet are non-overlapping:
+ * - find first 2 overlapping batches in sortedBatches set, say, prev and candidate.
+ * - remove any non-overlapping batches from sortedBatches encountered during the find operation and add them to a finalBatchList
+ * - based on various conditions of offset overlap and batch state differences combine the batches or
+ * create new batches, if required, and add to the sortedSet.
+ * - done
+ * - return the finalBatchList
+ *
+ * @return list of {@link PersisterStateBatch} representing non-overlapping combined batches
+ */
+ public List<PersisterStateBatch> combineStateBatches() {
+ pruneBatches();
+ mergeBatches();
+ return finalBatchList;
+ }
+
+ private void mergeBatches() {
+ if (combinedBatchList.size() < 2) {
+ finalBatchList = combinedBatchList;
+ return;
+ }
+
+ sortedBatches = new TreeSet<>(combinedBatchList);
Review Comment:
To be consistent, we probably want to initialize sortedBatches in the constructor as combinedBatchList and finalBatchList.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/PersisterStateBatchCombiner.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class PersisterStateBatchCombiner {
+ private List<PersisterStateBatch> combinedBatchList; // link between pruning and merging
+ private final long startOffset;
+ private TreeSet<PersisterStateBatch> sortedBatches;
+ private List<PersisterStateBatch> finalBatchList; // final list is built here
+
+ public PersisterStateBatchCombiner(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ initializeCombinedList(batchesSoFar, newBatches);
+ finalBatchList = new ArrayList<>(combinedBatchList.size() * 2); // heuristic size
+ this.startOffset = startOffset;
+ }
+
+ private void initializeCombinedList(List<PersisterStateBatch> batchesSoFar, List<PersisterStateBatch> newBatches) {
+ boolean soFarEmpty = batchesSoFar == null || batchesSoFar.isEmpty();
+ boolean newBatchesEmpty = newBatches == null || newBatches.isEmpty();
+
+ if (soFarEmpty && newBatchesEmpty) {
+ combinedBatchList = new ArrayList<>();
+ } else if (soFarEmpty) {
+ combinedBatchList = newBatches;
+ } else if (newBatchesEmpty) {
+ combinedBatchList = batchesSoFar;
+ } else {
+ combinedBatchList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedBatchList.addAll(batchesSoFar);
+ combinedBatchList.addAll(newBatches);
+ }
+ }
+
+ /**
+ * Algorithm: Merge current state batches and new batches into a single non-overlapping batch list.
+ * Input: batchesSoFar, newBatches, startOffset
+ * Output: combined list with non-overlapping batches (finalBatchList)
+ * <p>
+ * - Add both currentBatches and newBatches into a single list combinedBatchList
+ * - Remove/prune any batches from the combinedBatchList:
+ * - if batch.lastOffset < startOffset then remove batch from combinedBatchList
+ * - else if batch.firstOffset > startOffset then we will keep the batch
+ * - else if batch.firstOffset <= startOffset <= batch.lastOffset then keep [startOffset, batch.lastOffset] part only and discard rest.
+ * - create a treeset sortedBatches using pruned combinedBatchList
+ * - do repeat until all batches in sortedSet are non-overlapping:
+ * - find first 2 overlapping batches in sortedBatches set, say, prev and candidate.
+ * - remove any non-overlapping batches from sortedBatches encountered during the find operation and add them to a finalBatchList
+ * - based on various conditions of offset overlap and batch state differences combine the batches or
+ * create new batches, if required, and add to the sortedSet.
+ * - done
+ * - return the finalBatchList
+ *
+ * @return list of {@link PersisterStateBatch} representing non-overlapping combined batches
+ */
+ public List<PersisterStateBatch> combineStateBatches() {
+ pruneBatches();
+ mergeBatches();
+ return finalBatchList;
+ }
+
+ private void mergeBatches() {
+ if (combinedBatchList.size() < 2) {
+ finalBatchList = combinedBatchList;
+ return;
+ }
+
+ sortedBatches = new TreeSet<>(combinedBatchList);
+
+ BatchOverlapState overlapState = getOverlappingState();
+
+ while (overlapState != BatchOverlapState.EMPTY) {
+ PersisterStateBatch prev = overlapState.prev();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove both previous and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(prev);
+ sortedBatches.remove(candidate);
+
+ if (compareBatchState(candidate, prev) == 0) { // same state and overlap or contiguous
+ // overlap and same state (prev.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // prev: ------ ------- ------- ------- ------- -------- -------
+ // candidate: ------ ---- ---------- --- ---- ------- -------
+ handleSameStateOverlap(prev, candidate);
+ } else { // diff state and non-contiguous overlap
+ // overlap and diff state
+ // covers:
+ // case: 1 2* 3 4 5 6 7*
+ // prev: ------ ------- ------- ------- ------- -------- ------
+ // candidate: ------ ---- --------- ---- ---- ------- -------
+ // max batches: 1 2 2 3 2 2 2
+ // min batches: 1 1 1 1 1 2 1
+ // * not possible with treeset
+ handleDiffStateOverlap(prev, candidate);
+ }
+ overlapState = getOverlappingState();
+ }
+ finalBatchList.addAll(sortedBatches); // some non overlapping batches might have remained
+ }
+
+ /**
+ * Compares the non-offset state of 2 batches i.e. the deliveryCount and deliverState.
+ * <p>
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ *
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
Should we just use `PersisterStateBatch.compareTo` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1779349368
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/PersisterStateBatchCombiner.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class PersisterStateBatchCombiner {
+ private List<PersisterStateBatch> combinedBatchList; // link between pruning and merging
+ private final long startOffset;
+ private TreeSet<PersisterStateBatch> sortedBatches;
+ private List<PersisterStateBatch> finalBatchList; // final list is built here
+
+ public PersisterStateBatchCombiner(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ initializeCombinedList(batchesSoFar, newBatches);
+ finalBatchList = new ArrayList<>(combinedBatchList.size() * 2); // heuristic size
+ this.startOffset = startOffset;
+ }
+
+ private void initializeCombinedList(List<PersisterStateBatch> batchesSoFar, List<PersisterStateBatch> newBatches) {
+ boolean soFarEmpty = batchesSoFar == null || batchesSoFar.isEmpty();
+ boolean newBatchesEmpty = newBatches == null || newBatches.isEmpty();
+
+ if (soFarEmpty && newBatchesEmpty) {
+ combinedBatchList = new ArrayList<>();
+ } else if (soFarEmpty) {
+ combinedBatchList = newBatches;
+ } else if (newBatchesEmpty) {
+ combinedBatchList = batchesSoFar;
+ } else {
+ combinedBatchList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedBatchList.addAll(batchesSoFar);
+ combinedBatchList.addAll(newBatches);
+ }
+ }
+
+ /**
+ * Algorithm: Merge current state batches and new batches into a single non-overlapping batch list.
+ * Input: batchesSoFar, newBatches, startOffset
+ * Output: combined list with non-overlapping batches (finalBatchList)
+ * <p>
+ * - Add both currentBatches and newBatches into a single list combinedBatchList
+ * - Remove/prune any batches from the combinedBatchList:
+ * - if batch.lastOffset < startOffset then remove batch from combinedBatchList
+ * - else if batch.firstOffset > startOffset then we will keep the batch
+ * - else if batch.firstOffset <= startOffset <= batch.lastOffset then keep [startOffset, batch.lastOffset] part only and discard rest.
+ * - create a treeset sortedBatches using pruned combinedBatchList
+ * - do repeat until all batches in sortedSet are non-overlapping:
+ * - find first 2 overlapping batches in sortedBatches set, say, prev and candidate.
+ * - remove any non-overlapping batches from sortedBatches encountered during the find operation and add them to a finalBatchList
+ * - based on various conditions of offset overlap and batch state differences combine the batches or
+ * create new batches, if required, and add to the sortedSet.
+ * - done
+ * - return the finalBatchList
+ *
+ * @return list of {@link PersisterStateBatch} representing non-overlapping combined batches
+ */
+ public List<PersisterStateBatch> combineStateBatches() {
+ pruneBatches();
+ mergeBatches();
+ return finalBatchList;
+ }
+
+ private void mergeBatches() {
+ if (combinedBatchList.size() < 2) {
+ finalBatchList = combinedBatchList;
+ return;
+ }
+
+ sortedBatches = new TreeSet<>(combinedBatchList);
+
+ BatchOverlapState overlapState = getOverlappingState();
+
+ while (overlapState != BatchOverlapState.EMPTY) {
+ PersisterStateBatch prev = overlapState.prev();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove both previous and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(prev);
+ sortedBatches.remove(candidate);
+
+ if (compareBatchState(candidate, prev) == 0) { // same state and overlap or contiguous
+ // overlap and same state (prev.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // prev: ------ ------- ------- ------- ------- -------- -------
+ // candidate: ------ ---- ---------- --- ---- ------- -------
+ handleSameStateOverlap(prev, candidate);
+ } else { // diff state and non-contiguous overlap
+ // overlap and diff state
+ // covers:
+ // case: 1 2* 3 4 5 6 7*
+ // prev: ------ ------- ------- ------- ------- -------- ------
+ // candidate: ------ ---- --------- ---- ---- ------- -------
+ // max batches: 1 2 2 3 2 2 2
+ // min batches: 1 1 1 1 1 2 1
+ // * not possible with treeset
+ handleDiffStateOverlap(prev, candidate);
+ }
+ overlapState = getOverlappingState();
+ }
+ finalBatchList.addAll(sortedBatches); // some non overlapping batches might have remained
+ }
+
+ /**
+ * Compares the non-offset state of 2 batches i.e. the deliveryCount and deliverState.
+ * <p>
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ *
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
+ int deltaCount = Short.compare(b1.deliveryCount(), b2.deliveryCount());
+
+ // Delivery state could be:
+ // 0 - AVAILABLE (non-terminal)
+ // 1 - ACQUIRED - should not be persisted yet
+ // 2 - ACKNOWLEDGED (terminal)
+ // 3 - ARCHIVING - not implemented in KIP-932 - non-terminal - leads only to ARCHIVED
+ // 4 - ARCHIVED (terminal)
+
+ if (deltaCount == 0) { // same delivery count
+ return Byte.compare(b1.deliveryState(), b2.deliveryState());
+ }
+ return deltaCount;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * <p>
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example:
+ * ----- ---- ----- ----- -----
+ * ------
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @return object representing the overlap state
+ */
+ private BatchOverlapState getOverlappingState() {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.EMPTY;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch prev = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= prev.lastOffset() || // overlap
+ prev.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(prev, candidate) == 0) { // contiguous
Review Comment:
Only continuous batches with exactly same state are returned here.
We can perhaps rename the method to `getMergeCandidates()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1756270704
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,195 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ // will take care of overlapping batches
+ Queue<PersisterStateBatch> batchQueue = new LinkedList<>(
Review Comment:
Given that this is processing the batches we have so far, I'm not convinced that it is worth merging and pruning at this point. I understand that the start offset might have changed, but surely it's already optimised by the previous pass through.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,195 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ // will take care of overlapping batches
+ Queue<PersisterStateBatch> batchQueue = new LinkedList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(batchesSoFar),
+ startOffset
+ )
+ ));
+
+ // will take care of overlapping batches
+ List<PersisterStateBatch> modifiedNewBatches = new ArrayList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(newBatches),
+ startOffset
+ )
+ ));
+
+ for (PersisterStateBatch batch : modifiedNewBatches) {
+ for (int i = 0; i < batchQueue.size(); i++) {
+ PersisterStateBatch cur = batchQueue.poll();
+ // cur batch under inspection has no overlap with the new one
+ // we will need to add to our result
+ if (batch.lastOffset() < cur.firstOffset() || batch.firstOffset() > cur.lastOffset()) {
+ batchQueue.add(cur);
+ continue;
+ }
+
+ // Covers cases where we need to create a new interval
+ // from the current one such that they do not
+ // overlap with the new one.
+ // Following cases will not produce any new records so need not be handled.
+ // cur: ____ ______ ______ _____
+ // new: ________ ______ _________ _________
+
+
+ // covers
+ // cur: ______ _____ _____ ______
+ // batch: ___ _____ ___ ___
+ if (batch.firstOffset() >= cur.firstOffset() && batch.lastOffset() <= cur.lastOffset()) {
+ // extra batch needs to be created
+ if (batch.firstOffset() > cur.firstOffset()) {
+ batchQueue.add(
+ new PersisterStateBatch(
+ cur.firstOffset(),
+ batch.firstOffset() - 1,
+ cur.deliveryState(),
+ cur.deliveryCount()
+ )
+ );
+ }
+
+ // extra batch needs to be created
+ if (batch.lastOffset() < cur.lastOffset()) {
+ batchQueue.add(
+ new PersisterStateBatch(
+ batch.lastOffset() + 1,
+ cur.lastOffset(),
+ cur.deliveryState(),
+ cur.deliveryCount()
+ )
+ );
+ }
+ } else if (batch.firstOffset() < cur.firstOffset() && batch.lastOffset() < cur.lastOffset()) {
+ // covers
+ // ______
Review Comment:
I like the little diagrams. I suggest adding the `cur:` and `batch:` to these on lines 630 and 640 too.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,195 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ // will take care of overlapping batches
+ Queue<PersisterStateBatch> batchQueue = new LinkedList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(batchesSoFar),
+ startOffset
+ )
+ ));
+
+ // will take care of overlapping batches
+ List<PersisterStateBatch> modifiedNewBatches = new ArrayList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(newBatches),
+ startOffset
+ )
+ ));
+
+ for (PersisterStateBatch batch : modifiedNewBatches) {
+ for (int i = 0; i < batchQueue.size(); i++) {
+ PersisterStateBatch cur = batchQueue.poll();
+ // cur batch under inspection has no overlap with the new one
+ // we will need to add to our result
+ if (batch.lastOffset() < cur.firstOffset() || batch.firstOffset() > cur.lastOffset()) {
+ batchQueue.add(cur);
+ continue;
+ }
+
+ // Covers cases where we need to create a new interval
+ // from the current one such that they do not
+ // overlap with the new one.
+ // Following cases will not produce any new records so need not be handled.
+ // cur: ____ ______ ______ _____
+ // new: ________ ______ _________ _________
+
+
+ // covers
+ // cur: ______ _____ _____ ______
Review Comment:
The second case here overlaps with the second case above (matching first and last offsets). Logically that might be fine in the code, but it looks like a contradiction in the comments.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,195 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ // will take care of overlapping batches
+ Queue<PersisterStateBatch> batchQueue = new LinkedList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(batchesSoFar),
+ startOffset
+ )
+ ));
+
+ // will take care of overlapping batches
+ List<PersisterStateBatch> modifiedNewBatches = new ArrayList<>(
+ mergeBatches(
+ pruneBatches(
+ getSortedList(newBatches),
+ startOffset
+ )
+ ));
+
+ for (PersisterStateBatch batch : modifiedNewBatches) {
+ for (int i = 0; i < batchQueue.size(); i++) {
+ PersisterStateBatch cur = batchQueue.poll();
+ // cur batch under inspection has no overlap with the new one
+ // we will need to add to our result
+ if (batch.lastOffset() < cur.firstOffset() || batch.firstOffset() > cur.lastOffset()) {
+ batchQueue.add(cur);
+ continue;
+ }
+
+ // Covers cases where we need to create a new interval
+ // from the current one such that they do not
+ // overlap with the new one.
+ // Following cases will not produce any new records so need not be handled.
+ // cur: ____ ______ ______ _____
+ // new: ________ ______ _________ _________
Review Comment:
Probably ought to use `batch:` in the diagram. For these cases, I think the idea is that `cur` is superseded by `batch`, so `batchQueue.add(batch)` at line 650 is sufficient to add the new batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1763696711
##########
server-common/src/main/java/org/apache/kafka/server/group/share/PersisterStateBatch.java:
##########
@@ -95,4 +95,22 @@ public String toString() {
"deliveryCount=" + deliveryCount +
Review Comment:
firstOffset, lastOffset, deliveryCount and deliveryState is the order, so probably ought to have them in that order in the toString too.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,331 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ // No special handling for case 1
+ // here candidate can never have lower priority
+ // since the treeset order takes that into account
+ // and candidate is already in the treeset.
+
+ if (candidate.lastOffset() < last.lastOffset()) { // case 2
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
Review Comment:
Doesn't `candidate` need to be removed in this case? `last` "won".
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,331 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ // No special handling for case 1
Review Comment:
OK, but I suppose case 1 ends up going down the branch at line 708 then. Personally, I would prefer an if test to eliminate case 1 up front.
##########
server-common/src/main/java/org/apache/kafka/server/group/share/PersisterStateBatch.java:
##########
@@ -95,4 +95,22 @@ public String toString() {
"deliveryCount=" + deliveryCount +
")";
}
+
+ @Override
+ public int compareTo(Object o) {
+ PersisterStateBatch that = (PersisterStateBatch) o;
+ int deltaFirst = Long.compare(this.firstOffset(), that.firstOffset());
+ if (deltaFirst == 0) {
+ int deltaLast = Long.compare(this.lastOffset(), that.lastOffset());
+ if (deltaLast == 0) {
+ int countDelta = this.deliveryCount() - that.deliveryCount();
Review Comment:
Surely `deltaCount` given the pattern you established for all of the other comparisons.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,331 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ // No special handling for case 1
+ // here candidate can never have lower priority
+ // since the treeset order takes that into account
+ // and candidate is already in the treeset.
+
+ if (candidate.lastOffset() < last.lastOffset()) { // case 2
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.firstOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else { // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.lastOffset()
Review Comment:
Shouldn't this `candidate.firstOffset() > last.firstOffset()` in the comment?
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,331 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
Review Comment:
nit: Missing ')' in your comment ;)
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,331 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ // No special handling for case 1
+ // here candidate can never have lower priority
+ // since the treeset order takes that into account
+ // and candidate is already in the treeset.
+
+ if (candidate.lastOffset() < last.lastOffset()) { // case 2
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.firstOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else { // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.remove(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
Review Comment:
This line is redundant. Before line 678, both `last` and `candidate` are in the set. You remove `last` at 678, and then `candidate` "wins". It's already in the set. It doesn't need adding. I know that TreeSet will handle it. It's still a waste of effort and logically unnecessary.
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,331 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ // No special handling for case 1
+ // here candidate can never have lower priority
+ // since the treeset order takes that into account
+ // and candidate is already in the treeset.
+
+ if (candidate.lastOffset() < last.lastOffset()) { // case 2
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else { // last has lower priority
+ sortedBatches.add(candidate);
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.firstOffset(),
Review Comment:
Should be `last.lastOffset()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1771835934
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,322 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
.build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 (NP) 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
Review Comment:
Yes that true, have kept the example for completeness sake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1779349855
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/PersisterStateBatchCombiner.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class PersisterStateBatchCombiner {
+ private List<PersisterStateBatch> combinedBatchList; // link between pruning and merging
+ private final long startOffset;
+ private TreeSet<PersisterStateBatch> sortedBatches;
+ private List<PersisterStateBatch> finalBatchList; // final list is built here
+
+ public PersisterStateBatchCombiner(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ initializeCombinedList(batchesSoFar, newBatches);
+ finalBatchList = new ArrayList<>(combinedBatchList.size() * 2); // heuristic size
+ this.startOffset = startOffset;
+ }
+
+ private void initializeCombinedList(List<PersisterStateBatch> batchesSoFar, List<PersisterStateBatch> newBatches) {
+ boolean soFarEmpty = batchesSoFar == null || batchesSoFar.isEmpty();
+ boolean newBatchesEmpty = newBatches == null || newBatches.isEmpty();
+
+ if (soFarEmpty && newBatchesEmpty) {
+ combinedBatchList = new ArrayList<>();
+ } else if (soFarEmpty) {
+ combinedBatchList = newBatches;
+ } else if (newBatchesEmpty) {
+ combinedBatchList = batchesSoFar;
+ } else {
+ combinedBatchList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedBatchList.addAll(batchesSoFar);
+ combinedBatchList.addAll(newBatches);
+ }
+ }
+
+ /**
+ * Algorithm: Merge current state batches and new batches into a single non-overlapping batch list.
+ * Input: batchesSoFar, newBatches, startOffset
+ * Output: combined list with non-overlapping batches (finalBatchList)
+ * <p>
+ * - Add both currentBatches and newBatches into a single list combinedBatchList
+ * - Remove/prune any batches from the combinedBatchList:
+ * - if batch.lastOffset < startOffset then remove batch from combinedBatchList
+ * - else if batch.firstOffset > startOffset then we will keep the batch
+ * - else if batch.firstOffset <= startOffset <= batch.lastOffset then keep [startOffset, batch.lastOffset] part only and discard rest.
+ * - create a treeset sortedBatches using pruned combinedBatchList
+ * - do repeat until all batches in sortedSet are non-overlapping:
+ * - find first 2 overlapping batches in sortedBatches set, say, prev and candidate.
+ * - remove any non-overlapping batches from sortedBatches encountered during the find operation and add them to a finalBatchList
+ * - based on various conditions of offset overlap and batch state differences combine the batches or
+ * create new batches, if required, and add to the sortedSet.
+ * - done
+ * - return the finalBatchList
+ *
+ * @return list of {@link PersisterStateBatch} representing non-overlapping combined batches
+ */
+ public List<PersisterStateBatch> combineStateBatches() {
+ pruneBatches();
+ mergeBatches();
+ return finalBatchList;
+ }
+
+ private void mergeBatches() {
+ if (combinedBatchList.size() < 2) {
+ finalBatchList = combinedBatchList;
+ return;
+ }
+
+ sortedBatches = new TreeSet<>(combinedBatchList);
+
+ BatchOverlapState overlapState = getOverlappingState();
+
+ while (overlapState != BatchOverlapState.EMPTY) {
+ PersisterStateBatch prev = overlapState.prev();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove both previous and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(prev);
+ sortedBatches.remove(candidate);
+
+ if (compareBatchState(candidate, prev) == 0) { // same state and overlap or contiguous
+ // overlap and same state (prev.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // prev: ------ ------- ------- ------- ------- -------- -------
+ // candidate: ------ ---- ---------- --- ---- ------- -------
+ handleSameStateOverlap(prev, candidate);
+ } else { // diff state and non-contiguous overlap
Review Comment:
No it's not.
getOverlappingState returns continuous pair ONLY IF the pair same delivery count and state.
The condition in the if block will capture continuous case for same state.
If it reaches else it has to be overlapping and different state. The getOverlappingState does not return continuous and different state pairs.
In fact, it updates the treeset by removing any which are found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1777017514
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
I think ARCHIVING is not implemented in KIP_932, and that's because it's going to be in the DLQ KIP to follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on PR #17149:
URL: https://github.com/apache/kafka/pull/17149#issuecomment-2340558994
> I can think of some more permutations.
>
> What if some POSB was updated with multiple POSB, such as `{(111-120, 0, 1)}` updated with `{(111-113, 0, 2),(114-114, 2, 1), (115-119, 0, 2)}`?
>
> What if some POSB was updated with a partially overlapping POSB, such as `{(111-120, 0, 1)}` updated with `{(111-113, 0, 2)}`?
This is in keeping with the idea that SCR will not create new intervals (either due to partial interval state change or combining sequential intervals). This is fitting because the persister is a "dumb" component.
It simply maintains the intervals which callers send it in the order of arrival and prunes intervals which are expired due to start offset movement.
So,
- in this example: `{(111-120, 0, 1)} updated with {(111-113, 0, 2),(114-114, 2, 1), (115-119, 0, 2)}` - the dumb persister will keep all of the intervals. If start offset moves, it will prune the ones such that `lastOffset < startOffset` thereby not continuously increasing in size. If we add more intelligence, we will create `(119-120, 0, 1)` which was never sent.
- for this `(111-120, 0, 1)} updated with {(111-113, 0, 2)}` , it'll be the same situation - otherwise we are creating a new interval `(114-120, 0, 1)` which was never sent.
Both the situations are handled in the test case:
`Sets have partial overlap => result list contains batches from both sets (we do not add gaps).`
and `New set batch is contained in cur set => result list contains batches from both sets (we do not add gaps).`
The caller (ShareParition) is anyway doing the work of manipulating this information and creating the in memory state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on PR #17149:
URL: https://github.com/apache/kafka/pull/17149#issuecomment-2358244258
For better understanding, I will remove both the candidate and last from the sortedBatches and add the appropriate one whenever required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1775462423
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
Review Comment:
nit: formatting in the java doc (add blank line between paragraphs). same comment for other java docs throughout -- missing the blank line makes it harder to read
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776430833
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
We use the offsets to determine the relative positioning and then use just that state to determine how to break the batches. Based on state, overlapping batches could be merged or broken down.
example:
```
1.
------ [1,10,0,1]
------- [5,15,0,1]
=>
---------- [1,15,0,1]
2.
------ [1,10,0,1]
------- [5,15,2,1]
=>
---- [1,4,0,1]
------- [5,15,2,1]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1757064893
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/ShareCoordinatorShardTest.java:
##########
@@ -791,6 +793,304 @@ public void testNonSequentialBatchUpdates() {
verify(shard.getMetricsShard(), times(3)).record(ShareCoordinatorMetrics.SHARE_COORDINATOR_WRITE_SENSOR_NAME);
}
+ @Test
+ public void testStateBatchCombine() {
Review Comment:
But parametrization will not fail specific scenario which is often easy to debug and fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1759206937
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
Review Comment:
yes, there is only one set - sortedBatches from which we find the first overlapping pair and call the members last and candidate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776567400
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
Review Comment:
As per the algorithm, we need the non-overalapping prefix to be removed from the treeset so this might not work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776452137
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
Review Comment:
No the logic is not that simple. A single iteration over the sorted batches might not be enough.
The invariant is that the batches remain sorted even after manipulation
Consider:
```
-------- A [1,10,0,1]
--- B [5,7,0,2]
-------------- C [5,15,0,3]
```
A and B will combine to
```
---- [1,4,0,1]
---- [5,7,0,2]
--- [8,10,0,1]
```
Now when combining with C, we have 2 previous batches to consider.
Secondly,
```
----------- A [1,10,0,1]
--- B [5,7,0,2]
--- C [5,7,0,3]
```
A and B will combine to
```
---- [1,4,0,1]
---- [5,7,0,2]
--- [8,10,0,1]
--- <- C - we broke invariant for being sorted by batches
```
In the current impl, these situations are implicitly handled by virtue of the treeset. Any newly generated batches are pushed back into the treeset and the `getOverlappingState` method finds the first overlapping pair as well as returns the non-overlapping prefix.
The non-overlapping prefix is then REMOVED from the treeset hence, once a batch is no longer overlapping, it is only looked at once guaranteeing running time efficiency.
@AndrewJSchofield
@mumrah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1779348983
##########
share/src/main/java/org/apache/kafka/server/share/PersisterStateBatch.java:
##########
@@ -77,22 +77,40 @@ public boolean equals(Object o) {
PersisterStateBatch that = (PersisterStateBatch) o;
return firstOffset == that.firstOffset &&
lastOffset == that.lastOffset &&
- deliveryState == that.deliveryState &&
- deliveryCount == that.deliveryCount;
+ deliveryCount == that.deliveryCount &&
+ deliveryState == that.deliveryState;
}
@Override
public int hashCode() {
- return Objects.hash(firstOffset, lastOffset, deliveryState, deliveryCount);
+ return Objects.hash(firstOffset, lastOffset, deliveryCount, deliveryState);
}
@Override
public String toString() {
return "PersisterStateBatch(" +
"firstOffset=" + firstOffset + "," +
"lastOffset=" + lastOffset + "," +
- "deliveryState=" + deliveryState + "," +
- "deliveryCount=" + deliveryCount +
+ "deliveryCount=" + deliveryCount + "," +
+ "deliveryState=" + deliveryState +
")";
}
+
+ @Override
+ public int compareTo(Object o) {
+ PersisterStateBatch that = (PersisterStateBatch) o;
+ int deltaFirst = Long.compare(this.firstOffset(), that.firstOffset());
+ if (deltaFirst == 0) {
+ int deltaLast = Long.compare(this.lastOffset(), that.lastOffset());
+ if (deltaLast == 0) {
+ int deltaCount = this.deliveryCount() - that.deliveryCount();
Review Comment:
No, this was a concious decision:
Plz check discussion https://github.com/apache/kafka/pull/17149#discussion_r1777003384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776624490
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
Review Comment:
Yes, we do. But that logic is part of `mergeBatches` main loop. This method only looks at the offsets. However, the treeset it is looking at sort in the order
firstOffset --> lastOffset --> deliveryCount --> deliveryState
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776624490
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
Review Comment:
Yes, we do. But that logic is part of `mergeBatches` main loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776956765
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/StateBatchUtilTest.java:
##########
@@ -60,36 +60,60 @@ static class BatchTestHolder {
this.shouldRun = shouldRun;
}
+ static List<PersisterStateBatch> singleBatch(
+ long firstOffset,
+ long lastOffset,
+ byte deliveryState,
+ short deliveryCount
Review Comment:
I would let these be `int` so we can avoid casting in the test code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1777003384
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
The idea behind delivery count taking precedence is because it has the connotation of epoch. Higher number => higher chance of it being fresher.
If 2 batches have same delivery count then some states which are considered terminal like ACK or ARCHIVED take precedence. Since the ordinal in the delivery state enum is numerically higher for these, they blend in well.
Betweek ACK and ARCHIVED, it is acceptable to take any (we take ARCHIVED).
HOWEVER, ARCHIVED state in not implemented in kip-932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776991897
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/StateBatchUtil.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.coordinator.share.generated.ShareUpdateValue;
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.TreeSet;
+
+public class StateBatchUtil {
+ /**
+ * Util method which takes in 2 lists containing {@link PersisterStateBatch}
+ * and the startOffset.
+ * This method removes any batches where the lastOffset < startOffset, if the startOffset > -1.
+ * It then merges any contiguous intervals with same state. If states differ,
+ * based on various conditions it creates new non-overlapping batches preferring new ones.
+ * @param batchesSoFar - List containing current soft state of {@link PersisterStateBatch}
+ * @param newBatches - List containing {@link PersisterStateBatch} in incoming request
+ * @param startOffset - startOffset to consider when removing old batches.
+ * @return List containing combined batches
+ */
+ public static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
+ long startOffset
+ ) {
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ /**
+ * Encapsulates the main merge algorithm. Consider 2 batches (A, B):
+ * - Same state (delivery count and state)
+ * - If overlapping - merge into single batch
+ * - If contiguous (A.lastOffset + 1 == B.firstOffset) - merge batches into a single 1
+ * - Different state (delivery count or state differ)
+ * - Based on various cases:
+ * - swallow lower priority batch within bounds of offsets
+ * - break batch into other non-overlapping batches
+ * @param batches - List of {@link PersisterStateBatch}
+ * @return List of non-overlapping {@link PersisterStateBatch}
+ */
+ private static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ overlapState = BatchOverlapState.SENTINEL;
+ continue;
+ }
+
+ // remove both last and candidate for easier
+ // assessment about adding batches to sortedBatches
+ sortedBatches.remove(last);
+ sortedBatches.remove(candidate);
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset) due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non-contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2* 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+ // * not possible with treeset
+
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ // candidate can never have lower or equal priority
+ // since sortedBatches order takes that into account.
+ sortedBatches.add(candidate);
+ } else {
+ // case 2 is not possible with TreeSet. It is symmetric to case 3.
+ // case 3
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate priority is >= last
+ sortedBatches.add(candidate);
+ }
+ }
+ } else { // candidate.firstOffset() > last.firstOffset()
+ if (candidate.lastOffset() < last.lastOffset()) { // case 4
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+
+ sortedBatches.add(new PersisterStateBatch(
+ candidate.lastOffset() + 1,
+ last.lastOffset(),
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+ }
+ } else if (candidate.lastOffset() == last.lastOffset()) { // case 5
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+ } else {
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ } else { // case 6
+ if (compareBatchState(candidate, last) < 0) {
+ sortedBatches.add(last);
+
+ sortedBatches.add(new PersisterStateBatch(
+ last.lastOffset() + 1,
+ candidate.lastOffset(),
+ candidate.deliveryState(),
+ candidate.deliveryCount()
+ ));
+ } else {
+ // candidate has higher priority
+ sortedBatches.add(new PersisterStateBatch(
+ last.firstOffset(),
+ candidate.firstOffset() - 1,
+ last.deliveryState(),
+ last.deliveryCount()
+ ));
+
+ sortedBatches.add(candidate);
+ }
+ }
+ }
+ }
+ overlapState = getOverlappingState(sortedBatches);
+ }
+ finalBatches.addAll(sortedBatches); // some non overlapping batches might have remained
+ return finalBatches;
+ }
+
+ /**
+ * Accepts a sorted set of state batches and finds the first 2 batches which overlap.
+ * Overlap means that they have some offsets in common or, they are contiguous with the same state.
+ * Along with the 2 overlapping batches, also returns a list of non overlapping intervals
+ * prefixing them. For example
+ * _____ ____ _____ _____ _____
+ * ______ __
+ * <---------------> <-------->
+ * non-overlapping 1st overlapping pair
+ *
+ * @param sortedBatches - TreeSet representing sorted set of {@link PersisterStateBatch}
+ * @return object of {@link BatchOverlapState} representing overlapping pair and non-overlapping prefix
+ */
+ private static BatchOverlapState getOverlappingState(TreeSet<PersisterStateBatch> sortedBatches) {
+ if (sortedBatches == null || sortedBatches.isEmpty()) {
+ return BatchOverlapState.SENTINEL;
+ }
+ Iterator<PersisterStateBatch> iter = sortedBatches.iterator();
+ PersisterStateBatch last = iter.next();
+ List<PersisterStateBatch> nonOverlapping = new ArrayList<>(sortedBatches.size());
+ while (iter.hasNext()) {
+ PersisterStateBatch candidate = iter.next();
+ if (candidate.firstOffset() <= last.lastOffset() || // overlap
+ last.lastOffset() + 1 == candidate.firstOffset() && compareBatchState(last, candidate) == 0) { // contiguous
+ return new BatchOverlapState(
+ last,
+ candidate,
+ nonOverlapping
+ );
+ }
+ nonOverlapping.add(last);
+ last = candidate;
+ }
+ return new BatchOverlapState(null, null, nonOverlapping);
+ }
+
+ /**
+ * Compares the state of 2 batches i.e. the deliveryCount and deliverState.
+ * Uses standard compareTo contract x < y => +int, x > y => -int, x == y => 0
+ * @param b1 - {@link PersisterStateBatch} to compare
+ * @param b2 - {@link PersisterStateBatch} to compare
+ * @return int representing comparison result.
+ */
+ private static int compareBatchState(PersisterStateBatch b1, PersisterStateBatch b2) {
Review Comment:
Ok, so we first compare on delivery count and then break ties with delivery state. Why does the delivery count have higher precedence? Intuitively, I would assume the delivery state would be higher precedence than the count.
Actually, is it even possible to merge batches with a different delivery state?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. New merge batches algorithm. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1776535621
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/StateBatchUtilTest.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.share;
+
+import org.apache.kafka.server.share.PersisterStateBatch;
+
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class StateBatchUtilTest {
+ static class BatchTestHolder {
+ final String testName;
+ final List<PersisterStateBatch> curList;
+ final List<PersisterStateBatch> newList;
+ final List<PersisterStateBatch> expectedResult;
+ final long startOffset;
+ final boolean shouldRun;
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset) {
+ this(testName, curList, newList, expectedResult, startOffset, true);
+ }
+
+ BatchTestHolder(String testName,
+ List<PersisterStateBatch> curList,
+ List<PersisterStateBatch> newList,
+ List<PersisterStateBatch> expectedResult,
+ long startOffset,
+ boolean shouldRun) {
+ this.testName = testName;
+ this.curList = curList;
+ this.newList = newList;
+ this.expectedResult = expectedResult;
+ this.startOffset = startOffset;
+ this.shouldRun = shouldRun;
+ }
+
+ @Override
+ public String toString() {
+ return this.testName;
+ }
+ }
+
+ @SuppressWarnings({"MethodLength"})
+ private static Stream<BatchTestHolder> generator() {
+ return Stream.of(
+ new BatchTestHolder(
+ "Current batches with start offset midway are pruned.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "New batches with start offset midway are pruned.",
+ Collections.emptyList(),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 130, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(120, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ new BatchTestHolder(
+ "Both current and new batches empty.",
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ 120
+ ),
+
+ // same state
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first and last offset.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first offset, candidate last offset strictly smaller.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Last and candidate have same first offset, candidate last offset strictly larger.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 108, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset equal to last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset strictly larger and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Same state. Candidate first offset is last first offset + 1 (contiguous).",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ // different states
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset and last offset match last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset and last offset match last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset same and last offset smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 2),
+ new PersisterStateBatch(106, 110, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset same and last offset smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 105, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset same and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset same and last offset strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 3),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 1, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 110, (byte) 1, (short) 1),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset strictly larger and last offset strictly smaller than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 1, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 115, (byte) 1, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first offset strictly larger and last offset same as last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first offset strictly larger and last offset same as last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate higher state. Candidate first and last offsets strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 2)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 104, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Candidate lower state. Candidate first and last offsets strictly larger than last.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2)
+ ),
+ Collections.singletonList(
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 2),
+ new PersisterStateBatch(111, 115, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ // random cases
+ new BatchTestHolder(
+ "Handle overlapping batches within each list.",
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(121, 130, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1), // overlap with 1st batch
+ new PersisterStateBatch(123, 125, (byte) 0, (short) 1) // overlap with 2nd batch
+
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 119, (byte) 2, (short) 2),
+ new PersisterStateBatch(116, 123, (byte) 2, (short) 2) // overlap with first batch
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(111, 123, (byte) 2, (short) 2),
+ new PersisterStateBatch(124, 130, (byte) 0, (short) 1)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Handle overlapping batches with different priority.",
+ Collections.singletonList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1)
+
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(101, 105, (byte) 1, (short) 2),
+ new PersisterStateBatch(101, 115, (byte) 2, (short) 2),
+ new PersisterStateBatch(101, 120, (byte) 3, (short) 2)
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(100, 100, (byte) 0, (short) 1),
+ new PersisterStateBatch(101, 120, (byte) 3, (short) 2)
+ ),
+ -1
+ ),
+
+ new BatchTestHolder(
+ "Handle overlapping batches within each list with pruning.",
+ Arrays.asList(
+ new PersisterStateBatch(100, 110, (byte) 0, (short) 1),
+ new PersisterStateBatch(121, 130, (byte) 0, (short) 1),
+ new PersisterStateBatch(105, 115, (byte) 0, (short) 1) // overlap with 1st batch
+ ), //[(100-115, 0, 1), (121-130, 0, 1)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 119, (byte) 2, (short) 2),
+ new PersisterStateBatch(116, 123, (byte) 2, (short) 2) // overlap with first batch
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(120, 123, (byte) 2, (short) 2),
+ new PersisterStateBatch(124, 130, (byte) 0, (short) 1)
+ ),
+ 120
+ ),
+
+ // complex tests
+ new BatchTestHolder(
+ "Multiple higher state batch updates.",
+ Collections.singletonList(
+ new PersisterStateBatch(111, 120, (byte) 0, (short) 1)
+ ),
+ Arrays.asList(
+ new PersisterStateBatch(111, 113, (byte) 0, (short) 2),
+ new PersisterStateBatch(114, 114, (byte) 2, (short) 1),
+ new PersisterStateBatch(115, 119, (byte) 0, (short) 2)
+ ), //[(111-123, 2, 2)]
+ Arrays.asList(
+ new PersisterStateBatch(111, 113, (byte) 0, (short) 2),
+ new PersisterStateBatch(114, 114, (byte) 2, (short) 1),
+ new PersisterStateBatch(115, 119, (byte) 0, (short) 2),
+ new PersisterStateBatch(120, 120, (byte) 0, (short) 1)
+ ),
+ -1
+ )
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("generator")
+ public void testStateBatchCombine(BatchTestHolder test) {
Review Comment:
will break generator - test will exactly the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on PR #17149:
URL: https://github.com/apache/kafka/pull/17149#issuecomment-2340745790
> > I can think of some more permutations.
> > What if some POSB was updated with multiple POSB, such as `{(111-120, 0, 1)}` updated with `{(111-113, 0, 2),(114-114, 2, 1), (115-119, 0, 2)}`?
> > What if some POSB was updated with a partially overlapping POSB, such as `{(111-120, 0, 1)}` updated with `{(111-113, 0, 2)}`?
>
> This is in keeping with the idea that SCR will not create new intervals (either due to partial interval state change or combining sequential intervals). This is fitting because the persister is a "dumb" component. It simply maintains the intervals which callers send it in the order of arrival and prunes intervals which are expired due to start offset movement. So,
>
> * in this example: `{(111-120, 0, 1)} updated with {(111-113, 0, 2),(114-114, 2, 1), (115-119, 0, 2)}` - the dumb persister will keep all of the intervals. If start offset moves, it will prune the ones such that `lastOffset < startOffset` thereby not continuously increasing in size. If we add more intelligence, we will create `(119-120, 0, 1)` which was never sent.
> * for this `(111-120, 0, 1)} updated with {(111-113, 0, 2)}` , it'll be the same situation - otherwise we are creating a new interval `(114-120, 0, 1)` which was never sent.
> Both the situations are handled in the test case:
> `Sets have partial overlap => result list contains batches from both sets (we do not add gaps).`
> and `New set batch is contained in cur set => result list contains batches from both sets (we do not add gaps).`
>
> The caller (ShareParition) is anyway doing the work of manipulating this information and creating the in memory state.
This is fine with me in principle. We just need to ensure that @apoorvmittal10 and @adixitconfluent are aligned too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1759221583
##########
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorShard.java:
##########
@@ -537,38 +539,308 @@ private static ShareGroupOffset merge(ShareGroupOffset soFar, ShareUpdateValue n
.setStartOffset(newStartOffset)
.setLeaderEpoch(newLeaderEpoch)
.setStateBatches(combineStateBatches(currentBatches, newData.stateBatches().stream()
- .map(PersisterOffsetsStateBatch::from)
- .collect(Collectors.toCollection(LinkedHashSet::new)), newStartOffset))
- .build();
+ .map(ShareCoordinatorShard::toPersisterStateBatch)
+ .collect(Collectors.toList()), newStartOffset))
+ .build();
}
/**
- * Util method which takes in 2 collections containing {@link PersisterOffsetsStateBatch}
+ * Util method which takes in 2 collections containing {@link PersisterStateBatch}
* and the startOffset.
- * It removes all batches from the 1st collection which have the same first and last offset
- * as the batches in 2nd collection. It then creates a final list of batches which contains the
- * former result and all the batches in the 2nd collection. In set notation (A - B) U B (we prefer batches in B
- * which have same first and last offset in A).
- * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1.
- * @param currentBatch - collection containing current soft state of batches
- * @param newBatch - collection containing batches in incoming request
+ * This method checks any overlap between current state batches and new state batches.
+ * Based on various conditions it creates new non-overlapping records preferring new batches.
+ * Finally, it removes any batches where the lastOffset < startOffset, if the startOffset > -1 and
+ * merges any contiguous intervals with same state.
+ * @param batchesSoFar - collection containing current soft state of batches
+ * @param newBatches - collection containing batches in incoming request
* @param startOffset - startOffset to consider when removing old batches.
* @return List containing combined batches
*/
- private static List<PersisterOffsetsStateBatch> combineStateBatches(
- Collection<PersisterOffsetsStateBatch> currentBatch,
- Collection<PersisterOffsetsStateBatch> newBatch,
+ // visibility for testing
+ static List<PersisterStateBatch> combineStateBatches(
+ List<PersisterStateBatch> batchesSoFar,
+ List<PersisterStateBatch> newBatches,
long startOffset
) {
- currentBatch.removeAll(newBatch);
- List<PersisterOffsetsStateBatch> batchesToAdd = new LinkedList<>(currentBatch);
- batchesToAdd.addAll(newBatch);
- // Any batches where the last offset is < the current start offset
- // are now expired. We should remove them from the persister.
+ List<PersisterStateBatch> combinedList = new ArrayList<>(batchesSoFar.size() + newBatches.size());
+ combinedList.addAll(batchesSoFar);
+ combinedList.addAll(newBatches);
+
+ // sort keeping delivery state in mind
+ combinedList.sort(PersisterStateBatch::compareTo);
+
+ return mergeBatches(
+ pruneBatches(
+ combinedList,
+ startOffset
+ )
+ );
+ }
+
+ private static class BatchOverlapState {
+ private final PersisterStateBatch last;
+ private final PersisterStateBatch candidate;
+ private final List<PersisterStateBatch> nonOverlapping;
+ public static final BatchOverlapState SENTINEL = new BatchOverlapState(null, null, Collections.emptyList());
+
+ public BatchOverlapState(
+ PersisterStateBatch last,
+ PersisterStateBatch candidate,
+ List<PersisterStateBatch> nonOverlapping
+ ) {
+ this.last = last;
+ this.candidate = candidate;
+ this.nonOverlapping = nonOverlapping;
+ }
+
+ public PersisterStateBatch last() {
+ return last;
+ }
+
+ public PersisterStateBatch candidate() {
+ return candidate;
+ }
+
+ public List<PersisterStateBatch> nonOverlapping() {
+ return nonOverlapping;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (!(o instanceof BatchOverlapState)) return false;
+ BatchOverlapState that = (BatchOverlapState) o;
+ return Objects.equals(last, that.last) && Objects.equals(candidate, that.candidate) && Objects.equals(nonOverlapping, that.nonOverlapping);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(last, candidate, nonOverlapping);
+ }
+ }
+
+ static List<PersisterStateBatch> mergeBatches(List<PersisterStateBatch> batches) {
+ if (batches.size() < 2) {
+ return batches;
+ }
+ TreeSet<PersisterStateBatch> sortedBatches = new TreeSet<>(batches);
+ List<PersisterStateBatch> finalBatches = new ArrayList<>(batches.size() * 2); // heuristic size
+
+ BatchOverlapState overlapState = getOverlappingState(sortedBatches);
+
+ while (overlapState != BatchOverlapState.SENTINEL) {
+ PersisterStateBatch last = overlapState.last();
+ PersisterStateBatch candidate = overlapState.candidate();
+
+ // remove non overlapping prefix from sortedBatches,
+ // will make getting next overlapping pair efficient
+ // as a prefix batch which is non overlapping will only
+ // be checked once.
+ if (overlapState.nonOverlapping() != null) {
+ overlapState.nonOverlapping().forEach(sortedBatches::remove);
+ finalBatches.addAll(overlapState.nonOverlapping());
+ }
+
+ if (candidate == null) {
+ break;
+ }
+
+ // overlap and same state (last.firstOffset <= candidate.firstOffset due to sort
+ // covers:
+ // case: 1 2 3 4 5 6 7 (contiguous)
+ // last: ______ _______ _______ _______ _______ ________ _______
+ // candidate: ______ ____ __________ ___ ____ _______ _______
+ if (compareBatchState(candidate, last) == 0) {
+ sortedBatches.remove(last); // remove older smaller interval
+ sortedBatches.remove(candidate);
+
+ last = new PersisterStateBatch(
+ last.firstOffset(),
+ // cover cases
+ // last: ______ ________ _________
+ // candidate: ___ __________ _____
+ Math.max(candidate.lastOffset(), last.lastOffset()),
+ last.deliveryState(),
+ last.deliveryCount()
+ );
+
+ sortedBatches.add(last);
+ } else if (candidate.firstOffset() <= last.lastOffset()) { // non contiguous overlap
+ // overlap and different state
+ // covers:
+ // case: 1 2 3 4 5 6
+ // last: ______ _______ _______ _______ _______ ________
+ // candidate: ______ ____ _________ ____ ____ _______
+ // max batches: 1 2 2 3 2 2
+ // min batches: 1 1 1 1 1 2
+
+ sortedBatches.remove(last);
+ if (candidate.firstOffset() == last.firstOffset()) {
+ if (candidate.lastOffset() == last.lastOffset()) { // case 1
+ if (compareBatchState(candidate, last) < 0) { // candidate is lower priority
+ sortedBatches.add(last);
Review Comment:
actually on second look, there is no state comparison needed. By nature of the compareTo method we defined, candidate can never have lower priority if last.firstOffset == candidate.firstOffset and last.lastOffset == candidate.lastOffset and since it is already present in the treeset, we don't need to do anything for case 1.
Because of above properties, the original was also valid (since treeset will not allow duplicates).
will simplify the condition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1757017297
##########
checkstyle/suppressions.xml:
##########
@@ -340,6 +340,14 @@
<suppress checks="NPathComplexity"
files="CoordinatorRuntime.java"/>
+ <!-- share coordinator -->
+ <suppress checks="MethodLength"
+ files="ShareCoordinatorShardTest.java"/>
+ <suppress checks="NPathComplexity"
+ files="ShareCoordinatorShard.java"/>
+ <suppress checks="CyclomaticComplexity"
+ files="ShareCoordinatorShard.java"/>
Review Comment:
The problem will still remain, since the test cases are curated to create various scenarios. We cannot generate them deterministically. If I add a new private method to return the tests, it will still five MethodLength issue.
Best I can do is to suppress warning for this specific method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1757032820
##########
share-coordinator/src/test/java/org/apache/kafka/coordinator/share/ShareCoordinatorShardTest.java:
##########
@@ -791,6 +793,304 @@ public void testNonSequentialBatchUpdates() {
verify(shard.getMetricsShard(), times(3)).record(ShareCoordinatorMetrics.SHARE_COORDINATOR_WRITE_SENSOR_NAME);
}
+ @Test
+ public void testStateBatchCombine() {
Review Comment:
Parameterization will again result in MethodLength issue as the tests need to be placed in a generator method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1757054315
##########
server-common/src/main/java/org/apache/kafka/server/group/share/PersisterStateBatch.java:
##########
@@ -95,4 +95,18 @@ public String toString() {
"deliveryCount=" + deliveryCount +
")";
}
+
+ @Override
+ public int compareTo(Object o) {
+ PersisterStateBatch that = (PersisterStateBatch) o;
+ int deltaFirst = Long.compare(this.firstOffset(), that.firstOffset());
+ if (deltaFirst == 0) {
+ int deltaLast = Long.compare(this.lastOffset(), that.lastOffset());
+ if (deltaLast == 0) {
+ return this.deliveryCount() - that.deliveryCount();
+ }
+ return deltaLast;
+ }
+ return deltaFirst;
+ }
Review Comment:
its managed in the next revision
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17367: Share coordinator impl. Added additional tests. [3/N] [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17149:
URL: https://github.com/apache/kafka/pull/17149#discussion_r1757066535
##########
checkstyle/suppressions.xml:
##########
@@ -340,6 +340,14 @@
<suppress checks="NPathComplexity"
files="CoordinatorRuntime.java"/>
+ <!-- share coordinator -->
+ <suppress checks="MethodLength"
+ files="ShareCoordinatorShardTest.java"/>
+ <suppress checks="NPathComplexity"
+ files="ShareCoordinatorShard.java"/>
+ <suppress checks="CyclomaticComplexity"
+ files="ShareCoordinatorShard.java"/>
Review Comment:
Suppressing just that method makese sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org