You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "vadimkolodingg (via GitHub)" <gi...@apache.org> on 2025/05/16 14:42:35 UTC
[PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
vadimkolodingg opened a new pull request, #5825:
URL: https://github.com/apache/ignite-3/pull/5825
https://issues.apache.org/jira/browse/IGNITE-24072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "vadimkolodingg (via GitHub)" <gi...@apache.org>.
vadimkolodingg commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2097745713
##########
modules/partition-distribution/src/main/java/org/apache/ignite/internal/partitiondistribution/PendingAssignmentsCalculator.java:
##########
@@ -49,15 +53,79 @@ public PendingAssignmentsCalculator target(Assignments target) {
}
/**
- * Calculates the pending assignments queue between current stable and target assignments.
+ * Calculates the pending assignments queue between current stable (not included) and target assignments (included, last element).
*/
public AssignmentsQueue toQueue() {
assert stable != null;
assert target != null;
Review Comment:
calculator is made like builder, and it is not possible to make those fields final. I don't want to change builder to public constructor or make fields `final AtomicReference`
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/rebalance/RebalanceRaftGroupEventsListener.java:
##########
@@ -64,6 +64,7 @@
import org.apache.ignite.internal.partitiondistribution.Assignments;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "vadimkolodingg (via GitHub)" <gi...@apache.org>.
vadimkolodingg commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2161582294
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/rebalance/RebalanceRaftGroupEventsListener.java:
##########
@@ -308,13 +309,56 @@ private void doStableKeySwitch(
Entry switchAppendEntry = values.get(switchAppendKey);
Entry assignmentsChainEntry = values.get(assignmentsChainKey);
+ AssignmentsQueue pendingAssignmentsQueue = AssignmentsQueue.fromBytes(pendingEntry.value());
+
+ if (pendingAssignmentsQueue != null && pendingAssignmentsQueue.size() > 1) {
+
+ if (pendingAssignmentsQueue.peekFirst().nodes().equals(stableFromRaft)) {
+ pendingAssignmentsQueue.poll(); // remove, first element was already applied to the configuration by pending listeners
+ }
+
+ Assignments stable = Assignments.of(stableFromRaft, pendingAssignmentsQueue.peekFirst().timestamp());
+ pendingAssignmentsQueue = PendingAssignmentsCalculator.pendingAssignmentsCalculator()
+ .stable(stable)
+ .target(pendingAssignmentsQueue.peekLast())
+ .toQueue();
+
+ boolean updated = metaStorageMgr.invoke(iif(
+ revision(pendingPartAssignmentsKey).eq(pendingEntry.revision()),
+ ops(put(pendingPartAssignmentsKey, pendingAssignmentsQueue.toBytes())).yield(true),
+ ops().yield(false)
+ ))
+ .get().getAsBoolean();
Review Comment:
upd: IGNITE-17592 is closed, sync wait replaced with future chain as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2159075229
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteStringFormatter.format;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.colocationEnabled;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.runRace;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willSucceedIn;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.Cluster;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.TestRebalanceUtil;
+import org.apache.ignite.internal.TestWrappers;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.lang.RunnableX;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.placementdriver.ReplicaMeta;
+import org.apache.ignite.internal.placementdriver.message.PlacementDriverMessagesFactory;
+import org.apache.ignite.internal.placementdriver.message.StopLeaseProlongationMessage;
+import org.apache.ignite.internal.raft.Peer;
+import org.apache.ignite.internal.raft.PeersAndLearners;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.raft.RaftNodeId;
+import org.apache.ignite.internal.raft.server.impl.JraftServerImpl;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.apache.ignite.internal.table.distributed.disaster.TestDisasterRecoveryUtils;
+import org.apache.ignite.raft.jraft.Node;
+import org.apache.ignite.raft.jraft.RaftGroupService;
+import org.apache.ignite.raft.jraft.entity.PeerId;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
+ private static final int AUTO_SCALE_TIMEOUT_SECONDS = 2;
+ private static final String ZONE_NAME = "TEST_ZONE";
+ private static final String TABLE_NAME = "TEST_TABLE";
+
+ @Override
+ protected int initialNodes() {
+ return 4;
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by remove some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsReduce(stableAssignments);
+ assertThat("pending queue size = 1", expectedPendingAssignmentsQueue.size(), equalTo(1));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=1", pendingEvents, hasSize(1));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsGreaterThanOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by promote/demote some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=queue size", pendingEvents, hasSize(expectedPendingAssignmentsQueue.size()));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testScaleUp() {
+ createZoneAndTable(2, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(2));
+ assertThat(stable0.learners(), hasSize(0));
+
+ // new peer should be started
+ alterZone(3, 2);
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+
+ // new learner should be started
+ alterZone(4, 2);
+ var stable2 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable2.peers(), hasSize(3));
+ assertThat(stable2.learners(), hasSize(1));
+ }
+
+ @Test
+ void testScaleDown() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // peer goes offline
+ String stopNodeName = stable0.peers().stream()
+ .filter(p -> !p.consistentId().equals(leaseholder))
+ .findFirst().map(Peer::consistentId).orElseThrow();
+ cluster.stopNode(stopNodeName);
+ await().atMost(30, SECONDS).untilAsserted(() -> {
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+ assertThat(stableNames, not(hasItem(stopNodeName)));
+ });
+
+ // learner promoted to peer
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat("learner promoted to peer", stable1.peers(), hasItem(stable0.learners().iterator().next()));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+ }
+
+ @Test
+ void testAutomaticRebalanceWhenPeersMajorityIsLostAndNoAvailableLearners() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // learners and some of majority goes offline
+ Stream<Peer> someQuorumNodes = stable0.peers().stream().filter(p -> !p.consistentId().equals(leaseholder)).limit(1);
+ Stream<Peer> learnerNodes = stable0.learners().stream(); // prevent promote learner into peer
+ List<String> stopNodes = Stream.concat(learnerNodes, someQuorumNodes)
+ .map(Peer::consistentId).collect(toList());
+ assertThat(stopNodes, hasSize(2));
+
+ stopNodesInParallel(cluster, stopNodes);
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(2));
+
+ Assignments a = stablePartitionAssignmentsValue(TABLE_NAME);
+ assertNotNull(a, "stable assignments should not be null");
+ assertFalse(a.fromReset(), "stable should not have fromReset flag");
+ assertFalse(a.force(), "stable should not have force flag");
+
+ var stable1 = PeersAndLearners.fromAssignments(a.nodes());
+ assertThat(stable1.peers(), hasSize(2));
+ assertThat(stable1.learners(), hasSize(0));
+ });
+
+ assertNull(pendingAssignmentsValue(TABLE_NAME), "no ongoing or planned rebalance");
+ }
+
+ @Test
+ void testNodeRestartDuringQueueProcessing() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+ String restartNode = stableAssignments.stream()
+ .map(Assignment::consistentId).filter(name -> !name.equals(leaseholder)).findFirst().orElseThrow();
+
+ putPendingAssignments(raftLeader(TABLE_NAME), TABLE_NAME, expectedPendingAssignmentsQueue);
+ cluster.restartNode(cluster.nodeIndex(restartNode));
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(4));
Review Comment:
upd.: `stop` instead of `stopOrHalt` would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2159006189
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteStringFormatter.format;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.colocationEnabled;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.runRace;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willSucceedIn;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.Cluster;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.TestRebalanceUtil;
+import org.apache.ignite.internal.TestWrappers;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.lang.RunnableX;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.placementdriver.ReplicaMeta;
+import org.apache.ignite.internal.placementdriver.message.PlacementDriverMessagesFactory;
+import org.apache.ignite.internal.placementdriver.message.StopLeaseProlongationMessage;
+import org.apache.ignite.internal.raft.Peer;
+import org.apache.ignite.internal.raft.PeersAndLearners;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.raft.RaftNodeId;
+import org.apache.ignite.internal.raft.server.impl.JraftServerImpl;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.apache.ignite.internal.table.distributed.disaster.TestDisasterRecoveryUtils;
+import org.apache.ignite.raft.jraft.Node;
+import org.apache.ignite.raft.jraft.RaftGroupService;
+import org.apache.ignite.raft.jraft.entity.PeerId;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
+ private static final int AUTO_SCALE_TIMEOUT_SECONDS = 2;
+ private static final String ZONE_NAME = "TEST_ZONE";
+ private static final String TABLE_NAME = "TEST_TABLE";
+
+ @Override
+ protected int initialNodes() {
+ return 4;
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by remove some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsReduce(stableAssignments);
+ assertThat("pending queue size = 1", expectedPendingAssignmentsQueue.size(), equalTo(1));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=1", pendingEvents, hasSize(1));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsGreaterThanOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by promote/demote some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=queue size", pendingEvents, hasSize(expectedPendingAssignmentsQueue.size()));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testScaleUp() {
+ createZoneAndTable(2, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(2));
+ assertThat(stable0.learners(), hasSize(0));
+
+ // new peer should be started
+ alterZone(3, 2);
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+
+ // new learner should be started
+ alterZone(4, 2);
+ var stable2 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable2.peers(), hasSize(3));
+ assertThat(stable2.learners(), hasSize(1));
+ }
+
+ @Test
+ void testScaleDown() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // peer goes offline
+ String stopNodeName = stable0.peers().stream()
+ .filter(p -> !p.consistentId().equals(leaseholder))
+ .findFirst().map(Peer::consistentId).orElseThrow();
+ cluster.stopNode(stopNodeName);
+ await().atMost(30, SECONDS).untilAsserted(() -> {
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+ assertThat(stableNames, not(hasItem(stopNodeName)));
+ });
+
+ // learner promoted to peer
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat("learner promoted to peer", stable1.peers(), hasItem(stable0.learners().iterator().next()));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+ }
+
+ @Test
+ void testAutomaticRebalanceWhenPeersMajorityIsLostAndNoAvailableLearners() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // learners and some of majority goes offline
+ Stream<Peer> someQuorumNodes = stable0.peers().stream().filter(p -> !p.consistentId().equals(leaseholder)).limit(1);
+ Stream<Peer> learnerNodes = stable0.learners().stream(); // prevent promote learner into peer
+ List<String> stopNodes = Stream.concat(learnerNodes, someQuorumNodes)
+ .map(Peer::consistentId).collect(toList());
+ assertThat(stopNodes, hasSize(2));
+
+ stopNodesInParallel(cluster, stopNodes);
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(2));
+
+ Assignments a = stablePartitionAssignmentsValue(TABLE_NAME);
+ assertNotNull(a, "stable assignments should not be null");
+ assertFalse(a.fromReset(), "stable should not have fromReset flag");
+ assertFalse(a.force(), "stable should not have force flag");
+
+ var stable1 = PeersAndLearners.fromAssignments(a.nodes());
+ assertThat(stable1.peers(), hasSize(2));
+ assertThat(stable1.learners(), hasSize(0));
+ });
+
+ assertNull(pendingAssignmentsValue(TABLE_NAME), "no ongoing or planned rebalance");
+ }
+
+ @Test
+ void testNodeRestartDuringQueueProcessing() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+ String restartNode = stableAssignments.stream()
+ .map(Assignment::consistentId).filter(name -> !name.equals(leaseholder)).findFirst().orElseThrow();
+
+ putPendingAssignments(raftLeader(TABLE_NAME), TABLE_NAME, expectedPendingAssignmentsQueue);
+ cluster.restartNode(cluster.nodeIndex(restartNode));
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(4));
Review Comment:
The default failure handler is `noop` so if something fails during the recovery - we may not see it.
Please add the failure handler configuration that would stop the node in case of problem:
`failureHandler.type: stop` (see `StopNodeOrHaltFailureHandlerConfigurationSchema`)
See also `@ConfigOverrides` for test config change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2159049527
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteStringFormatter.format;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.colocationEnabled;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.runRace;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willSucceedIn;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.Cluster;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.TestRebalanceUtil;
+import org.apache.ignite.internal.TestWrappers;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.lang.RunnableX;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.placementdriver.ReplicaMeta;
+import org.apache.ignite.internal.placementdriver.message.PlacementDriverMessagesFactory;
+import org.apache.ignite.internal.placementdriver.message.StopLeaseProlongationMessage;
+import org.apache.ignite.internal.raft.Peer;
+import org.apache.ignite.internal.raft.PeersAndLearners;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.raft.RaftNodeId;
+import org.apache.ignite.internal.raft.server.impl.JraftServerImpl;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.apache.ignite.internal.table.distributed.disaster.TestDisasterRecoveryUtils;
+import org.apache.ignite.raft.jraft.Node;
+import org.apache.ignite.raft.jraft.RaftGroupService;
+import org.apache.ignite.raft.jraft.entity.PeerId;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
+ private static final int AUTO_SCALE_TIMEOUT_SECONDS = 2;
+ private static final String ZONE_NAME = "TEST_ZONE";
+ private static final String TABLE_NAME = "TEST_TABLE";
+
+ @Override
+ protected int initialNodes() {
+ return 4;
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by remove some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsReduce(stableAssignments);
+ assertThat("pending queue size = 1", expectedPendingAssignmentsQueue.size(), equalTo(1));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=1", pendingEvents, hasSize(1));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsGreaterThanOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by promote/demote some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=queue size", pendingEvents, hasSize(expectedPendingAssignmentsQueue.size()));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testScaleUp() {
+ createZoneAndTable(2, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(2));
+ assertThat(stable0.learners(), hasSize(0));
+
+ // new peer should be started
+ alterZone(3, 2);
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+
+ // new learner should be started
+ alterZone(4, 2);
+ var stable2 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable2.peers(), hasSize(3));
+ assertThat(stable2.learners(), hasSize(1));
+ }
+
+ @Test
+ void testScaleDown() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // peer goes offline
+ String stopNodeName = stable0.peers().stream()
+ .filter(p -> !p.consistentId().equals(leaseholder))
+ .findFirst().map(Peer::consistentId).orElseThrow();
+ cluster.stopNode(stopNodeName);
+ await().atMost(30, SECONDS).untilAsserted(() -> {
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+ assertThat(stableNames, not(hasItem(stopNodeName)));
+ });
+
+ // learner promoted to peer
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat("learner promoted to peer", stable1.peers(), hasItem(stable0.learners().iterator().next()));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+ }
+
+ @Test
+ void testAutomaticRebalanceWhenPeersMajorityIsLostAndNoAvailableLearners() {
Review Comment:
The test name is misleading: you have quorum=2 and stop 2 nodes of 4, and one of them is learner, this is not the majority loss.
Whan I meant was disaster recovery scenario, with actual loss of majority of consensus nodes and then calling `DisasterRecoveryManager#resetPartitions` in order to recover the partitions. Basically I expect that it would work out-of-box but I suggest to check it. Please add this scenario.
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteStringFormatter.format;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.enabledColocation;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.runRace;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willSucceedIn;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.Cluster;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.TestRebalanceUtil;
+import org.apache.ignite.internal.TestWrappers;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.lang.RunnableX;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.placementdriver.ReplicaMeta;
+import org.apache.ignite.internal.placementdriver.message.PlacementDriverMessagesFactory;
+import org.apache.ignite.internal.placementdriver.message.StopLeaseProlongationMessage;
+import org.apache.ignite.internal.raft.Peer;
+import org.apache.ignite.internal.raft.PeersAndLearners;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.raft.RaftNodeId;
+import org.apache.ignite.internal.raft.server.impl.JraftServerImpl;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.apache.ignite.internal.table.distributed.disaster.TestDisasterRecoveryUtils;
+import org.apache.ignite.raft.jraft.Node;
+import org.apache.ignite.raft.jraft.RaftGroupService;
+import org.apache.ignite.raft.jraft.entity.PeerId;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
+ private static final int AUTO_SCALE_TIMEOUT_SECONDS = 2;
+ private static final String ZONE_NAME = "TEST_ZONE";
+ private static final String TABLE_NAME = "TEST_TABLE";
+
+ @Override
+ protected int initialNodes() {
+ return 4;
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by remove some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsReduce(stableAssignments);
+ assertThat("pending queue size = 1", expectedPendingAssignmentsQueue.size(), equalTo(1));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=1", pendingEvents, hasSize(1));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsGreaterThanOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by promote/demote some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=queue size", pendingEvents, hasSize(expectedPendingAssignmentsQueue.size()));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testScaleUp() {
+ createZoneAndTable(2, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(2));
+ assertThat(stable0.learners(), hasSize(0));
+
+ // new peer should be started
+ alterZone(3, 2);
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+
+ // new learner should be started
+ alterZone(4, 2);
+ var stable2 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable2.peers(), hasSize(3));
+ assertThat(stable2.learners(), hasSize(1));
+ }
+
+ @Test
+ void testScaleDown() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // peer goes offline
+ String stopNodeName = stable0.peers().stream()
+ .filter(p -> !p.consistentId().equals(leaseholder))
+ .findFirst().map(Peer::consistentId).orElseThrow();
+ cluster.stopNode(stopNodeName);
+ await().atMost(30, SECONDS).untilAsserted(() -> {
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+ assertThat(stableNames, not(hasItem(stopNodeName)));
+ });
+
+ // learner promoted to peer
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat("learner promoted to peer", stable1.peers(), hasItem(stable0.learners().iterator().next()));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+ }
+
+ @Test
+ void testAutomaticRebalanceWhenPeersMajorityIsLostAndNoAvailableLearners() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // learners and some of majority goes offline
+ Stream<Peer> someQuorumNodes = stable0.peers().stream().filter(p -> !p.consistentId().equals(leaseholder)).limit(1);
+ Stream<Peer> learnerNodes = stable0.learners().stream(); // prevent promote learner into peer
+ List<String> stopNodes = Stream.concat(learnerNodes, someQuorumNodes)
+ .map(Peer::consistentId).collect(toList());
+ assertThat(stopNodes, hasSize(2));
+
+ stopNodesInParallel(cluster, stopNodes);
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(2));
+
+ Assignments a = stablePartitionAssignmentsValue(TABLE_NAME);
+ assertNotNull(a, "stable assignments should not be null");
+ assertFalse(a.fromReset(), "stable has no fromReset flag");
+ assertFalse(a.force(), "stable has no force flag");
+
+ var stable1 = PeersAndLearners.fromAssignments(a.nodes());
+ assertThat(stable1.peers(), hasSize(2));
+ assertThat(stable1.learners(), hasSize(0));
+ });
+
+ assertNull(pendingAssignmentsValue(TABLE_NAME), "no ongoing or planned rebalance");
+ }
+
+ @Test
+ void testNodeRestartDuringQueueProcessing() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+ String restartNode = stableAssignments.stream()
+ .map(Assignment::consistentId).filter(name -> !name.equals(leaseholder)).findFirst().orElseThrow();
+
+ putPendingAssignments(raftLeader(TABLE_NAME), TABLE_NAME, expectedPendingAssignmentsQueue);
+ cluster.restartNode(cluster.nodeIndex(restartNode));
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(4));
+
+ var expected = expectedPendingAssignmentsQueue.peekLast().nodes().stream()
+ .map(Assignment::consistentId).collect(toSet());
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+
+ assertThat(stableNames, equalTo(expected));
+ assertThat(stableNames, hasItem(restartNode));
+ });
+ }
+
+ @Test
+ void testRaftLeaderChangedDuringAssignmentsQueueProcessing() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ var leaderBefore = raftLeader(TABLE_NAME);
+
+ putPendingAssignments(raftLeader(TABLE_NAME), TABLE_NAME, expectedPendingAssignmentsQueue);
+ cancelLease(leaderBefore, TABLE_NAME);
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ var leaderAfter = raftLeader(TABLE_NAME);
Review Comment:
Here you don't check that `leaderAfter ` is not the same as `leaderBefore` - and most likely it is.
`cancelLease` only stops the lease prolongation for the current primary replica, but has no effect on raft group itself. The most reliable way to change the leader in raft group is to stop and restart the current one.
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteStringFormatter.format;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.colocationEnabled;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.runRace;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willSucceedIn;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.Cluster;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.TestRebalanceUtil;
+import org.apache.ignite.internal.TestWrappers;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.lang.RunnableX;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.placementdriver.ReplicaMeta;
+import org.apache.ignite.internal.placementdriver.message.PlacementDriverMessagesFactory;
+import org.apache.ignite.internal.placementdriver.message.StopLeaseProlongationMessage;
+import org.apache.ignite.internal.raft.Peer;
+import org.apache.ignite.internal.raft.PeersAndLearners;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.raft.RaftNodeId;
+import org.apache.ignite.internal.raft.server.impl.JraftServerImpl;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.apache.ignite.internal.table.distributed.disaster.TestDisasterRecoveryUtils;
+import org.apache.ignite.raft.jraft.Node;
+import org.apache.ignite.raft.jraft.RaftGroupService;
+import org.apache.ignite.raft.jraft.entity.PeerId;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
+ private static final int AUTO_SCALE_TIMEOUT_SECONDS = 2;
+ private static final String ZONE_NAME = "TEST_ZONE";
+ private static final String TABLE_NAME = "TEST_TABLE";
+
+ @Override
+ protected int initialNodes() {
+ return 4;
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by remove some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsReduce(stableAssignments);
+ assertThat("pending queue size = 1", expectedPendingAssignmentsQueue.size(), equalTo(1));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=1", pendingEvents, hasSize(1));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsGreaterThanOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by promote/demote some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=queue size", pendingEvents, hasSize(expectedPendingAssignmentsQueue.size()));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testScaleUp() {
+ createZoneAndTable(2, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(2));
+ assertThat(stable0.learners(), hasSize(0));
+
+ // new peer should be started
+ alterZone(3, 2);
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+
+ // new learner should be started
+ alterZone(4, 2);
+ var stable2 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable2.peers(), hasSize(3));
+ assertThat(stable2.learners(), hasSize(1));
+ }
+
+ @Test
+ void testScaleDown() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // peer goes offline
+ String stopNodeName = stable0.peers().stream()
+ .filter(p -> !p.consistentId().equals(leaseholder))
+ .findFirst().map(Peer::consistentId).orElseThrow();
+ cluster.stopNode(stopNodeName);
+ await().atMost(30, SECONDS).untilAsserted(() -> {
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+ assertThat(stableNames, not(hasItem(stopNodeName)));
+ });
+
+ // learner promoted to peer
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat("learner promoted to peer", stable1.peers(), hasItem(stable0.learners().iterator().next()));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+ }
+
+ @Test
+ void testAutomaticRebalanceWhenPeersMajorityIsLostAndNoAvailableLearners() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // learners and some of majority goes offline
+ Stream<Peer> someQuorumNodes = stable0.peers().stream().filter(p -> !p.consistentId().equals(leaseholder)).limit(1);
+ Stream<Peer> learnerNodes = stable0.learners().stream(); // prevent promote learner into peer
+ List<String> stopNodes = Stream.concat(learnerNodes, someQuorumNodes)
+ .map(Peer::consistentId).collect(toList());
+ assertThat(stopNodes, hasSize(2));
+
+ stopNodesInParallel(cluster, stopNodes);
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(2));
+
+ Assignments a = stablePartitionAssignmentsValue(TABLE_NAME);
+ assertNotNull(a, "stable assignments should not be null");
+ assertFalse(a.fromReset(), "stable should not have fromReset flag");
+ assertFalse(a.force(), "stable should not have force flag");
+
+ var stable1 = PeersAndLearners.fromAssignments(a.nodes());
+ assertThat(stable1.peers(), hasSize(2));
+ assertThat(stable1.learners(), hasSize(0));
+ });
+
+ assertNull(pendingAssignmentsValue(TABLE_NAME), "no ongoing or planned rebalance");
+ }
+
+ @Test
+ void testNodeRestartDuringQueueProcessing() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+ String restartNode = stableAssignments.stream()
+ .map(Assignment::consistentId).filter(name -> !name.equals(leaseholder)).findFirst().orElseThrow();
+
+ putPendingAssignments(raftLeader(TABLE_NAME), TABLE_NAME, expectedPendingAssignmentsQueue);
+ cluster.restartNode(cluster.nodeIndex(restartNode));
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(4));
Review Comment:
The default failure handler is `noop` so if something fails during the recovery - we may not see it.
Please add the failure handler configuration that would stop the node in case of problem:
`failureHandler.type: stopOrHalt` (see `StopNodeOrHaltFailureHandlerConfigurationSchema`)
See also `@ConfigOverrides` for test config change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2159006189
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteStringFormatter.format;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.colocationEnabled;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.runRace;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willSucceedIn;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.Cluster;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.TestRebalanceUtil;
+import org.apache.ignite.internal.TestWrappers;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.lang.RunnableX;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.placementdriver.ReplicaMeta;
+import org.apache.ignite.internal.placementdriver.message.PlacementDriverMessagesFactory;
+import org.apache.ignite.internal.placementdriver.message.StopLeaseProlongationMessage;
+import org.apache.ignite.internal.raft.Peer;
+import org.apache.ignite.internal.raft.PeersAndLearners;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.raft.RaftNodeId;
+import org.apache.ignite.internal.raft.server.impl.JraftServerImpl;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.apache.ignite.internal.table.distributed.disaster.TestDisasterRecoveryUtils;
+import org.apache.ignite.raft.jraft.Node;
+import org.apache.ignite.raft.jraft.RaftGroupService;
+import org.apache.ignite.raft.jraft.entity.PeerId;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
+ private static final int AUTO_SCALE_TIMEOUT_SECONDS = 2;
+ private static final String ZONE_NAME = "TEST_ZONE";
+ private static final String TABLE_NAME = "TEST_TABLE";
+
+ @Override
+ protected int initialNodes() {
+ return 4;
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by remove some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsReduce(stableAssignments);
+ assertThat("pending queue size = 1", expectedPendingAssignmentsQueue.size(), equalTo(1));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=1", pendingEvents, hasSize(1));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsGreaterThanOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by promote/demote some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=queue size", pendingEvents, hasSize(expectedPendingAssignmentsQueue.size()));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testScaleUp() {
+ createZoneAndTable(2, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(2));
+ assertThat(stable0.learners(), hasSize(0));
+
+ // new peer should be started
+ alterZone(3, 2);
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+
+ // new learner should be started
+ alterZone(4, 2);
+ var stable2 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable2.peers(), hasSize(3));
+ assertThat(stable2.learners(), hasSize(1));
+ }
+
+ @Test
+ void testScaleDown() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // peer goes offline
+ String stopNodeName = stable0.peers().stream()
+ .filter(p -> !p.consistentId().equals(leaseholder))
+ .findFirst().map(Peer::consistentId).orElseThrow();
+ cluster.stopNode(stopNodeName);
+ await().atMost(30, SECONDS).untilAsserted(() -> {
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+ assertThat(stableNames, not(hasItem(stopNodeName)));
+ });
+
+ // learner promoted to peer
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat("learner promoted to peer", stable1.peers(), hasItem(stable0.learners().iterator().next()));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+ }
+
+ @Test
+ void testAutomaticRebalanceWhenPeersMajorityIsLostAndNoAvailableLearners() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // learners and some of majority goes offline
+ Stream<Peer> someQuorumNodes = stable0.peers().stream().filter(p -> !p.consistentId().equals(leaseholder)).limit(1);
+ Stream<Peer> learnerNodes = stable0.learners().stream(); // prevent promote learner into peer
+ List<String> stopNodes = Stream.concat(learnerNodes, someQuorumNodes)
+ .map(Peer::consistentId).collect(toList());
+ assertThat(stopNodes, hasSize(2));
+
+ stopNodesInParallel(cluster, stopNodes);
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(2));
+
+ Assignments a = stablePartitionAssignmentsValue(TABLE_NAME);
+ assertNotNull(a, "stable assignments should not be null");
+ assertFalse(a.fromReset(), "stable should not have fromReset flag");
+ assertFalse(a.force(), "stable should not have force flag");
+
+ var stable1 = PeersAndLearners.fromAssignments(a.nodes());
+ assertThat(stable1.peers(), hasSize(2));
+ assertThat(stable1.learners(), hasSize(0));
+ });
+
+ assertNull(pendingAssignmentsValue(TABLE_NAME), "no ongoing or planned rebalance");
+ }
+
+ @Test
+ void testNodeRestartDuringQueueProcessing() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+ String restartNode = stableAssignments.stream()
+ .map(Assignment::consistentId).filter(name -> !name.equals(leaseholder)).findFirst().orElseThrow();
+
+ putPendingAssignments(raftLeader(TABLE_NAME), TABLE_NAME, expectedPendingAssignmentsQueue);
+ cluster.restartNode(cluster.nodeIndex(restartNode));
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(4));
Review Comment:
The default failure handler is `noop` so if something fails during the recovery - we may not see it.
Please add the failure handler configuration that would stop the node in case of problem:
`failureHandler.type: stop` (see `StopNodeFailureHandlerConfigurationSchema`)
See also `@ConfigOverrides` for test config change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2097643294
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/rebalance/RebalanceRaftGroupEventsListener.java:
##########
@@ -308,13 +309,56 @@ private void doStableKeySwitch(
Entry switchAppendEntry = values.get(switchAppendKey);
Entry assignmentsChainEntry = values.get(assignmentsChainKey);
+ AssignmentsQueue pendingAssignmentsQueue = AssignmentsQueue.fromBytes(pendingEntry.value());
Review Comment:
This method is already fat enough... Let's maybe move this code to separate method, WDYT? Same for zone listener
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "vadimkolodingg (via GitHub)" <gi...@apache.org>.
vadimkolodingg commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2097733787
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/rebalance/RebalanceRaftGroupEventsListener.java:
##########
@@ -308,13 +309,56 @@ private void doStableKeySwitch(
Entry switchAppendEntry = values.get(switchAppendKey);
Entry assignmentsChainEntry = values.get(assignmentsChainKey);
+ AssignmentsQueue pendingAssignmentsQueue = AssignmentsQueue.fromBytes(pendingEntry.value());
+
+ if (pendingAssignmentsQueue != null && pendingAssignmentsQueue.size() > 1) {
+
+ if (pendingAssignmentsQueue.peekFirst().nodes().equals(stableFromRaft)) {
+ pendingAssignmentsQueue.poll(); // remove, first element was already applied to the configuration by pending listeners
+ }
+
+ Assignments stable = Assignments.of(stableFromRaft, pendingAssignmentsQueue.peekFirst().timestamp());
+ pendingAssignmentsQueue = PendingAssignmentsCalculator.pendingAssignmentsCalculator()
+ .stable(stable)
+ .target(pendingAssignmentsQueue.peekLast())
+ .toQueue();
+
+ boolean updated = metaStorageMgr.invoke(iif(
+ revision(pendingPartAssignmentsKey).eq(pendingEntry.revision()),
+ ops(put(pendingPartAssignmentsKey, pendingAssignmentsQueue.toBytes())).yield(true),
+ ops().yield(false)
+ ))
+ .get().getAsBoolean();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "vadimkolodingg (via GitHub)" <gi...@apache.org>.
vadimkolodingg commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2097763459
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/rebalance/RebalanceRaftGroupEventsListener.java:
##########
@@ -308,13 +309,56 @@ private void doStableKeySwitch(
Entry switchAppendEntry = values.get(switchAppendKey);
Entry assignmentsChainEntry = values.get(assignmentsChainKey);
+ AssignmentsQueue pendingAssignmentsQueue = AssignmentsQueue.fromBytes(pendingEntry.value());
Review Comment:
unfortuantely, added lines use quick return and recursion, which limits ways to refactor.
For me, it's more clear leaving those lines as is than adding method with a bunch of arguments (which is just single `metaStorageMgr.invoke`)
```java
boolean updated = pendingAssignmentsPollAndUpdate(metaStorageMgr, pendingKey, revision, stable, pendingQueue)
if (updated) {
return;
} else {
doStableKeySwitch(...)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "vadimkolodingg (via GitHub)" <gi...@apache.org>.
vadimkolodingg commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2161583085
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteStringFormatter.format;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.enabledColocation;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.runRace;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willSucceedIn;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.Cluster;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.TestRebalanceUtil;
+import org.apache.ignite.internal.TestWrappers;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.lang.RunnableX;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.placementdriver.ReplicaMeta;
+import org.apache.ignite.internal.placementdriver.message.PlacementDriverMessagesFactory;
+import org.apache.ignite.internal.placementdriver.message.StopLeaseProlongationMessage;
+import org.apache.ignite.internal.raft.Peer;
+import org.apache.ignite.internal.raft.PeersAndLearners;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.raft.RaftNodeId;
+import org.apache.ignite.internal.raft.server.impl.JraftServerImpl;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.apache.ignite.internal.table.distributed.disaster.TestDisasterRecoveryUtils;
+import org.apache.ignite.raft.jraft.Node;
+import org.apache.ignite.raft.jraft.RaftGroupService;
+import org.apache.ignite.raft.jraft.entity.PeerId;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
+ private static final int AUTO_SCALE_TIMEOUT_SECONDS = 2;
+ private static final String ZONE_NAME = "TEST_ZONE";
+ private static final String TABLE_NAME = "TEST_TABLE";
+
+ @Override
+ protected int initialNodes() {
+ return 4;
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by remove some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsReduce(stableAssignments);
+ assertThat("pending queue size = 1", expectedPendingAssignmentsQueue.size(), equalTo(1));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=1", pendingEvents, hasSize(1));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsGreaterThanOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by promote/demote some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=queue size", pendingEvents, hasSize(expectedPendingAssignmentsQueue.size()));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testScaleUp() {
+ createZoneAndTable(2, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(2));
+ assertThat(stable0.learners(), hasSize(0));
+
+ // new peer should be started
+ alterZone(3, 2);
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+
+ // new learner should be started
+ alterZone(4, 2);
+ var stable2 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable2.peers(), hasSize(3));
+ assertThat(stable2.learners(), hasSize(1));
+ }
+
+ @Test
+ void testScaleDown() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // peer goes offline
+ String stopNodeName = stable0.peers().stream()
+ .filter(p -> !p.consistentId().equals(leaseholder))
+ .findFirst().map(Peer::consistentId).orElseThrow();
+ cluster.stopNode(stopNodeName);
+ await().atMost(30, SECONDS).untilAsserted(() -> {
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+ assertThat(stableNames, not(hasItem(stopNodeName)));
+ });
+
+ // learner promoted to peer
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat("learner promoted to peer", stable1.peers(), hasItem(stable0.learners().iterator().next()));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+ }
+
+ @Test
+ void testAutomaticRebalanceWhenPeersMajorityIsLostAndNoAvailableLearners() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // learners and some of majority goes offline
+ Stream<Peer> someQuorumNodes = stable0.peers().stream().filter(p -> !p.consistentId().equals(leaseholder)).limit(1);
+ Stream<Peer> learnerNodes = stable0.learners().stream(); // prevent promote learner into peer
+ List<String> stopNodes = Stream.concat(learnerNodes, someQuorumNodes)
+ .map(Peer::consistentId).collect(toList());
+ assertThat(stopNodes, hasSize(2));
+
+ stopNodesInParallel(cluster, stopNodes);
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(2));
+
+ Assignments a = stablePartitionAssignmentsValue(TABLE_NAME);
+ assertNotNull(a, "stable assignments should not be null");
+ assertFalse(a.fromReset(), "stable has no fromReset flag");
+ assertFalse(a.force(), "stable has no force flag");
+
+ var stable1 = PeersAndLearners.fromAssignments(a.nodes());
+ assertThat(stable1.peers(), hasSize(2));
+ assertThat(stable1.learners(), hasSize(0));
+ });
+
+ assertNull(pendingAssignmentsValue(TABLE_NAME), "no ongoing or planned rebalance");
+ }
+
+ @Test
+ void testNodeRestartDuringQueueProcessing() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+ String restartNode = stableAssignments.stream()
+ .map(Assignment::consistentId).filter(name -> !name.equals(leaseholder)).findFirst().orElseThrow();
+
+ putPendingAssignments(raftLeader(TABLE_NAME), TABLE_NAME, expectedPendingAssignmentsQueue);
+ cluster.restartNode(cluster.nodeIndex(restartNode));
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ assertThat(dataNodes(TABLE_NAME), hasSize(4));
+
+ var expected = expectedPendingAssignmentsQueue.peekLast().nodes().stream()
+ .map(Assignment::consistentId).collect(toSet());
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+
+ assertThat(stableNames, equalTo(expected));
+ assertThat(stableNames, hasItem(restartNode));
+ });
+ }
+
+ @Test
+ void testRaftLeaderChangedDuringAssignmentsQueueProcessing() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ var leaderBefore = raftLeader(TABLE_NAME);
+
+ putPendingAssignments(raftLeader(TABLE_NAME), TABLE_NAME, expectedPendingAssignmentsQueue);
+ cancelLease(leaderBefore, TABLE_NAME);
+
+ await().atMost(120, SECONDS).untilAsserted(() -> {
+ var leaderAfter = raftLeader(TABLE_NAME);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "vadimkolodingg (via GitHub)" <gi...@apache.org>.
vadimkolodingg commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2161598051
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteStringFormatter.format;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.colocationEnabled;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.runRace;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willSucceedIn;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.Cluster;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.TestRebalanceUtil;
+import org.apache.ignite.internal.TestWrappers;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.lang.RunnableX;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.placementdriver.ReplicaMeta;
+import org.apache.ignite.internal.placementdriver.message.PlacementDriverMessagesFactory;
+import org.apache.ignite.internal.placementdriver.message.StopLeaseProlongationMessage;
+import org.apache.ignite.internal.raft.Peer;
+import org.apache.ignite.internal.raft.PeersAndLearners;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.raft.RaftNodeId;
+import org.apache.ignite.internal.raft.server.impl.JraftServerImpl;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.apache.ignite.internal.table.distributed.disaster.TestDisasterRecoveryUtils;
+import org.apache.ignite.raft.jraft.Node;
+import org.apache.ignite.raft.jraft.RaftGroupService;
+import org.apache.ignite.raft.jraft.entity.PeerId;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
+ private static final int AUTO_SCALE_TIMEOUT_SECONDS = 2;
+ private static final String ZONE_NAME = "TEST_ZONE";
+ private static final String TABLE_NAME = "TEST_TABLE";
+
+ @Override
+ protected int initialNodes() {
+ return 4;
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by remove some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsReduce(stableAssignments);
+ assertThat("pending queue size = 1", expectedPendingAssignmentsQueue.size(), equalTo(1));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=1", pendingEvents, hasSize(1));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testDoStableKeySwitchWhenPendingQueueIsGreaterThanOne() {
+ createZoneAndTable(4, 2);
+
+ Set<Assignment> stableAssignments = stablePartitionAssignments(TABLE_NAME);
+
+ // update pending assignments by promote/demote some in stable
+ AssignmentsQueue expectedPendingAssignmentsQueue = assignmentsPromoteDemote(stableAssignments);
+ assertThat("pending queue size >= 2", expectedPendingAssignmentsQueue.size(), greaterThanOrEqualTo(2));
+
+ recordAssignmentsEvents(cluster.aliveNode(), recorder -> {
+
+ putPendingAssignments(cluster.aliveNode(), TABLE_NAME, expectedPendingAssignmentsQueue);
+
+ await().untilAsserted(() -> {
+ Deque<AssignmentsQueue> pendingEvents = recorder.pendingAssignmentsEvents(TABLE_NAME);
+ Deque<Assignments> stableEvents = recorder.stableAssignmentsEvents(TABLE_NAME);
+
+ assertThat("pending assignment events N=queue size", pendingEvents, hasSize(expectedPendingAssignmentsQueue.size()));
+ assertThat(pendingEvents.peekFirst(), equalTo(expectedPendingAssignmentsQueue));
+
+ assertThat("only one stable switch", stableEvents, hasSize(1));
+ assertThat(stableEvents.peekLast(), equalTo(pendingEvents.peekLast().peekLast()));
+ });
+ });
+ }
+
+ @Test
+ void testScaleUp() {
+ createZoneAndTable(2, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(2));
+ assertThat(stable0.learners(), hasSize(0));
+
+ // new peer should be started
+ alterZone(3, 2);
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+
+ // new learner should be started
+ alterZone(4, 2);
+ var stable2 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable2.peers(), hasSize(3));
+ assertThat(stable2.learners(), hasSize(1));
+ }
+
+ @Test
+ void testScaleDown() {
+ createZoneAndTable(4, 2);
+
+ var stable0 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat(stable0.peers(), hasSize(3));
+ assertThat(stable0.learners(), hasSize(1));
+
+ String leaseholder = primaryReplicaMeta(TABLE_NAME).getLeaseholder();
+
+ // peer goes offline
+ String stopNodeName = stable0.peers().stream()
+ .filter(p -> !p.consistentId().equals(leaseholder))
+ .findFirst().map(Peer::consistentId).orElseThrow();
+ cluster.stopNode(stopNodeName);
+ await().atMost(30, SECONDS).untilAsserted(() -> {
+ var stableNames = stablePartitionAssignments(TABLE_NAME).stream()
+ .map(Assignment::consistentId).collect(toSet());
+ assertThat(stableNames, not(hasItem(stopNodeName)));
+ });
+
+ // learner promoted to peer
+ var stable1 = PeersAndLearners.fromAssignments(stablePartitionAssignments(TABLE_NAME));
+ assertThat("learner promoted to peer", stable1.peers(), hasItem(stable0.learners().iterator().next()));
+ assertThat(stable1.peers(), hasSize(3));
+ assertThat(stable1.learners(), hasSize(0));
+ }
+
+ @Test
+ void testAutomaticRebalanceWhenPeersMajorityIsLostAndNoAvailableLearners() {
Review Comment:
test has been rewritten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2096310737
##########
modules/partition-distribution/src/main/java/org/apache/ignite/internal/partitiondistribution/PendingAssignmentsCalculator.java:
##########
@@ -49,15 +53,79 @@ public PendingAssignmentsCalculator target(Assignments target) {
}
/**
- * Calculates the pending assignments queue between current stable and target assignments.
+ * Calculates the pending assignments queue between current stable (not included) and target assignments (included, last element).
*/
public AssignmentsQueue toQueue() {
assert stable != null;
assert target != null;
Review Comment:
Let's make the both fields final.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/rebalance/RebalanceRaftGroupEventsListener.java:
##########
@@ -308,13 +309,56 @@ private void doStableKeySwitch(
Entry switchAppendEntry = values.get(switchAppendKey);
Entry assignmentsChainEntry = values.get(assignmentsChainKey);
+ AssignmentsQueue pendingAssignmentsQueue = AssignmentsQueue.fromBytes(pendingEntry.value());
+
+ if (pendingAssignmentsQueue != null && pendingAssignmentsQueue.size() > 1) {
+
+ if (pendingAssignmentsQueue.peekFirst().nodes().equals(stableFromRaft)) {
+ pendingAssignmentsQueue.poll(); // remove, first element was already applied to the configuration by pending listeners
+ }
+
+ Assignments stable = Assignments.of(stableFromRaft, pendingAssignmentsQueue.peekFirst().timestamp());
+ pendingAssignmentsQueue = PendingAssignmentsCalculator.pendingAssignmentsCalculator()
+ .stable(stable)
+ .target(pendingAssignmentsQueue.peekLast())
+ .toQueue();
+
+ boolean updated = metaStorageMgr.invoke(iif(
+ revision(pendingPartAssignmentsKey).eq(pendingEntry.revision()),
+ ops(put(pendingPartAssignmentsKey, pendingAssignmentsQueue.toBytes())).yield(true),
+ ops().yield(false)
+ ))
+ .get().getAsBoolean();
Review Comment:
Please add the TODO here:
`// TODO: https://issues.apache.org/jira/browse/IGNITE-17592 Remove synchronous wait`
Same for zones' listener
##########
modules/partition-distribution/src/main/java/org/apache/ignite/internal/partitiondistribution/PendingAssignmentsCalculator.java:
##########
@@ -49,15 +53,79 @@ public PendingAssignmentsCalculator target(Assignments target) {
}
/**
- * Calculates the pending assignments queue between current stable and target assignments.
+ * Calculates the pending assignments queue between current stable (not included) and target assignments (included, last element).
*/
public AssignmentsQueue toQueue() {
assert stable != null;
assert target != null;
- // naive version with only one configuration switch, calculation will be changed during IGNITE-23790 epic
- AssignmentsQueue queue = new AssignmentsQueue(target);
- assert Objects.equals(queue.peekLast(), target) : "Target assignments should be equal to the last element in the queue";
+ if (target.force()) {
+ return new AssignmentsQueue(target);
+ }
Review Comment:
Why so? Are you sure that such transition is always possible?
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/rebalance/RebalanceRaftGroupEventsListener.java:
##########
@@ -64,6 +64,7 @@
import org.apache.ignite.internal.partitiondistribution.Assignments;
Review Comment:
The logging of pending assignments is broken now. See `Received update on pending assignments` in TableManager: there is only first element of the queue logged.
Same with logging of stable assignments change event: see `Received update on stable assignments`
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignments;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.enabledColocation;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasSize;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
Review Comment:
This is nice basic test coverage, but I'm afraid it still may be not enough for such a serious change. Let's add some more integration rebalance test with promoting/demotion of nodes (through zone replicas, quorum size, filter changes, for example):
- Scale up scenarios: if there are not enough peers, then peers should be added, otherwise learners should be added;
- Scale down scenarios: if there are learners, and the peer node goes offline, one of the learners should be turned into peer. This should not cause the actual data rebalancing (but the process should pass the regular assignments’ switch flow);
- Multi-step rebalance should be checked when a group leader is changed in the middle of pending assignments queue processing (checking that nothing is broken in ZoneRebalanceRaftGroupEventsListener#onLeaderElected );
- Scenario of the node restart during the multi-step rebalance should be checked (making sure that nothing is broken on node recovery, including the processing of pending assignments queue during the node recovery);
- Disaster recovery: if the majority of peers is lost, and there is no ability to turn a learner into peer (they may be offline, outdated, etc.), then disaster recovery should work as for regular zones without learners;
- Disaster recovery: the case when there are less nodes than configured quorum size, meaning that the configured value will be overridden.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "vadimkolodingg (via GitHub)" <gi...@apache.org>.
vadimkolodingg commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2097775498
##########
modules/partition-distribution/src/main/java/org/apache/ignite/internal/partitiondistribution/PendingAssignmentsCalculator.java:
##########
@@ -49,15 +53,79 @@ public PendingAssignmentsCalculator target(Assignments target) {
}
/**
- * Calculates the pending assignments queue between current stable and target assignments.
+ * Calculates the pending assignments queue between current stable (not included) and target assignments (included, last element).
*/
public AssignmentsQueue toQueue() {
assert stable != null;
assert target != null;
- // naive version with only one configuration switch, calculation will be changed during IGNITE-23790 epic
- AssignmentsQueue queue = new AssignmentsQueue(target);
- assert Objects.equals(queue.peekLast(), target) : "Target assignments should be equal to the last element in the queue";
+ if (target.force()) {
+ return new AssignmentsQueue(target);
+ }
Review Comment:
forcing assignments is part of recovery process where all stable can be dead, so there is no possible transition - need to start new raft group from scratch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "vadimkolodingg (via GitHub)" <gi...@apache.org>.
vadimkolodingg commented on code in PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825#discussion_r2158779992
##########
modules/distribution-zones/src/integrationTest/java/org/apache/ignite/internal/rebalance/ItRebalanceByPendingAssignmentsQueueTest.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.rebalance;
+
+import static java.util.Optional.ofNullable;
+import static org.apache.ignite.internal.TestRebalanceUtil.pendingPartitionAssignmentsKey;
+import static org.apache.ignite.internal.TestRebalanceUtil.stablePartitionAssignments;
+import static org.apache.ignite.internal.TestWrappers.unwrapIgniteImpl;
+import static org.apache.ignite.internal.TestWrappers.unwrapTableViewInternal;
+import static org.apache.ignite.internal.lang.IgniteSystemProperties.enabledColocation;
+import static org.apache.ignite.internal.partitiondistribution.PendingAssignmentsCalculator.pendingAssignmentsCalculator;
+import static org.apache.ignite.internal.rebalance.ItRebalanceByPendingAssignmentsQueueTest.AssignmentsRecorder.recordAssignmentsEvents;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.CompletableFutures.nullCompletedFuture;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
+import static org.hamcrest.Matchers.hasSize;
+
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.function.Consumer;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.internal.ClusterPerTestIntegrationTest;
+import org.apache.ignite.internal.app.IgniteImpl;
+import org.apache.ignite.internal.catalog.Catalog;
+import org.apache.ignite.internal.catalog.CatalogManager;
+import org.apache.ignite.internal.catalog.descriptors.CatalogZoneDescriptor;
+import org.apache.ignite.internal.distributionzones.rebalance.RebalanceUtil;
+import org.apache.ignite.internal.distributionzones.rebalance.ZoneRebalanceUtil;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.partitiondistribution.Assignment;
+import org.apache.ignite.internal.partitiondistribution.Assignments;
+import org.apache.ignite.internal.partitiondistribution.AssignmentsQueue;
+import org.apache.ignite.internal.raft.RaftGroupEventsListener;
+import org.apache.ignite.internal.replicator.PartitionGroupId;
+import org.apache.ignite.internal.replicator.TablePartitionId;
+import org.apache.ignite.internal.replicator.ZonePartitionId;
+import org.apache.ignite.internal.table.TableViewInternal;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for rebalance - handling pending {@link AssignmentsQueue} by {@link RaftGroupEventsListener}.
+ */
+class ItRebalanceByPendingAssignmentsQueueTest extends ClusterPerTestIntegrationTest {
Review Comment:
tests added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-24072 RaftGroupEventsListener process pending assignments queue [ignite-3]
Posted by "Pochatkin (via GitHub)" <gi...@apache.org>.
Pochatkin merged PR #5825:
URL: https://github.com/apache/ignite-3/pull/5825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org