You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "sergeyuttsel (via GitHub)" <gi...@apache.org> on 2023/02/28 13:26:57 UTC
[GitHub] [ignite-3] sergeyuttsel opened a new pull request, #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
sergeyuttsel opened a new pull request, #1729:
URL: https://github.com/apache/ignite-3/pull/1729
https://issues.apache.org/jira/browse/IGNITE-18756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1132317536
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -954,6 +1288,92 @@ public void onError(Throwable e) {
};
}
+ private WatchListener createMetastorageDataNodesListener() {
+ return new WatchListener() {
+ @Override
+ public void onUpdate(WatchEvent evt) {
+ if (!busyLock.enterBusy()) {
+ throw new IgniteInternalException(NODE_STOPPING_ERR, new NodeStoppingException());
+ }
+
+ try {
+ int zoneId = 0;
+
+ Set<String> newDataNodes = null;
+
+ long scaleUpRevision = 0;
+
+ long scaleDownRevision = 0;
+
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
+
+ if (startsWith(e.key(), zoneDataNodesKey().bytes())) {
+ zoneId = extractZoneId(e.key());
+
+ byte[] dataNodesBytes = e.value();
+
+ if (dataNodesBytes != null) {
+ newDataNodes = DistributionZonesUtil.dataNodes(fromBytes(dataNodesBytes));
+ } else {
+ newDataNodes = emptySet();
+ }
+ } else if (startsWith(e.key(), zoneScaleUpChangeTriggerKey().bytes())) {
+ if (e.value() != null) {
+ scaleUpRevision = bytesToLong(e.value());
+ }
+ } else if (startsWith(e.key(), zoneScaleDownChangeTriggerKey().bytes())) {
+ if (e.value() != null) {
+ scaleDownRevision = bytesToLong(e.value());
+ }
+ }
+ }
+
+ synchronized (dataNodesMutex) {
+ DataNodes dataNodesMeta = dataNodes.get(zoneId);
+
+ if (dataNodesMeta != null) {
+ dataNodesMeta.nodes(newDataNodes);
+
+ if (scaleUpRevision > 0) {
Review Comment:
This code was reworked but I think it will not be more clear if make util method for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163051670
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,910 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.DistributionZoneManager.ZoneState;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TableChange;
+import org.apache.ignite.internal.schema.configuration.TableConfiguration;
+import org.apache.ignite.internal.schema.configuration.TableView;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultEntry;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
Review Comment:
I've tried to simplified. Please review it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] alievmirza commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163605737
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -631,7 +820,15 @@ public void stop() throws Exception {
logicalTopologyService.removeEventListener(topologyEventListener);
- metaStorageManager.unregisterWatch(watchListener);
+ metaStorageManager.unregisterWatch(topologyWatchListener);
+ metaStorageManager.unregisterWatch(dataNodesWatchListener);
+
+ topVerTracker.update(Long.MAX_VALUE);
Review Comment:
Let's move it a constant value and add a comment, why Long.MAX_VALUE)
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -578,6 +759,10 @@ private ConfigurationListener<Integer> onUpdateScaleUp() {
zoneState.stopScaleUp();
}
+ if (newScaleUp > 0) {
Review Comment:
Please add explaining comment here, why we need to do `zoneState.scaleUpRevisionTracker().update(lastScaleUpRevision)` in this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156787450
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -917,24 +1213,69 @@ public CompletableFuture<Void> onUpdate(WatchEvent evt) {
}
try {
- assert evt.single() : "Expected an event with one entry but was an event with several entries with keys: "
+ assert evt.entryEvents().size() == 2 :
+ "Expected an event with logical topology and logical topology version entries but was events with keys: "
+ evt.entryEvents().stream().map(entry -> entry.newEntry() == null ? "null" : entry.newEntry().key())
.collect(toList());
- Entry newEntry = evt.entryEvent().newEntry();
+ long topVer = 0;
+
+ byte[] newLogicalTopologyBytes = null;
+
+ Set<String> newLogicalTopology = null;
+
+ long revision = 0;
- long revision = newEntry.revision();
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
- byte[] newLogicalTopologyBytes = newEntry.value();
+ if (Arrays.equals(e.key(), zonesLogicalTopologyVersionKey().bytes())) {
+ topVer = bytesToLong(e.value());
- Set<String> newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ revision = e.revision();
+ } else if (Arrays.equals(e.key(), zonesLogicalTopologyKey().bytes())) {
+ newLogicalTopologyBytes = e.value();
+
+ newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ }
+ }
+
+ assert newLogicalTopology != null;
+ assert revision > 0;
+
+ Set<String> newLogicalTopology0 = newLogicalTopology;
Set<String> removedNodes =
- logicalTopology.stream().filter(node -> !newLogicalTopology.contains(node)).collect(toSet());
+ logicalTopology.stream().filter(node -> !newLogicalTopology0.contains(node)).collect(toSet());
Set<String> addedNodes =
newLogicalTopology.stream().filter(node -> !logicalTopology.contains(node)).collect(toSet());
+ synchronized (dataNodesMutex) {
+ lastTopVer = topVer;
+
+ //Associates topology version and scale up meta storage revision.
+ if (!addedNodes.isEmpty()) {
+ topVerAndScaleUpRevision.put(topVer, revision);
+
+ topVerAndScaleUpRevision.headMap(topVer).clear();
+ }
+
+ //Associates topology version and scale down meta storage revision.
+ if (!removedNodes.isEmpty()) {
+ topVerAndScaleDownRevision.put(topVer, revision);
+
+ topVerAndScaleDownRevision.headMap(topVer).clear();
Review Comment:
Fixed. The map was removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1160505089
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +540,111 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ )
+ .thenCompose(ignored -> getDataNodesFuture(zoneId));
Review Comment:
formatting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156786051
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -500,6 +545,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null && zoneState.scaleUpRevision() < scaleUpRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ zoneState.revisionScaleUpFutures().ceilingEntry(scaleUpRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleUpFut0 = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleUpFut0 == null) {
+ topVerScaleUpFut0 = new CompletableFuture<>();
+
+ zoneState.revisionScaleUpFutures().put(scaleUpRevision, topVerScaleUpFut0);
+ }
+
+ topVerScaleUpFut0.handle((ignored0, e) -> {
+ if (e == null) {
+ topVerScaleUpFut.complete(null);
+ } else {
+ topVerScaleUpFut.completeExceptionally(e);
+ }
+
+ return null;
+ });
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ }
+ });
+
+ CompletableFuture<Void> topVerScaleDownFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ CompletableFuture<Void> topVerScaleDownFut0 = null;
+
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
Review Comment:
Fixed. As we discussed we will use this method in the configuration closure so we don't need to throw this exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1157139671
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
Review Comment:
What about future chaining here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1157533479
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
+
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = getImmediateTimers(zoneId, topVerFut);
+
+ CompletableFuture<Void> topVerScaleUpFut = scaleUpAwaiting(zoneId, timerValuesFut);
+
+ CompletableFuture<Void> topVerScaleDownFut = scaleDownAwaiting(zoneId, timerValuesFut);
+
+ return getDataNodesFuture(zoneId, topVerScaleUpFut, topVerScaleDownFut);
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ return topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ return completedFuture(null);
+ }
+ }
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be scheduled immediate.
+ *
+ * @param zoneId Zone id.
+ * @param topVerFut Future for chaining.
+ * @return Future.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId, CompletableFuture<Void> topVerFut) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
+
+ timerValuesFut.complete(new IgniteBiTuple<>(
+ zoneCfg.dataNodesAutoAdjustScaleUp().value() == 0,
+ zoneCfg.dataNodesAutoAdjustScaleDown().value() == 0
+ ));
+ });
+
+ return timerValuesFut;
+ }
+
+ /**
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than waits for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param timerValuesFut Future for chaining.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> scaleUpAwaiting(
+ int zoneId,
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut
+ ) {
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ timerValuesFut.thenAccept(timerValues -> {
+ boolean immediateScaleUp = timerValues.get1();
+
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp) {
+ if (zoneState.scaleUpRevision() < lastScaleUpRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
Review Comment:
Could you please explain what are you doing here? I mean not only the line, but the scaleUpAwaiting itself?
The code seams to be too verbose. I'd expect simpler logic here.
if (zonesState.scaleUpRevision < lastScaleUpRevision){
await();
}
By the way you have several places of similar logic for awaiting some long values: lastTopVer, lastScaleUpRevision, lastScaleDownRevision.
Instead of explicit future creation I'd rather implement some wrapper class e.g. LongAwaiter or even generic one with CompletableFuture<Long> await(value) method that will handle all that internally and atomically. Maybe we even have such class. It seams to be a sort of long -> long VersiionedValue.
With such Awaiter it'll be much easier to implement all that lastTopVer, lastScaleDownRevision awaiting logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1159663907
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
+
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = getImmediateTimers(zoneId, topVerFut);
+
+ CompletableFuture<Void> topVerScaleUpFut = scaleUpAwaiting(zoneId, timerValuesFut);
+
+ CompletableFuture<Void> topVerScaleDownFut = scaleDownAwaiting(zoneId, timerValuesFut);
+
+ return getDataNodesFuture(zoneId, topVerScaleUpFut, topVerScaleDownFut);
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ return topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ return completedFuture(null);
+ }
+ }
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be scheduled immediate.
+ *
+ * @param zoneId Zone id.
+ * @param topVerFut Future for chaining.
+ * @return Future.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId, CompletableFuture<Void> topVerFut) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
+
+ timerValuesFut.complete(new IgniteBiTuple<>(
+ zoneCfg.dataNodesAutoAdjustScaleUp().value() == 0,
Review Comment:
Yes. I created the constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156784302
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null && zoneState.scaleUpRevision() < scaleUpRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ zoneState.revisionScaleUpFutures().ceilingEntry(scaleUpRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleUpFut0 = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleUpFut0 == null) {
+ topVerScaleUpFut0 = new CompletableFuture<>();
+
+ zoneState.revisionScaleUpFutures().put(scaleUpRevision, topVerScaleUpFut0);
+ }
+
+ topVerScaleUpFut0.handle((ignored0, e) -> {
+ if (e == null) {
+ topVerScaleUpFut.complete(null);
+ } else {
+ topVerScaleUpFut.completeExceptionally(e);
+ }
+
+ return null;
+ });
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ }
+ });
+
+ CompletableFuture<Void> topVerScaleDownFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
Review Comment:
Actually the chaining is correct. I updated it to make it more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1132316122
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when dataNodesAutoAdjustScaleUp
+ * or dataNodesAutoAdjustScaleDown equals to 0.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private DistributionZonesConfiguration zonesConfiguration;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ @BeforeEach
+ void setUp() throws Exception {
+ VaultManager vaultManager = mock(VaultManager.class);
+
+ when(vaultManager.get(any())).thenReturn(completedFuture(null));
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ metaStorageManager = mock(MetaStorageManager.class);
+
+ doAnswer(invocation -> {
+ ByteArray key = invocation.getArgument(0);
+
+ WatchListener watchListener = invocation.getArgument(1);
+
+ if (Arrays.equals(key.bytes(), zoneLogicalTopologyPrefix().bytes())) {
+ topologyWatchListener = watchListener;
+ } else if (Arrays.equals(key.bytes(), zonesDataNodesPrefix().bytes())) {
+ dataNodesWatchListener = watchListener;
+ }
+
+ return null;
+ }).when(metaStorageManager).registerPrefixWatch(any(), any());
+
+ when(metaStorageManager.invoke(any())).thenReturn(completedFuture(StatementResultImpl.builder().result(new byte[] {0}).build()));
+
+ vaultManager.start();
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ LogicalTopologyServiceImpl logicalTopologyService = new LogicalTopologyServiceImpl(logicalTopology, cmgManager);
+
+ TablesConfiguration tablesConfiguration = mock(TablesConfiguration.class);
+
+ clusterCfgMgr = new ConfigurationManager(
+ List.of(DistributionZonesConfiguration.KEY),
+ Set.of(),
+ new TestConfigurationStorage(DISTRIBUTED),
+ List.of(),
+ List.of()
+ );
+
+ clusterCfgMgr.start();
+
+ zonesConfiguration = clusterCfgMgr.configurationRegistry()
+ .getConfiguration(DistributionZonesConfiguration.KEY);
+
+ distributionZoneManager = new DistributionZoneManager(
+ zonesConfiguration,
+ tablesConfiguration,
+ metaStorageManager,
+ logicalTopologyService,
+ vaultManager,
+ "node"
+ );
+
+ mockCmgLocalNodes();
+
+ distributionZoneManager.start();
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(0).dataNodesAutoAdjustScaleDown(0).build())
+ .get(3, TimeUnit.SECONDS);
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ distributionZoneManager.stop();
+ clusterCfgMgr.stop();
+ clusterStateStorage.stop();
+ }
+
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUpAndScaleDown() throws Exception {
+ LOG.info("Topology with added and removed nodes.");
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ Set<String> dataNodes = Set.of("node0", "node2");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ DistributionZoneManager.DataNodes dataNodesMeta = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ assertTrue(waitForCondition(() -> dataNodesMeta.revisionScaleUpFutures().size() == 1, 3_000));
+
+ assertTrue(testData.dataNodesMeta.revisionScaleDownFutures().isEmpty());
+
+ CompletableFuture<Void> revisionUpFut = testData.dataNodesMeta.revisionScaleUpFutures().get((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionUpFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, true, testData.dataNodesRevision2, testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(() -> revisionUpFut.isDone(), 3000));
+
+ assertTrue(waitForCondition(() -> dataNodesMeta.revisionScaleDownFutures().size() == 1, 3_000));
+
+ CompletableFuture<Void> revisionDownFut = testData.dataNodesMeta.revisionScaleDownFutures()
+ .get((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionDownFut.isDone());
+
+ assertFalse(testData.dataNodesUpFut3.isDone());
+ assertFalse(testData.dataNodesDownFut3.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, false, testData.dataNodesRevision2, testData.dataNodesRevision2 + 2);
+
+ assertTrue(waitForCondition(() -> revisionDownFut.isDone(), 3000));
+
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, TimeUnit.SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, TimeUnit.SECONDS));
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+ }
+
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUp() throws Exception {
+ LOG.info("Topology with added nodes.");
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ Set<String> dataNodes = Set.of("node0", "node1", "node2");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ DistributionZoneManager.DataNodes dataNodesMeta = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ assertTrue(waitForCondition(() -> dataNodesMeta.revisionScaleUpFutures().size() == 1, 3_000));
+
+ assertTrue(testData.dataNodesMeta.revisionScaleDownFutures().isEmpty());
+
+ CompletableFuture<Void> revisionUpFut = testData.dataNodesMeta.revisionScaleUpFutures().get((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionUpFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, true, testData.dataNodesRevision2, testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(() -> revisionUpFut.isDone(), 3000));
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, TimeUnit.SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, TimeUnit.SECONDS));
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+ }
+
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleDown() throws Exception {
+ LOG.info("Topology with removed nodes.");
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ Set<String> dataNodes = Set.of("node0");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ assertTrue(waitForCondition(() -> {
+ DistributionZoneManager.DataNodes dataNodesMeta0 = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ if (dataNodesMeta0 == null) {
+ return false;
+ }
+
+ return dataNodesMeta0.revisionScaleDownFutures().size() == 1;
+ },
+ 3_000));
+
+ assertTrue(testData.dataNodesMeta.revisionScaleUpFutures().isEmpty());
+
+ CompletableFuture<Void> revisionDownFut = testData.dataNodesMeta.revisionScaleDownFutures()
+ .get((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionDownFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, false, testData.dataNodesRevision2, testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(() -> revisionDownFut.isDone(), 3000));
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, TimeUnit.SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, TimeUnit.SECONDS));
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+ }
+
+ private static class TestSeveralScaleUpAndSeveralScaleDownDataObject {
+ private long topVer2;
+ private long dataNodesRevision2;
+ private CompletableFuture<Void> topVerUpFut2;
+ private CompletableFuture<Void> topVerDownFut2;
+ private DistributionZoneManager.DataNodes dataNodesMeta;
+ private CompletableFuture<Set<String>> dataNodesUpFut3;
+ private CompletableFuture<Set<String>> dataNodesDownFut3;
+
+ public TestSeveralScaleUpAndSeveralScaleDownDataObject(
+ long topVer2,
+ long dataNodesRevision2,
+ CompletableFuture<Void> topVerUpFut2,
+ CompletableFuture<Void> topVerDownFut2,
+ DistributionZoneManager.DataNodes dataNodesMeta,
+ CompletableFuture<Set<String>> dataNodesUpFut3,
+ CompletableFuture<Set<String>> dataNodesDownFut3) {
+ this.topVer2 = topVer2;
+ this.dataNodesRevision2 = dataNodesRevision2;
+ this.topVerUpFut2 = topVerUpFut2;
+ this.topVerDownFut2 = topVerDownFut2;
+ this.dataNodesMeta = dataNodesMeta;
+ this.dataNodesUpFut3 = dataNodesUpFut3;
+ this.dataNodesDownFut3 = dataNodesDownFut3;
+ }
+ }
+
+ private TestSeveralScaleUpAndSeveralScaleDownDataObject testSeveralScaleUpAndSeveralScaleDownGeneral() throws Exception {
+ LOG.info("Topology with added nodes.");
+
+ CompletableFuture<Set<String>> dataNodesUpFut0 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut1 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut2 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 2);
+ CompletableFuture<Set<String>> dataNodesUpFut3 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 11);
+
+ assertTrue(waitForCondition(() -> distributionZoneManager.topVerFutures().size() == 3,
+ 3_000));
+
+ int topVer0 = 2;
+
+ int dataNodesRevision0 = 2;
+
+ CompletableFuture<Void> topVerUpFut0 = distributionZoneManager.topVerFutures().get(1L);
+ CompletableFuture<Void> topVerUpFut1 = distributionZoneManager.topVerFutures().get(2L);
+ CompletableFuture<Void> topVerUpFut2 = distributionZoneManager.topVerFutures().get(11L);
+
+ assertFalse(topVerUpFut0.isDone());
+ assertFalse(topVerUpFut1.isDone());
+ assertFalse(topVerUpFut2.isDone());
+
+ Set<String> threeNodes = Set.of("node0", "node1", "node2");
+
+ topologyWatchListenerOnUpdate(threeNodes, topVer0, dataNodesRevision0);
+
+ assertTrue(waitForCondition(() -> topVerUpFut0.isDone(), 3_000));
+ assertTrue(waitForCondition(() -> topVerUpFut1.isDone(), 3_000));
+ assertFalse(topVerUpFut2.isDone());
+
+ assertTrue(waitForCondition(() -> {
+ DistributionZoneManager.DataNodes dataNodesMeta0 = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ if (dataNodesMeta0 == null) {
+ return false;
+ }
+
+ return dataNodesMeta0.revisionScaleUpFutures().size() == 1;
+ },
+ 3_000));
+
+ DistributionZoneManager.DataNodes dataNodesMeta = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ CompletableFuture<Void> revision2Fut = dataNodesMeta.revisionScaleUpFutures().get((long) dataNodesRevision0);
+
+ assertFalse(revision2Fut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, threeNodes, true, dataNodesRevision0, dataNodesRevision0 + 1);
+
+ assertTrue(waitForCondition(() -> revision2Fut.isDone(),
+ 3_000));
+
+ assertEquals(threeNodes, dataNodesUpFut0.get());
+ assertEquals(threeNodes, dataNodesUpFut1.get());
+ assertEquals(threeNodes, dataNodesUpFut2.get());
+ assertFalse(dataNodesUpFut3.isDone());
+
+ assertTrue(distributionZoneManager.topVerFutures().size() == 1);
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+
+
+ LOG.info("Topology with removed nodes.");
+
+ CompletableFuture<Set<String>> dataNodesDownFut0 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut1 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut2 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 5);
+ CompletableFuture<Set<String>> dataNodesDownFut3 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 6);
+
+ assertTrue(waitForCondition(() -> distributionZoneManager.topVerFutures().size() == 4,
+ 3_000));
+
+ CompletableFuture<Void> topVerDownFut0 = distributionZoneManager.topVerFutures().get(4L);
+ CompletableFuture<Void> topVerDownFut1 = distributionZoneManager.topVerFutures().get(5L);
+ CompletableFuture<Void> topVerDownFut2 = distributionZoneManager.topVerFutures().get(6L);
+
+ assertFalse(topVerDownFut0.isDone());
+ assertFalse(topVerDownFut1.isDone());
+ assertFalse(topVerDownFut2.isDone());
+
+ int topVer1 = 5;
+
+ int dataNodesRevision1 = dataNodesRevision0 + 2;
+
+ Set<String> twoNodes = Set.of("node0", "node1");
+
+ topologyWatchListenerOnUpdate(twoNodes, topVer1, dataNodesRevision1);
+
+ assertTrue(waitForCondition(() -> topVerDownFut0.isDone(), 3_000));
+ assertTrue(waitForCondition(() -> topVerDownFut1.isDone(), 3_000));
+ assertFalse(waitForCondition(() -> topVerDownFut2.isDone(), 3_000));
+
+ assertTrue(waitForCondition(() -> dataNodesMeta.revisionScaleDownFutures().size() == 1, 3_000));
+
+ CompletableFuture<Void> revision5Fut = dataNodesMeta.revisionScaleDownFutures().get((long) dataNodesRevision1);
+
+ assertFalse(revision5Fut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, twoNodes, false, dataNodesRevision1, dataNodesRevision1 + 1);
+
+ assertTrue(waitForCondition(() -> revision5Fut.isDone(), 3_000));
+
+ assertEquals(twoNodes, dataNodesDownFut0.get());
+ assertEquals(twoNodes, dataNodesDownFut1.get());
+ assertEquals(twoNodes, dataNodesDownFut2.get());
+ assertFalse(dataNodesDownFut3.isDone());
+
+ assertEquals(2, distributionZoneManager.topVerFutures().size());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+
+ int topVer2 = 20;
+
+ int dataNodesRevision2 = dataNodesRevision1 + 2;
+
+ return new TestSeveralScaleUpAndSeveralScaleDownDataObject(
+ topVer2,
+ dataNodesRevision2,
+ topVerUpFut2,
+ topVerDownFut2,
+ dataNodesMeta,
+ dataNodesUpFut3,
+ dataNodesDownFut3
+ );
+ }
+
+ @Test
+ void testScaleUpAndThenScaleDown() throws Exception {
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 5);
+
+ AtomicReference<CompletableFuture<Void>> topVerFut = new AtomicReference<>();
+
+ assertTrue(waitForCondition(() -> {
+ topVerFut.set(distributionZoneManager.topVerFutures().get(5L));
+
+ return topVerFut.get() != null;
+ },
+ 3_000));
+
+ assertFalse(topVerFut.get().isDone());
+
+ topologyWatchListenerOnUpdate(Set.of("node0", "node1"), 100, 10);
+
+ assertTrue(waitForCondition(() -> topVerFut.get().isDone(),
+ 3_000));
+
+ AtomicReference<CompletableFuture<Void>> revisionFut = new AtomicReference<>();
+
+ assertTrue(waitForCondition(() -> {
+ DistributionZoneManager.DataNodes dataNodes = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ if (dataNodes == null) {
+ return false;
+ }
+
+ revisionFut.set(dataNodes.revisionScaleUpFutures().get(10L));
+
+ return revisionFut.get() != null;
+ },
+ 3_000));
+
+ assertFalse(revisionFut.get().isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, Set.of("node0", "node1"), true, 10, 11);
+
+ assertTrue(waitForCondition(() -> revisionFut.get().isDone(),
+ 3_000));
+
+ assertEquals(Set.of("node0", "node1"), dataNodesFut.get());
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+
+ dataNodesFut = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 106);
+
+ topVerFut.set(null);
+
+ assertTrue(waitForCondition(() -> {
+ topVerFut.set(distributionZoneManager.topVerFutures().get(106L));
+
+ return topVerFut.get() != null;
+ },
+ 3_000));
+
+ assertFalse(topVerFut.get().isDone());
+
+ topologyWatchListenerOnUpdate(Set.of("node0"), 200, 12);
+
+ assertTrue(waitForCondition(() -> topVerFut.get().isDone(),
+ 3_000));
+
+ revisionFut.set(null);
+
+ assertTrue(waitForCondition(() -> {
+ DistributionZoneManager.DataNodes dataNodes = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ if (dataNodes == null) {
+ return false;
+ }
+
+ revisionFut.set(dataNodes.revisionScaleDownFutures().get(12L));
+
+ return revisionFut.get() != null;
+ },
+ 3_000));
+
+ assertFalse(revisionFut.get().isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, Set.of("node0"), false, 12, 13);
+
+ assertTrue(waitForCondition(() -> revisionFut.get().isDone(),
+ 3_000));
+
+ assertEquals(Set.of("node0"), dataNodesFut.get());
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+ }
+
+ @Test
+ void testAwaitingScaleUpOnly() throws Exception {
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(Integer.MAX_VALUE).dataNodesAutoAdjustScaleDown(Integer.MAX_VALUE).build())
+ .get(3, TimeUnit.SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(0)
+ .dataNodesAutoAdjustScaleDown(Integer.MAX_VALUE)
+ .build()
+ )
+ .get(3, TimeUnit.SECONDS);
+
+ int zoneId = distributionZoneManager.getZoneId("zone1");
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.getDataNodes(zoneId, 1);
+
+ CompletableFuture<Void> topVerUpFut = distributionZoneManager.topVerFutures().get(1L);
+
+ assertFalse(topVerUpFut.isDone());
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ topologyWatchListenerOnUpdate(nodes0, 1, 1);
+
+ dataNodesWatchListenerOnUpdate(zoneId, nodes0, true, 1, 2);
+
+ dataNodesFut.get(3, TimeUnit.SECONDS);
+
+ dataNodesFut = distributionZoneManager.getDataNodes(zoneId, 2);
+
+ Set<String> nodes1 = Set.of("node0");
+
+ topologyWatchListenerOnUpdate(nodes1, 2, 2);
+
+ dataNodesFut.get(3, TimeUnit.SECONDS);
+ }
+
+ @Test
+ void testAwaitingScaleDownOnly() throws Exception {
Review Comment:
Added asserts for other zones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] alievmirza commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1161731849
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,133 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ * The returned future can be completed with {@link DistributionZoneNotFoundException} and {@link DistributionZoneWasRemovedException}
+ * in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
Review Comment:
waits for observing, please add {@link } to topologyWatchListener
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,133 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ * The returned future can be completed with {@link DistributionZoneNotFoundException} and {@link DistributionZoneWasRemovedException}
+ * in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ return inBusyLock(busyLock, () -> topVerTracker.waitFor(topVer));
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be started immediate.
Review Comment:
immediately
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
Review Comment:
Comment from Alex is still actual, you have added implementation details of mechanism of awaiting, which is ok (but this details are too complicated and with grammar mistakes, please try to simplify it, maybe explain only one scenario with scale up, and mention that scale down logic is the same), but at the first place we must add some short explanation of the method purpose.
> The method for obtaining data nodes of the specified zone.
This is not just a method for obtaining data nodes, but it takes into account provided {@code topVer} and compares it with the explored by DZM topology version, please add short javadoc in the beginning so it could be clear what this method actually does
> @return The data nodes future.
This is also quite short, we should say that the completion of this future depends on the explored by DZM topology version etc.
Also please fix typos, like `than` -> `then`
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,910 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.DistributionZoneManager.ZoneState;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TableChange;
+import org.apache.ignite.internal.schema.configuration.TableConfiguration;
+import org.apache.ignite.internal.schema.configuration.TableView;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultEntry;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private DistributionZonesConfiguration zonesConfiguration;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ @BeforeEach
+ void setUp() {
+ vaultManager = mock(VaultManager.class);
+
+ when(vaultManager.get(zonesLogicalTopologyKey())).thenReturn(completedFuture(new VaultEntry(zonesLogicalTopologyKey(), null)));
+
+ when(vaultManager.get(zonesLogicalTopologyVersionKey()))
+ .thenReturn(completedFuture(new VaultEntry(zonesLogicalTopologyVersionKey(), longToBytes(0))));
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ metaStorageManager = mock(MetaStorageManager.class);
+
+ doAnswer(invocation -> {
+ ByteArray key = invocation.getArgument(0);
+
+ WatchListener watchListener = invocation.getArgument(1);
+
+ if (Arrays.equals(key.bytes(), zoneLogicalTopologyPrefix().bytes())) {
+ topologyWatchListener = watchListener;
+ } else if (Arrays.equals(key.bytes(), zonesDataNodesPrefix().bytes())) {
+ dataNodesWatchListener = watchListener;
+ }
+
+ return null;
+ }).when(metaStorageManager).registerPrefixWatch(any(), any());
+
+ when(metaStorageManager.invoke(any()))
+ .thenReturn(completedFuture(StatementResultImpl.builder().result(new byte[] {0}).build()));
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ LogicalTopologyServiceImpl logicalTopologyService = new LogicalTopologyServiceImpl(logicalTopology, cmgManager);
+
+ TablesConfiguration tablesConfiguration = mock(TablesConfiguration.class);
+
+ NamedConfigurationTree<TableConfiguration, TableView, TableChange> tables = mock(NamedConfigurationTree.class);
+
+ when(tablesConfiguration.tables()).thenReturn(tables);
+
+ NamedListView<TableView> tablesValue = mock(NamedListView.class);
+
+ when(tables.value()).thenReturn(tablesValue);
+
+ when(tablesValue.size()).thenReturn(0);
+
+ clusterCfgMgr = new ConfigurationManager(
+ List.of(DistributionZonesConfiguration.KEY),
+ Set.of(),
+ new TestConfigurationStorage(DISTRIBUTED),
+ List.of(),
+ List.of()
+ );
+
+ clusterCfgMgr.start();
+
+ zonesConfiguration = clusterCfgMgr.configurationRegistry()
+ .getConfiguration(DistributionZonesConfiguration.KEY);
+
+ distributionZoneManager = new DistributionZoneManager(
+ zonesConfiguration,
+ tablesConfiguration,
+ metaStorageManager,
+ logicalTopologyService,
+ vaultManager,
+ "node"
+ );
+
+ mockCmgLocalNodes();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ distributionZoneManager.stop();
+ clusterCfgMgr.stop();
+ clusterStateStorage.stop();
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added and removed nodes.
+ */
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUpAndScaleDown() throws Exception {
+ startZoneManager(0);
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ LOG.info("Topology with added and removed nodes.");
+
+ Set<String> dataNodes = Set.of("node0", "node2");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ CompletableFuture<Void> revisionUpFut = distributionZoneManager.zonesState().get(DEFAULT_ZONE_ID)
+ .scaleUpRevisionTracker().waitFor(testData.dataNodesRevision2);
+
+ assertFalse(revisionUpFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, true, testData.dataNodesRevision2,
+ testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(revisionUpFut::isDone, 3000));
+
+ CompletableFuture<Void> revisionDownFut = distributionZoneManager.zonesState().get(DEFAULT_ZONE_ID).scaleDownRevisionTracker()
+ .waitFor(testData.dataNodesRevision2);
+
+ assertFalse(revisionDownFut.isDone());
+
+ assertFalse(testData.dataNodesUpFut3.isDone());
+ assertFalse(testData.dataNodesDownFut3.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, false, testData.dataNodesRevision2,
+ testData.dataNodesRevision2 + 2);
+
+ assertTrue(waitForCondition(revisionDownFut::isDone, 3000));
+
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added nodes.
+ */
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUp() throws Exception {
+ startZoneManager(0);
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ LOG.info("Topology with added nodes.");
+
+ Set<String> dataNodes = Set.of("node0", "node1", "node2");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ CompletableFuture<Void> revisionUpFut = distributionZoneManager.zonesState().get(DEFAULT_ZONE_ID)
+ .scaleUpRevisionTracker().waitFor(testData.dataNodesRevision2);
+
+ assertFalse(revisionUpFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, true, testData.dataNodesRevision2,
+ testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(revisionUpFut::isDone, 3000));
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with removed nodes.
+ */
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleDown() throws Exception {
+ startZoneManager(0);
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ LOG.info("Topology with removed nodes.");
+
+ Set<String> dataNodes = Set.of("node0");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ CompletableFuture<Void> revisionDownFut = distributionZoneManager.zonesState().get(DEFAULT_ZONE_ID)
+ .scaleDownRevisionTracker().waitFor((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionDownFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, false, testData.dataNodesRevision2,
+ testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(revisionDownFut::isDone, 3000));
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, SECONDS));
+ }
+
+ private static class TestSeveralScaleUpAndSeveralScaleDownDataObject {
+ private final long topVer2;
+ private final long dataNodesRevision2;
+ private final CompletableFuture<Void> topVerUpFut2;
+ private final CompletableFuture<Void> topVerDownFut2;
+ private final CompletableFuture<Set<String>> dataNodesUpFut3;
+ private final CompletableFuture<Set<String>> dataNodesDownFut3;
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject(
+ long topVer2,
+ long dataNodesRevision2,
+ CompletableFuture<Void> topVerUpFut2,
+ CompletableFuture<Void> topVerDownFut2,
+ CompletableFuture<Set<String>> dataNodesUpFut3,
+ CompletableFuture<Set<String>> dataNodesDownFut3) {
+ this.topVer2 = topVer2;
+ this.dataNodesRevision2 = dataNodesRevision2;
+ this.topVerUpFut2 = topVerUpFut2;
+ this.topVerDownFut2 = topVerDownFut2;
+ this.dataNodesUpFut3 = dataNodesUpFut3;
+ this.dataNodesDownFut3 = dataNodesDownFut3;
+ }
+ }
+
+ /**
+ * This method invokes {@link DistributionZoneManager#topologyVersionedDataNodes(int, long)} with default and non-default zone id
+ * and different logical topology version. Collects data nodes futures.
+ * Simulates new logical topology with new nodes. Check that some of data nodes futures are completed.
+ * Simulates new logical topology with removed nodes. Check that some of data nodes futures are completed.
+ *
+ * @return Structure with data for continue testing.
+ */
+ private TestSeveralScaleUpAndSeveralScaleDownDataObject testSeveralScaleUpAndSeveralScaleDownGeneral()
Review Comment:
I do not understand this concept with `TestSeveralScaleUpAndSeveralScaleDownDataObject`, many things are done in one method, it is hard to read this code, please do something with that.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,133 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ * The returned future can be completed with {@link DistributionZoneNotFoundException} and {@link DistributionZoneWasRemovedException}
+ * in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ return inBusyLock(busyLock, () -> topVerTracker.waitFor(topVer));
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be started immediate.
+ *
+ * @param zoneId Zone id.
+ * @return Future with the result.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId) {
+ return inBusyLock(busyLock, () -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
+
+ return completedFuture(new IgniteBiTuple<>(
+ zoneCfg.dataNodesAutoAdjustScaleUp().value() == IMMEDIATE_TIMER_VALUE,
+ zoneCfg.dataNodesAutoAdjustScaleDown().value() == IMMEDIATE_TIMER_VALUE
+ ));
+ });
+ }
+
+ /**
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than waits for writing data nodes triggered
Review Comment:
Also it could be useful to mention case with `DistributionZoneWasRemovedException`
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -898,45 +1083,62 @@ private void initDataNodesFromVaultManager() {
}
try {
- vaultMgr.get(zonesLogicalTopologyKey())
- .thenAccept(vaultEntry -> {
- if (!busyLock.enterBusy()) {
- throw new IgniteInternalException(NODE_STOPPING_ERR, new NodeStoppingException());
- }
+ long appliedRevision = metaStorageManager.appliedRevision();
- try {
- long appliedRevision = metaStorageManager.appliedRevision();
+ lastScaleUpRevision = appliedRevision;
- if (vaultEntry != null && vaultEntry.value() != null) {
- logicalTopology = fromBytes(vaultEntry.value());
+ lastScaleDownRevision = appliedRevision;
- // init keys and data nodes for default zone
- saveDataNodesAndUpdateTriggerKeysInMetaStorage(
- DEFAULT_ZONE_ID,
- appliedRevision,
- logicalTopology
- );
+ VaultEntry topVerEntry = vaultMgr.get(zonesLogicalTopologyVersionKey()).join();
- zonesConfiguration.distributionZones().value().forEach(zone -> {
- int zoneId = zone.zoneId();
+ if (topVerEntry != null && topVerEntry.value() != null) {
+ topVerTracker.update(bytesToLong(topVerEntry.value()));
+ }
- saveDataNodesAndUpdateTriggerKeysInMetaStorage(
- zoneId,
- appliedRevision,
- logicalTopology
- );
- });
- }
- } finally {
- busyLock.leaveBusy();
- }
- });
+ VaultEntry topologyEntry = vaultMgr.get(zonesLogicalTopologyKey()).join();
+
+ if (topologyEntry != null && topologyEntry.value() != null) {
+ logicalTopology = fromBytes(topologyEntry.value());
+
+ // init keys and data nodes for default zone
+ saveDataNodesAndUpdateTriggerKeysInMetaStorage(
+ DEFAULT_ZONE_ID,
+ appliedRevision,
+ logicalTopology
+ );
+
+ zonesConfiguration.distributionZones().value().forEach(zone -> {
+ int zoneId = zone.zoneId();
+
+ saveDataNodesAndUpdateTriggerKeysInMetaStorage(
+ zoneId,
+ appliedRevision,
+ logicalTopology
+ );
+ });
+ }
+
+ zonesState.values().forEach(zoneState -> {
+ zoneState.scaleUpRevisionTracker().update(lastScaleUpRevision);
+
+ zoneState.scaleDownRevisionTracker().update(lastScaleDownRevision);
+
+ zoneState.nodes(logicalTopology);
+ });
+
+ assert topologyEntry == null || topologyEntry.value() == null || logicalTopology.equals(fromBytes(topologyEntry.value()))
+ : "DistributionZoneManager.logicalTopology was changed after initialization from the vault manager.";
Review Comment:
Please, remove `DistributionZoneManager.logicalTopology` from this message and try to rephrase it. This code can seen by user
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,133 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ * The returned future can be completed with {@link DistributionZoneNotFoundException} and {@link DistributionZoneWasRemovedException}
+ * in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ return inBusyLock(busyLock, () -> topVerTracker.waitFor(topVer));
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be started immediate.
+ *
+ * @param zoneId Zone id.
+ * @return Future with the result.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId) {
+ return inBusyLock(busyLock, () -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
+
+ return completedFuture(new IgniteBiTuple<>(
+ zoneCfg.dataNodesAutoAdjustScaleUp().value() == IMMEDIATE_TIMER_VALUE,
+ zoneCfg.dataNodesAutoAdjustScaleDown().value() == IMMEDIATE_TIMER_VALUE
+ ));
+ });
+ }
+
+ /**
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than waits for writing data nodes triggered
Review Comment:
Please rephrase this javadoc and one below, it is really hard to understand what is going on here
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -945,24 +1147,57 @@ public CompletableFuture<Void> onUpdate(WatchEvent evt) {
}
try {
- assert evt.single() : "Expected an event with one entry but was an event with several entries with keys: "
+ assert evt.entryEvents().size() == 2 :
+ "Expected an event with logical topology and logical topology version entries but was events with keys: "
+ evt.entryEvents().stream().map(entry -> entry.newEntry() == null ? "null" : entry.newEntry().key())
.collect(toList());
- Entry newEntry = evt.entryEvent().newEntry();
+ long topVer = 0;
+
+ byte[] newLogicalTopologyBytes = null;
+
+ Set<String> newLogicalTopology = null;
+
+ long revision = 0;
+
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
- long revision = newEntry.revision();
+ if (Arrays.equals(e.key(), zonesLogicalTopologyVersionKey().bytes())) {
+ topVer = bytesToLong(e.value());
- byte[] newLogicalTopologyBytes = newEntry.value();
+ revision = e.revision();
+ } else if (Arrays.equals(e.key(), zonesLogicalTopologyKey().bytes())) {
+ newLogicalTopologyBytes = e.value();
- Set<String> newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ }
+ }
+
+ assert newLogicalTopology != null;
+ assert revision > 0;
+
+ Set<String> newLogicalTopology0 = newLogicalTopology;
Set<String> removedNodes =
- logicalTopology.stream().filter(node -> !newLogicalTopology.contains(node)).collect(toSet());
+ logicalTopology.stream().filter(node -> !newLogicalTopology0.contains(node)).collect(toSet());
Set<String> addedNodes =
newLogicalTopology.stream().filter(node -> !logicalTopology.contains(node)).collect(toSet());
+ if (!addedNodes.isEmpty()) {
+ lastScaleUpRevision = revision;
+ }
+
+ if (!removedNodes.isEmpty()) {
+ lastScaleDownRevision = revision;
+ }
+
+ //The topology version must be updated after the lastScaleUpRevision and lastScaleDownRevision are updated.
+ //This is necessary in order to when topology version waiters will be notified that topology version is updated
Review Comment:
please try to rephrase the second sentence, it is unclear what you wanted to say
##########
modules/runner/src/integrationTest/java/org/apache/ignite/internal/sql/api/ItSqlSynchronousApiTest.java:
##########
@@ -92,6 +93,8 @@ protected IgniteSql igniteSql() {
return CLUSTER_NODES.get(0).sql();
}
+
+ @Disabled
Review Comment:
Disabled without a ticket. Are this issues still valid?
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,910 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.DistributionZoneManager.ZoneState;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TableChange;
+import org.apache.ignite.internal.schema.configuration.TableConfiguration;
+import org.apache.ignite.internal.schema.configuration.TableView;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultEntry;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private DistributionZonesConfiguration zonesConfiguration;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ @BeforeEach
+ void setUp() {
+ vaultManager = mock(VaultManager.class);
+
+ when(vaultManager.get(zonesLogicalTopologyKey())).thenReturn(completedFuture(new VaultEntry(zonesLogicalTopologyKey(), null)));
+
+ when(vaultManager.get(zonesLogicalTopologyVersionKey()))
+ .thenReturn(completedFuture(new VaultEntry(zonesLogicalTopologyVersionKey(), longToBytes(0))));
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ metaStorageManager = mock(MetaStorageManager.class);
+
+ doAnswer(invocation -> {
+ ByteArray key = invocation.getArgument(0);
+
+ WatchListener watchListener = invocation.getArgument(1);
+
+ if (Arrays.equals(key.bytes(), zoneLogicalTopologyPrefix().bytes())) {
+ topologyWatchListener = watchListener;
+ } else if (Arrays.equals(key.bytes(), zonesDataNodesPrefix().bytes())) {
+ dataNodesWatchListener = watchListener;
+ }
+
+ return null;
+ }).when(metaStorageManager).registerPrefixWatch(any(), any());
+
+ when(metaStorageManager.invoke(any()))
+ .thenReturn(completedFuture(StatementResultImpl.builder().result(new byte[] {0}).build()));
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ LogicalTopologyServiceImpl logicalTopologyService = new LogicalTopologyServiceImpl(logicalTopology, cmgManager);
+
+ TablesConfiguration tablesConfiguration = mock(TablesConfiguration.class);
Review Comment:
Please rewrite it to a `@InjectConfiguration`
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -945,24 +1147,57 @@ public CompletableFuture<Void> onUpdate(WatchEvent evt) {
}
try {
- assert evt.single() : "Expected an event with one entry but was an event with several entries with keys: "
+ assert evt.entryEvents().size() == 2 :
+ "Expected an event with logical topology and logical topology version entries but was events with keys: "
+ evt.entryEvents().stream().map(entry -> entry.newEntry() == null ? "null" : entry.newEntry().key())
.collect(toList());
- Entry newEntry = evt.entryEvent().newEntry();
+ long topVer = 0;
+
+ byte[] newLogicalTopologyBytes = null;
+
+ Set<String> newLogicalTopology = null;
+
+ long revision = 0;
+
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
- long revision = newEntry.revision();
+ if (Arrays.equals(e.key(), zonesLogicalTopologyVersionKey().bytes())) {
+ topVer = bytesToLong(e.value());
- byte[] newLogicalTopologyBytes = newEntry.value();
+ revision = e.revision();
+ } else if (Arrays.equals(e.key(), zonesLogicalTopologyKey().bytes())) {
+ newLogicalTopologyBytes = e.value();
- Set<String> newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ }
+ }
+
+ assert newLogicalTopology != null;
Review Comment:
asserts without a message
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,910 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.DistributionZoneManager.ZoneState;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TableChange;
+import org.apache.ignite.internal.schema.configuration.TableConfiguration;
+import org.apache.ignite.internal.schema.configuration.TableView;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultEntry;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
Review Comment:
Please do something with the readability of this class, it is hard to understand scenarios, javadocs and test names do not help, `TestSeveralScaleUpAndSeveralScaleDownDataObject` looks too complicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163726630
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,148 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining the data nodes of the specified zone.
+ * If {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} and
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate then it waits that the data nodes
+ * are up-to-date for the passed topology version.
+ *
+ * <p>If the values of auto adjust scale up and auto adjust scale down are zero, then on the cluster topology changes
+ * the data nodes for the zone should be updated immediately. Therefore, this method must return the data nodes which is calculated
+ * based on the topology with passed or greater version. Since the date nodes value is updated asynchronously, this method waits for
+ * the date nodes to be updated with new nodes in the topology if the value of auto adjust scale up is 0. And also waits for
+ * the date nodes to be updated with nodes that have left the topology if the value of auto adjust scale down is 0.
+ * After the zone manager has observed the logical topology change and the data nodes value is updated according to cluster topology,
+ * then this method completes the returned future with the current value of data nodes.
+ *
+ * <p>If the value of auto adjust scale up is greater than zero, then it is not necessary to wait for the data nodes update triggered
+ * by new nodes in cluster topology. Similarly if the value of auto adjust scale down is greater than zero, then it is not necessary to
+ * wait for the data nodes update triggered by new nodes that have left the topology in cluster topology.
+ *
+ * <p>The returned future can be completed with {@link DistributionZoneNotFoundException} and
+ * {@link DistributionZoneWasRemovedException} in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future which will be completed with data nodes for the zoneId or with exception.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager waits for observing passed topology version or greater version
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] vldpyatkov merged pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "vldpyatkov (via GitHub)" <gi...@apache.org>.
vldpyatkov merged PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163729678
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -578,6 +759,10 @@ private ConfigurationListener<Integer> onUpdateScaleUp() {
zoneState.stopScaleUp();
}
+ if (newScaleUp > 0) {
Review Comment:
I've added comment with explanations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1131487090
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
Review Comment:
Fixed.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ boolean immediateScaleUp = immediateScaleUp0;
+ boolean immediateScaleDown = immediateScaleDown0;
+
+ if (!immediateScaleUp && !immediateScaleDown) {
+
+ DataNodes dataNodes = this.dataNodes.get(zoneId);
+
+ Set<String> nodes;
+
+ if (dataNodes == null) {
+ nodes = emptySet();
+ } else {
+ nodes = this.dataNodes.get(zoneId).nodes();
+ }
+
+ return completedFuture(nodes);
+ }
+
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.get(topVer);
+
+ if (topVerFut == null) {
+ topVerFut = new CompletableFuture<>();
+
+ topVerFutures.put(topVer, topVerFut);
+ }
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ topVerFut.thenAcceptAsync(ignored -> {
+ synchronized (dataNodesMutex) {
+ CompletableFuture<Void> topVerScaleUpFut = null;
+
+ if (immediateScaleUp) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null
+ && (dataNodes.get(zoneId) == null || dataNodes.get(zoneId).scaleUpRevision() < scaleUpRevision)) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ dataNodes.get(zoneId).revisionScaleUpFutures().ceilingEntry(scaleUpRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleUpFut = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleUpFut == null) {
+ topVerScaleUpFut = new CompletableFuture<>();
+
+ dataNodes.get(zoneId).revisionScaleUpFutures().put(scaleUpRevision, topVerScaleUpFut);
+ }
+ } else {
+ topVerScaleUpFut = completedFuture(null);
+ }
+ } else {
+ topVerScaleUpFut = completedFuture(null);
+ }
+
+ topVerScaleUpFut.thenAcceptAsync(ignored0 -> {
+ CompletableFuture<Void> topVerScaleDownFut = null;
+
+ synchronized (dataNodesMutex) {
+ if (immediateScaleDown) {
+ Map.Entry<Long, Long> scaleDownRevisionEntry = topVerAndScaleDownRevision.ceilingEntry(topVer);
+
+ Long scaleDownRevision = null;
+
+ if (scaleDownRevisionEntry != null) {
+ scaleDownRevision = scaleDownRevisionEntry.getValue();
+ }
+
+ if (scaleDownRevision != null
+ && (dataNodes.get(zoneId) == null || dataNodes.get(zoneId).scaleDownRevision() < scaleDownRevision)) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ dataNodes.get(zoneId).revisionScaleDownFutures().ceilingEntry(scaleDownRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleDownFut = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleDownFut == null) {
+ topVerScaleDownFut = new CompletableFuture<>();
+
+ dataNodes.get(zoneId).revisionScaleDownFutures.put(scaleDownRevision, topVerScaleDownFut);
+ }
+ } else {
+ topVerScaleDownFut = completedFuture(null);
+ }
+ } else {
+ topVerScaleDownFut = completedFuture(null);
+ }
+ }
+
+ topVerScaleDownFut.thenAcceptAsync(ignored1 -> {
Review Comment:
I rework it. Seems now it is more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1152768257
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -163,6 +176,30 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** Data nodes modification mutex. */
+ private final Object dataNodesMutex = new Object();
+
+ /** The last topology version which was observed by distribution zone manager. */
+ private long lastTopVer;
Review Comment:
Seems that lastTopVer onStart initializatoin is missing. And, probably, corresponding test is also missing. Same is about topVerAndScaleUpRevision etc.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -917,24 +1213,69 @@ public CompletableFuture<Void> onUpdate(WatchEvent evt) {
}
try {
- assert evt.single() : "Expected an event with one entry but was an event with several entries with keys: "
+ assert evt.entryEvents().size() == 2 :
+ "Expected an event with logical topology and logical topology version entries but was events with keys: "
+ evt.entryEvents().stream().map(entry -> entry.newEntry() == null ? "null" : entry.newEntry().key())
.collect(toList());
- Entry newEntry = evt.entryEvent().newEntry();
+ long topVer = 0;
+
+ byte[] newLogicalTopologyBytes = null;
+
+ Set<String> newLogicalTopology = null;
+
+ long revision = 0;
- long revision = newEntry.revision();
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
- byte[] newLogicalTopologyBytes = newEntry.value();
+ if (Arrays.equals(e.key(), zonesLogicalTopologyVersionKey().bytes())) {
+ topVer = bytesToLong(e.value());
- Set<String> newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ revision = e.revision();
+ } else if (Arrays.equals(e.key(), zonesLogicalTopologyKey().bytes())) {
+ newLogicalTopologyBytes = e.value();
+
+ newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ }
+ }
+
+ assert newLogicalTopology != null;
+ assert revision > 0;
+
+ Set<String> newLogicalTopology0 = newLogicalTopology;
Set<String> removedNodes =
- logicalTopology.stream().filter(node -> !newLogicalTopology.contains(node)).collect(toSet());
+ logicalTopology.stream().filter(node -> !newLogicalTopology0.contains(node)).collect(toSet());
Set<String> addedNodes =
newLogicalTopology.stream().filter(node -> !logicalTopology.contains(node)).collect(toSet());
+ synchronized (dataNodesMutex) {
+ lastTopVer = topVer;
+
+ //Associates topology version and scale up meta storage revision.
+ if (!addedNodes.isEmpty()) {
+ topVerAndScaleUpRevision.put(topVer, revision);
+
+ topVerAndScaleUpRevision.headMap(topVer).clear();
+ }
+
+ //Associates topology version and scale down meta storage revision.
+ if (!removedNodes.isEmpty()) {
+ topVerAndScaleDownRevision.put(topVer, revision);
+
+ topVerAndScaleDownRevision.headMap(topVer).clear();
Review Comment:
So, topVerAndScaleDownRevision is a map of one element only?
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -163,6 +176,30 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** Data nodes modification mutex. */
+ private final Object dataNodesMutex = new Object();
+
+ /** The last topology version which was observed by distribution zone manager. */
+ private long lastTopVer;
+
+ /**
+ * The map contains futures which are completed when zone manager observe appropriate logical topology version.
+ * Map (topology version -> future).
+ */
+ private final NavigableMap<Long, CompletableFuture<Void>> topVerFutures;
+
+ /**
+ * The map stores the correspondence between logical topology version and meta storage revision
+ * on which scale up timer was started.
+ */
+ private final NavigableMap<Long, Long> topVerAndScaleUpRevision;
Review Comment:
Seems that cleanup is missing.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
Review Comment:
Why do you need up0/down0? I believe that it's possible to use atomics directly.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
Review Comment:
Naming is still confusing.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
Review Comment:
According to the javadoc it's not clear what we will get from the method if proposed topVer already passed. E.g. currently it's topVer == 10, and we ask topVer ==5.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
Review Comment:
It's a bit confusing.
You have a map
with one element only
```
topVerAndScaleUpRevision.put(topVer, revision);
topVerAndScaleUpRevision.headMap(topVer).clear();
```
and you read only one element, the last one
`topVerAndScaleUpRevision.ceilingEntry(topVer);`
Why? ))
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
Review Comment:
I believe that we already have zone by id utility method, if not - please add.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
Review Comment:
I'd rather add several private methods like
1. awaitTopologyVersion(topVer)
2. isScaleUpImmediate(zoneId) / isScaleDownImmediate(zoneId)
3. ...
in order to use them inside topologyVersionDataNodes.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
Review Comment:
What if there's no corresponding zone with given Id on a local node? Is it possible?
Please consider following flow:
1. Node A:: create zone1.
2. Node B:: create table with zone = zone1.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null && zoneState.scaleUpRevision() < scaleUpRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ zoneState.revisionScaleUpFutures().ceilingEntry(scaleUpRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleUpFut0 = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleUpFut0 == null) {
+ topVerScaleUpFut0 = new CompletableFuture<>();
+
+ zoneState.revisionScaleUpFutures().put(scaleUpRevision, topVerScaleUpFut0);
+ }
+
+ topVerScaleUpFut0.handle((ignored0, e) -> {
+ if (e == null) {
+ topVerScaleUpFut.complete(null);
+ } else {
+ topVerScaleUpFut.completeExceptionally(e);
+ }
+
+ return null;
+ });
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ }
+ });
+
+ CompletableFuture<Void> topVerScaleDownFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
Review Comment:
The chaining is incorrect. 651::topVerFut.thenAccept will be executed before 598::topVerFut.thenAccept, so that immediateScaleDown will have uninitialized value (false). By the way, why there are no failing tests showing that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163050101
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -898,45 +1083,62 @@ private void initDataNodesFromVaultManager() {
}
try {
- vaultMgr.get(zonesLogicalTopologyKey())
- .thenAccept(vaultEntry -> {
- if (!busyLock.enterBusy()) {
- throw new IgniteInternalException(NODE_STOPPING_ERR, new NodeStoppingException());
- }
+ long appliedRevision = metaStorageManager.appliedRevision();
- try {
- long appliedRevision = metaStorageManager.appliedRevision();
+ lastScaleUpRevision = appliedRevision;
- if (vaultEntry != null && vaultEntry.value() != null) {
- logicalTopology = fromBytes(vaultEntry.value());
+ lastScaleDownRevision = appliedRevision;
- // init keys and data nodes for default zone
- saveDataNodesAndUpdateTriggerKeysInMetaStorage(
- DEFAULT_ZONE_ID,
- appliedRevision,
- logicalTopology
- );
+ VaultEntry topVerEntry = vaultMgr.get(zonesLogicalTopologyVersionKey()).join();
- zonesConfiguration.distributionZones().value().forEach(zone -> {
- int zoneId = zone.zoneId();
+ if (topVerEntry != null && topVerEntry.value() != null) {
+ topVerTracker.update(bytesToLong(topVerEntry.value()));
+ }
- saveDataNodesAndUpdateTriggerKeysInMetaStorage(
- zoneId,
- appliedRevision,
- logicalTopology
- );
- });
- }
- } finally {
- busyLock.leaveBusy();
- }
- });
+ VaultEntry topologyEntry = vaultMgr.get(zonesLogicalTopologyKey()).join();
+
+ if (topologyEntry != null && topologyEntry.value() != null) {
+ logicalTopology = fromBytes(topologyEntry.value());
+
+ // init keys and data nodes for default zone
+ saveDataNodesAndUpdateTriggerKeysInMetaStorage(
+ DEFAULT_ZONE_ID,
+ appliedRevision,
+ logicalTopology
+ );
+
+ zonesConfiguration.distributionZones().value().forEach(zone -> {
+ int zoneId = zone.zoneId();
+
+ saveDataNodesAndUpdateTriggerKeysInMetaStorage(
+ zoneId,
+ appliedRevision,
+ logicalTopology
+ );
+ });
+ }
+
+ zonesState.values().forEach(zoneState -> {
+ zoneState.scaleUpRevisionTracker().update(lastScaleUpRevision);
+
+ zoneState.scaleDownRevisionTracker().update(lastScaleDownRevision);
+
+ zoneState.nodes(logicalTopology);
+ });
+
+ assert topologyEntry == null || topologyEntry.value() == null || logicalTopology.equals(fromBytes(topologyEntry.value()))
+ : "DistributionZoneManager.logicalTopology was changed after initialization from the vault manager.";
Review Comment:
I've tried to fix it. Please check it again.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -945,24 +1147,57 @@ public CompletableFuture<Void> onUpdate(WatchEvent evt) {
}
try {
- assert evt.single() : "Expected an event with one entry but was an event with several entries with keys: "
+ assert evt.entryEvents().size() == 2 :
+ "Expected an event with logical topology and logical topology version entries but was events with keys: "
+ evt.entryEvents().stream().map(entry -> entry.newEntry() == null ? "null" : entry.newEntry().key())
.collect(toList());
- Entry newEntry = evt.entryEvent().newEntry();
+ long topVer = 0;
+
+ byte[] newLogicalTopologyBytes = null;
+
+ Set<String> newLogicalTopology = null;
+
+ long revision = 0;
+
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
- long revision = newEntry.revision();
+ if (Arrays.equals(e.key(), zonesLogicalTopologyVersionKey().bytes())) {
+ topVer = bytesToLong(e.value());
- byte[] newLogicalTopologyBytes = newEntry.value();
+ revision = e.revision();
+ } else if (Arrays.equals(e.key(), zonesLogicalTopologyKey().bytes())) {
+ newLogicalTopologyBytes = e.value();
- Set<String> newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ }
+ }
+
+ assert newLogicalTopology != null;
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1132038085
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -159,6 +167,36 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** Data nodes modification mutex. */
+ private final Object dataNodesMutex = new Object();
+
+ /** The last topology version wich was observed by distribution zone manager. */
+ private long lastTopVer;
+
+ /**
+ * Contains data nodes and meta info for zones.
+ * Map (zone id -> data nodes).
+ */
+ private final Map<Integer, DataNodes> dataNodes = new ConcurrentHashMap<>();
Review Comment:
Yes. I think we can use HashMap and TreeMap. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1131487320
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -954,6 +1288,92 @@ public void onError(Throwable e) {
};
}
+ private WatchListener createMetastorageDataNodesListener() {
Review Comment:
Added java doc and comments into the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] alievmirza commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1134277628
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -209,17 +253,21 @@ public DistributionZoneManager(
this.logicalTopologyService = logicalTopologyService;
this.vaultMgr = vaultMgr;
- this.watchListener = createMetastorageListener();
+ this.topologyWatchListener = createMetastorageTopologyListener();
+
+ this.dataNodesWatchListener = createMetastorageDataNodesListener();
zonesState = new ConcurrentHashMap<>();
- logicalTopology = Collections.emptySet();
+ logicalTopology = emptySet();
executor = new ScheduledThreadPoolExecutor(
Math.min(Runtime.getRuntime().availableProcessors() * 3, 20),
new NamedThreadFactory(NamedThreadFactory.threadPrefix(nodeName, DISTRIBUTION_ZONE_MANAGER_POOL_NAME), LOG),
new ThreadPoolExecutor.DiscardPolicy()
);
+
+ dataNodes.put(DEFAULT_ZONE_ID, new DataNodes());
Review Comment:
`dataNodes` is not thread safe and you put value to it without synchronisation. Also, I would move creation of `dataNodes` from class filed to the constructor.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +550,183 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown}
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
Review Comment:
What should we return in this method in scenario, when scale up / down are non-null, and we restart some node? So, scenario is: we had cluster with data nodes [A,B,C], C was restarted, and some table is going to be created from C after restart, which data nodes we should get to calculate assignments? In your code, it seems that we will return an empty data nodes, is it a correct behaviour?
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +550,183 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown}
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ dataNodes.putIfAbsent(zoneId, new DataNodes());
+
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ boolean immediateScaleUp = immediateScaleUp0;
+ boolean immediateScaleDown = immediateScaleDown0;
+
+ if (!immediateScaleUp && !immediateScaleDown) {
+ synchronized (dataNodesMutex) {
+
+ DataNodes dataNodes = this.dataNodes.get(zoneId);
+
+ Set<String> nodes;
+
+ if (dataNodes == null) {
+ nodes = emptySet();
+ } else {
+ nodes = this.dataNodes.get(zoneId).nodes();
+ }
+
+
+ return completedFuture(nodes);
+ }
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null && dataNodes.get(zoneId).scaleUpRevision() < scaleUpRevision) {
Review Comment:
seems like it is possible that while we were waiting for completion of `topVerFut`, zone was deleted, potential NPE here
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +550,183 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown}
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
Review Comment:
Note, that I'm not talking about the first start of the cluster, where we agreed to return an empty data nodes, as far as user set non-immediate scale up/down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1135747291
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +550,183 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown}
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
Review Comment:
I tried to fix it. Please look at changes in DistributionZoneManager#initDataNodesFromVaultManager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1132483363
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ boolean immediateScaleUp = immediateScaleUp0;
+ boolean immediateScaleDown = immediateScaleDown0;
+
+ if (!immediateScaleUp && !immediateScaleDown) {
+
+ DataNodes dataNodes = this.dataNodes.get(zoneId);
+
+ Set<String> nodes;
+
+ if (dataNodes == null) {
+ nodes = emptySet();
+ } else {
+ nodes = this.dataNodes.get(zoneId).nodes();
+ }
+
+ return completedFuture(nodes);
+ }
+
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.get(topVer);
+
+ if (topVerFut == null) {
+ topVerFut = new CompletableFuture<>();
+
+ topVerFutures.put(topVer, topVerFut);
Review Comment:
Thanks, I go with `computeIfAbsent`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156780212
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
Review Comment:
Fixed. AtomicBoolean was removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1157209171
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
+
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = getImmediateTimers(zoneId, topVerFut);
+
+ CompletableFuture<Void> topVerScaleUpFut = scaleUpAwaiting(zoneId, timerValuesFut);
+
+ CompletableFuture<Void> topVerScaleDownFut = scaleDownAwaiting(zoneId, timerValuesFut);
+
+ return getDataNodesFuture(zoneId, topVerScaleUpFut, topVerScaleDownFut);
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ synchronized (dataNodesMutex) {
Review Comment:
Do you really need `synchronized (dataNodesMutex)` or it's possible to use non-blocking approach? In other words, why do you need `synchronized (dataNodesMutex)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1164133367
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -631,7 +820,15 @@ public void stop() throws Exception {
logicalTopologyService.removeEventListener(topologyEventListener);
- metaStorageManager.unregisterWatch(watchListener);
+ metaStorageManager.unregisterWatch(topologyWatchListener);
+ metaStorageManager.unregisterWatch(dataNodesWatchListener);
+
+ topVerTracker.update(Long.MAX_VALUE);
Review Comment:
Discussed in private. I added comment but I didn't create the constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1131487507
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -954,6 +1288,92 @@ public void onError(Throwable e) {
};
}
+ private WatchListener createMetastorageDataNodesListener() {
+ return new WatchListener() {
+ @Override
+ public void onUpdate(WatchEvent evt) {
+ if (!busyLock.enterBusy()) {
+ throw new IgniteInternalException(NODE_STOPPING_ERR, new NodeStoppingException());
+ }
+
+ try {
+ int zoneId = 0;
+
+ Set<String> newDataNodes = null;
+
+ long scaleUpRevision = 0;
+
+ long scaleDownRevision = 0;
+
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
+
+ if (startsWith(e.key(), zoneDataNodesKey().bytes())) {
+ zoneId = extractZoneId(e.key());
+
+ byte[] dataNodesBytes = e.value();
+
+ if (dataNodesBytes != null) {
+ newDataNodes = DistributionZonesUtil.dataNodes(fromBytes(dataNodesBytes));
+ } else {
+ newDataNodes = emptySet();
+ }
+ } else if (startsWith(e.key(), zoneScaleUpChangeTriggerKey().bytes())) {
+ if (e.value() != null) {
+ scaleUpRevision = bytesToLong(e.value());
+ }
+ } else if (startsWith(e.key(), zoneScaleDownChangeTriggerKey().bytes())) {
+ if (e.value() != null) {
+ scaleDownRevision = bytesToLong(e.value());
+ }
+ }
+ }
+
+ synchronized (dataNodesMutex) {
+ DataNodes dataNodesMeta = dataNodes.get(zoneId);
+
+ if (dataNodesMeta != null) {
+ dataNodesMeta.nodes(newDataNodes);
+
+ if (scaleUpRevision > 0) {
+ dataNodesMeta.scaleUpRevision(scaleUpRevision);
+ }
+
+ if (scaleDownRevision > 0) {
+ dataNodesMeta.scaleDownRevision(scaleDownRevision);
+ }
+
+ if (scaleUpRevision > 0) {
+ SortedMap<Long, CompletableFuture<Void>> revisionScaleUpFuts =
+ dataNodes.get(zoneId).revisionScaleUpFutures().headMap(scaleUpRevision, true);
+
+ revisionScaleUpFuts.values().forEach(v -> v.complete(null));
+
+ revisionScaleUpFuts.clear();
+ }
+
+ if (scaleDownRevision > 0) {
+ SortedMap<Long, CompletableFuture<Void>> revisionScaleDownFuts =
+ dataNodes.get(zoneId).revisionScaleDownFutures().headMap(scaleDownRevision, true);
+
+ revisionScaleDownFuts.values().forEach(v -> v.complete(null));
+
+ revisionScaleDownFuts.clear();
+ }
+ }
+ }
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ @Override
+ public void onError(Throwable e) {
+ LOG.warn("Unable to process logical topology event", e);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156780610
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
Review Comment:
Yes, we have the utility method. Fixed.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1161803114
##########
modules/runner/src/integrationTest/java/org/apache/ignite/internal/sql/api/ItSqlSynchronousApiTest.java:
##########
@@ -92,6 +93,8 @@ protected IgniteSql igniteSql() {
return CLUSTER_NODES.get(0).sql();
}
+
+ @Disabled
Review Comment:
I removed it. These tests were fixed by other ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1159664733
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
+
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = getImmediateTimers(zoneId, topVerFut);
+
+ CompletableFuture<Void> topVerScaleUpFut = scaleUpAwaiting(zoneId, timerValuesFut);
+
+ CompletableFuture<Void> topVerScaleDownFut = scaleDownAwaiting(zoneId, timerValuesFut);
+
+ return getDataNodesFuture(zoneId, topVerScaleUpFut, topVerScaleDownFut);
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ return topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ return completedFuture(null);
+ }
+ }
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be scheduled immediate.
+ *
+ * @param zoneId Zone id.
+ * @param topVerFut Future for chaining.
+ * @return Future.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId, CompletableFuture<Void> topVerFut) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
+
+ timerValuesFut.complete(new IgniteBiTuple<>(
+ zoneCfg.dataNodesAutoAdjustScaleUp().value() == 0,
+ zoneCfg.dataNodesAutoAdjustScaleDown().value() == 0
+ ));
+ });
+
+ return timerValuesFut;
+ }
+
+ /**
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than waits for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param timerValuesFut Future for chaining.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> scaleUpAwaiting(
+ int zoneId,
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut
+ ) {
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ timerValuesFut.thenAccept(timerValues -> {
+ boolean immediateScaleUp = timerValues.get1();
+
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp) {
+ if (zoneState.scaleUpRevision() < lastScaleUpRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
Review Comment:
Thanks. I fixed it by using PendingComparableValuesTracker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] alievmirza commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163572626
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,148 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining the data nodes of the specified zone.
+ * If {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} and
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate then it waits that the data nodes
+ * are up-to-date for the passed topology version.
+ *
+ * <p>If the values of auto adjust scale up and auto adjust scale down are zero, then on the cluster topology changes
+ * the data nodes for the zone should be updated immediately. Therefore, this method must return the data nodes which is calculated
+ * based on the topology with passed or greater version. Since the date nodes value is updated asynchronously, this method waits for
+ * the date nodes to be updated with new nodes in the topology if the value of auto adjust scale up is 0. And also waits for
+ * the date nodes to be updated with nodes that have left the topology if the value of auto adjust scale down is 0.
+ * After the zone manager has observed the logical topology change and the data nodes value is updated according to cluster topology,
+ * then this method completes the returned future with the current value of data nodes.
+ *
+ * <p>If the value of auto adjust scale up is greater than zero, then it is not necessary to wait for the data nodes update triggered
+ * by new nodes in cluster topology. Similarly if the value of auto adjust scale down is greater than zero, then it is not necessary to
+ * wait for the data nodes update triggered by new nodes that have left the topology in cluster topology.
+ *
+ * <p>The returned future can be completed with {@link DistributionZoneNotFoundException} and
+ * {@link DistributionZoneWasRemovedException} in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future which will be completed with data nodes for the zoneId or with exception.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager waits for observing passed topology version or greater version
Review Comment:
Waits for DistributionZoneManager waits?
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,148 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining the data nodes of the specified zone.
+ * If {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} and
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate then it waits that the data nodes
+ * are up-to-date for the passed topology version.
+ *
+ * <p>If the values of auto adjust scale up and auto adjust scale down are zero, then on the cluster topology changes
+ * the data nodes for the zone should be updated immediately. Therefore, this method must return the data nodes which is calculated
+ * based on the topology with passed or greater version. Since the date nodes value is updated asynchronously, this method waits for
+ * the date nodes to be updated with new nodes in the topology if the value of auto adjust scale up is 0. And also waits for
+ * the date nodes to be updated with nodes that have left the topology if the value of auto adjust scale down is 0.
+ * After the zone manager has observed the logical topology change and the data nodes value is updated according to cluster topology,
+ * then this method completes the returned future with the current value of data nodes.
+ *
+ * <p>If the value of auto adjust scale up is greater than zero, then it is not necessary to wait for the data nodes update triggered
+ * by new nodes in cluster topology. Similarly if the value of auto adjust scale down is greater than zero, then it is not necessary to
+ * wait for the data nodes update triggered by new nodes that have left the topology in cluster topology.
+ *
+ * <p>The returned future can be completed with {@link DistributionZoneNotFoundException} and
+ * {@link DistributionZoneWasRemovedException} in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future which will be completed with data nodes for the zoneId or with exception.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager waits for observing passed topology version or greater version
+ * in {@link DistributionZoneManager#topologyWatchListener}.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ return inBusyLock(busyLock, () -> topVerTracker.waitFor(topVer));
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be started immediately.
+ *
+ * <p>The returned future can be completed with {@link DistributionZoneNotFoundException}
Review Comment:
It's kinda strange that it completes with `DistributionZoneNotFoundException` in case when the distribution zone was removed, but not with `DistributionZoneWasRemovedException`
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -168,6 +186,19 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** The tracker for the last topology version which was observed by distribution zone manager. */
+ private final PendingComparableValuesTracker<Long> topVerTracker;
+
+ /**
+ * The last meta storage revision on which scale up timer was started.
Review Comment:
formatting
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -168,6 +186,19 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** The tracker for the last topology version which was observed by distribution zone manager. */
+ private final PendingComparableValuesTracker<Long> topVerTracker;
+
+ /**
+ * The last meta storage revision on which scale up timer was started.
+ */
+ private volatile long lastScaleUpRevision;
+
+ /**
+ * The last meta storage revision on which scale down timer was started.
Review Comment:
formatting
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,148 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining the data nodes of the specified zone.
+ * If {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} and
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate then it waits that the data nodes
+ * are up-to-date for the passed topology version.
+ *
+ * <p>If the values of auto adjust scale up and auto adjust scale down are zero, then on the cluster topology changes
+ * the data nodes for the zone should be updated immediately. Therefore, this method must return the data nodes which is calculated
+ * based on the topology with passed or greater version. Since the date nodes value is updated asynchronously, this method waits for
+ * the date nodes to be updated with new nodes in the topology if the value of auto adjust scale up is 0. And also waits for
Review Comment:
comma before `and`
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,148 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining the data nodes of the specified zone.
+ * If {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} and
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate then it waits that the data nodes
+ * are up-to-date for the passed topology version.
+ *
+ * <p>If the values of auto adjust scale up and auto adjust scale down are zero, then on the cluster topology changes
+ * the data nodes for the zone should be updated immediately. Therefore, this method must return the data nodes which is calculated
+ * based on the topology with passed or greater version. Since the date nodes value is updated asynchronously, this method waits for
+ * the date nodes to be updated with new nodes in the topology if the value of auto adjust scale up is 0. And also waits for
+ * the date nodes to be updated with nodes that have left the topology if the value of auto adjust scale down is 0.
+ * After the zone manager has observed the logical topology change and the data nodes value is updated according to cluster topology,
+ * then this method completes the returned future with the current value of data nodes.
+ *
+ * <p>If the value of auto adjust scale up is greater than zero, then it is not necessary to wait for the data nodes update triggered
+ * by new nodes in cluster topology. Similarly if the value of auto adjust scale down is greater than zero, then it is not necessary to
+ * wait for the data nodes update triggered by new nodes that have left the topology in cluster topology.
+ *
+ * <p>The returned future can be completed with {@link DistributionZoneNotFoundException} and
+ * {@link DistributionZoneWasRemovedException} in case when the distribution zone was removed during method execution.
Review Comment:
We have two possible exceptions in case when the distribution zone was removed during method execution. How is it possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1135743095
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -209,17 +253,21 @@ public DistributionZoneManager(
this.logicalTopologyService = logicalTopologyService;
this.vaultMgr = vaultMgr;
- this.watchListener = createMetastorageListener();
+ this.topologyWatchListener = createMetastorageTopologyListener();
+
+ this.dataNodesWatchListener = createMetastorageDataNodesListener();
zonesState = new ConcurrentHashMap<>();
- logicalTopology = Collections.emptySet();
+ logicalTopology = emptySet();
executor = new ScheduledThreadPoolExecutor(
Math.min(Runtime.getRuntime().availableProcessors() * 3, 20),
new NamedThreadFactory(NamedThreadFactory.threadPrefix(nodeName, DISTRIBUTION_ZONE_MANAGER_POOL_NAME), LOG),
new ThreadPoolExecutor.DiscardPolicy()
);
+
+ dataNodes.put(DEFAULT_ZONE_ID, new DataNodes());
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163049320
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,133 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ * The returned future can be completed with {@link DistributionZoneNotFoundException} and {@link DistributionZoneWasRemovedException}
+ * in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
Review Comment:
Fixed.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,133 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ * The returned future can be completed with {@link DistributionZoneNotFoundException} and {@link DistributionZoneWasRemovedException}
+ * in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ return inBusyLock(busyLock, () -> topVerTracker.waitFor(topVer));
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be started immediate.
Review Comment:
Thanks. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1164134875
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,148 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining the data nodes of the specified zone.
+ * If {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} and
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate then it waits that the data nodes
+ * are up-to-date for the passed topology version.
+ *
+ * <p>If the values of auto adjust scale up and auto adjust scale down are zero, then on the cluster topology changes
+ * the data nodes for the zone should be updated immediately. Therefore, this method must return the data nodes which is calculated
+ * based on the topology with passed or greater version. Since the date nodes value is updated asynchronously, this method waits for
+ * the date nodes to be updated with new nodes in the topology if the value of auto adjust scale up is 0. And also waits for
+ * the date nodes to be updated with nodes that have left the topology if the value of auto adjust scale down is 0.
+ * After the zone manager has observed the logical topology change and the data nodes value is updated according to cluster topology,
+ * then this method completes the returned future with the current value of data nodes.
+ *
+ * <p>If the value of auto adjust scale up is greater than zero, then it is not necessary to wait for the data nodes update triggered
+ * by new nodes in cluster topology. Similarly if the value of auto adjust scale down is greater than zero, then it is not necessary to
+ * wait for the data nodes update triggered by new nodes that have left the topology in cluster topology.
+ *
+ * <p>The returned future can be completed with {@link DistributionZoneNotFoundException} and
+ * {@link DistributionZoneWasRemovedException} in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future which will be completed with data nodes for the zoneId or with exception.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager waits for observing passed topology version or greater version
+ * in {@link DistributionZoneManager#topologyWatchListener}.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ return inBusyLock(busyLock, () -> topVerTracker.waitFor(topVer));
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be started immediately.
+ *
+ * <p>The returned future can be completed with {@link DistributionZoneNotFoundException}
Review Comment:
I've updated the java doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1132308183
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when dataNodesAutoAdjustScaleUp
+ * or dataNodesAutoAdjustScaleDown equals to 0.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
Review Comment:
1) testScaleUpScaleDownWhileAwaitingDataNodes
2) testRemoveZoneWhileAwaitingDataNodes
3) Actually this invariant is tested in testWithOutAwaiting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156779272
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
Review Comment:
Fixed. I use single 'long lastScaleUpRevision' and 'long lastScaleDownRevision' now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156781982
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
Review Comment:
Actually it is not fixed. Seems it is the last issue of my pr :) We will discuss naming in the team.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1157493102
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -937,24 +1269,65 @@ public CompletableFuture<Void> onUpdate(WatchEvent evt) {
}
try {
- assert evt.single() : "Expected an event with one entry but was an event with several entries with keys: "
+ assert evt.entryEvents().size() == 2 :
+ "Expected an event with logical topology and logical topology version entries but was events with keys: "
+ evt.entryEvents().stream().map(entry -> entry.newEntry() == null ? "null" : entry.newEntry().key())
.collect(toList());
- Entry newEntry = evt.entryEvent().newEntry();
+ long topVer = 0;
+
+ byte[] newLogicalTopologyBytes = null;
- long revision = newEntry.revision();
+ Set<String> newLogicalTopology = null;
- byte[] newLogicalTopologyBytes = newEntry.value();
+ long revision = 0;
- Set<String> newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
+
+ if (Arrays.equals(e.key(), zonesLogicalTopologyVersionKey().bytes())) {
+ topVer = bytesToLong(e.value());
+
+ revision = e.revision();
+ } else if (Arrays.equals(e.key(), zonesLogicalTopologyKey().bytes())) {
+ newLogicalTopologyBytes = e.value();
+
+ newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ }
+ }
+
+ assert newLogicalTopology != null;
+ assert revision > 0;
+
+ Set<String> newLogicalTopology0 = newLogicalTopology;
Set<String> removedNodes =
- logicalTopology.stream().filter(node -> !newLogicalTopology.contains(node)).collect(toSet());
+ logicalTopology.stream().filter(node -> !newLogicalTopology0.contains(node)).collect(toSet());
Set<String> addedNodes =
newLogicalTopology.stream().filter(node -> !logicalTopology.contains(node)).collect(toSet());
+ synchronized (dataNodesMutex) {
+ lastTopVer = topVer;
+
+ //Associates topology version and scale up meta storage revision.
+ if (!addedNodes.isEmpty()) {
Review Comment:
- Seems that lastScaleUp/DownRevision update is a bit misplaced. We already have `!added/remvoedNodes.isEmpty()` checks right in place where we schedule corresponding timers. So that I'd rather move `lastScaleUpRevision = revision;` and `lastScaleDownRevision = revision;` there.
- I'd also move `lastTopVer = topVer` along with `topVerFuts.values().forEach(v -> v.complete(null));` after scheduleTimers calls. Thus we will have HB relation between setting lastScaleUp/DownRevision and lastTopVer update, so that it'll be possible to update lastScaleUp/DownRevision without synchronized (dataNodesMutex). Volatile will be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1160499180
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -132,6 +143,10 @@ public class DistributionZoneManager implements IgniteComponent {
/** Default number of zone partitions. */
public static final int DEFAULT_PARTITION_COUNT = 25;
+ /** Value for the distribution zones' timers which means that data nodes changing for distribution zone will be
Review Comment:
Incorrect javadoc format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] alievmirza commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1127397612
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ boolean immediateScaleUp = immediateScaleUp0;
+ boolean immediateScaleDown = immediateScaleDown0;
+
+ if (!immediateScaleUp && !immediateScaleDown) {
+
+ DataNodes dataNodes = this.dataNodes.get(zoneId);
+
+ Set<String> nodes;
+
+ if (dataNodes == null) {
+ nodes = emptySet();
+ } else {
+ nodes = this.dataNodes.get(zoneId).nodes();
+ }
+
+ return completedFuture(nodes);
+ }
+
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.get(topVer);
+
+ if (topVerFut == null) {
+ topVerFut = new CompletableFuture<>();
+
+ topVerFutures.put(topVer, topVerFut);
+ }
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ topVerFut.thenAcceptAsync(ignored -> {
Review Comment:
Using async without setting an explicit executor is a bad idea
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZonesUtil.java:
##########
@@ -136,6 +171,10 @@ static ByteArray zonesLogicalTopologyVersionKey() {
return DISTRIBUTION_ZONES_LOGICAL_TOPOLOGY_VERSION_KEY;
}
+ static ByteArray zonesDataNodesPrefix() {
Review Comment:
javadoc
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when dataNodesAutoAdjustScaleUp
+ * or dataNodesAutoAdjustScaleDown equals to 0.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private DistributionZonesConfiguration zonesConfiguration;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ @BeforeEach
+ void setUp() throws Exception {
+ VaultManager vaultManager = mock(VaultManager.class);
+
+ when(vaultManager.get(any())).thenReturn(completedFuture(null));
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ metaStorageManager = mock(MetaStorageManager.class);
+
+ doAnswer(invocation -> {
+ ByteArray key = invocation.getArgument(0);
+
+ WatchListener watchListener = invocation.getArgument(1);
+
+ if (Arrays.equals(key.bytes(), zoneLogicalTopologyPrefix().bytes())) {
+ topologyWatchListener = watchListener;
+ } else if (Arrays.equals(key.bytes(), zonesDataNodesPrefix().bytes())) {
+ dataNodesWatchListener = watchListener;
+ }
+
+ return null;
+ }).when(metaStorageManager).registerPrefixWatch(any(), any());
+
+ when(metaStorageManager.invoke(any())).thenReturn(completedFuture(StatementResultImpl.builder().result(new byte[] {0}).build()));
+
+ vaultManager.start();
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ LogicalTopologyServiceImpl logicalTopologyService = new LogicalTopologyServiceImpl(logicalTopology, cmgManager);
+
+ TablesConfiguration tablesConfiguration = mock(TablesConfiguration.class);
+
+ clusterCfgMgr = new ConfigurationManager(
+ List.of(DistributionZonesConfiguration.KEY),
+ Set.of(),
+ new TestConfigurationStorage(DISTRIBUTED),
+ List.of(),
+ List.of()
+ );
+
+ clusterCfgMgr.start();
+
+ zonesConfiguration = clusterCfgMgr.configurationRegistry()
+ .getConfiguration(DistributionZonesConfiguration.KEY);
+
+ distributionZoneManager = new DistributionZoneManager(
+ zonesConfiguration,
+ tablesConfiguration,
+ metaStorageManager,
+ logicalTopologyService,
+ vaultManager,
+ "node"
+ );
+
+ mockCmgLocalNodes();
+
+ distributionZoneManager.start();
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(0).dataNodesAutoAdjustScaleDown(0).build())
+ .get(3, TimeUnit.SECONDS);
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ distributionZoneManager.stop();
+ clusterCfgMgr.stop();
+ clusterStateStorage.stop();
+ }
+
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUpAndScaleDown() throws Exception {
+ LOG.info("Topology with added and removed nodes.");
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ Set<String> dataNodes = Set.of("node0", "node2");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ DistributionZoneManager.DataNodes dataNodesMeta = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ assertTrue(waitForCondition(() -> dataNodesMeta.revisionScaleUpFutures().size() == 1, 3_000));
+
+ assertTrue(testData.dataNodesMeta.revisionScaleDownFutures().isEmpty());
+
+ CompletableFuture<Void> revisionUpFut = testData.dataNodesMeta.revisionScaleUpFutures().get((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionUpFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, true, testData.dataNodesRevision2, testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(() -> revisionUpFut.isDone(), 3000));
+
+ assertTrue(waitForCondition(() -> dataNodesMeta.revisionScaleDownFutures().size() == 1, 3_000));
+
+ CompletableFuture<Void> revisionDownFut = testData.dataNodesMeta.revisionScaleDownFutures()
+ .get((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionDownFut.isDone());
+
+ assertFalse(testData.dataNodesUpFut3.isDone());
+ assertFalse(testData.dataNodesDownFut3.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, false, testData.dataNodesRevision2, testData.dataNodesRevision2 + 2);
+
+ assertTrue(waitForCondition(() -> revisionDownFut.isDone(), 3000));
+
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, TimeUnit.SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, TimeUnit.SECONDS));
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+ }
+
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUp() throws Exception {
+ LOG.info("Topology with added nodes.");
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ Set<String> dataNodes = Set.of("node0", "node1", "node2");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ DistributionZoneManager.DataNodes dataNodesMeta = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ assertTrue(waitForCondition(() -> dataNodesMeta.revisionScaleUpFutures().size() == 1, 3_000));
+
+ assertTrue(testData.dataNodesMeta.revisionScaleDownFutures().isEmpty());
+
+ CompletableFuture<Void> revisionUpFut = testData.dataNodesMeta.revisionScaleUpFutures().get((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionUpFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, true, testData.dataNodesRevision2, testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(() -> revisionUpFut.isDone(), 3000));
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, TimeUnit.SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, TimeUnit.SECONDS));
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+ }
+
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleDown() throws Exception {
+ LOG.info("Topology with removed nodes.");
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ Set<String> dataNodes = Set.of("node0");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ assertTrue(waitForCondition(() -> {
+ DistributionZoneManager.DataNodes dataNodesMeta0 = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ if (dataNodesMeta0 == null) {
+ return false;
+ }
+
+ return dataNodesMeta0.revisionScaleDownFutures().size() == 1;
+ },
+ 3_000));
+
+ assertTrue(testData.dataNodesMeta.revisionScaleUpFutures().isEmpty());
+
+ CompletableFuture<Void> revisionDownFut = testData.dataNodesMeta.revisionScaleDownFutures()
+ .get((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionDownFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, false, testData.dataNodesRevision2, testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(() -> revisionDownFut.isDone(), 3000));
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, TimeUnit.SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, TimeUnit.SECONDS));
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+ }
+
+ private static class TestSeveralScaleUpAndSeveralScaleDownDataObject {
+ private long topVer2;
+ private long dataNodesRevision2;
+ private CompletableFuture<Void> topVerUpFut2;
+ private CompletableFuture<Void> topVerDownFut2;
+ private DistributionZoneManager.DataNodes dataNodesMeta;
+ private CompletableFuture<Set<String>> dataNodesUpFut3;
+ private CompletableFuture<Set<String>> dataNodesDownFut3;
+
+ public TestSeveralScaleUpAndSeveralScaleDownDataObject(
+ long topVer2,
+ long dataNodesRevision2,
+ CompletableFuture<Void> topVerUpFut2,
+ CompletableFuture<Void> topVerDownFut2,
+ DistributionZoneManager.DataNodes dataNodesMeta,
+ CompletableFuture<Set<String>> dataNodesUpFut3,
+ CompletableFuture<Set<String>> dataNodesDownFut3) {
+ this.topVer2 = topVer2;
+ this.dataNodesRevision2 = dataNodesRevision2;
+ this.topVerUpFut2 = topVerUpFut2;
+ this.topVerDownFut2 = topVerDownFut2;
+ this.dataNodesMeta = dataNodesMeta;
+ this.dataNodesUpFut3 = dataNodesUpFut3;
+ this.dataNodesDownFut3 = dataNodesDownFut3;
+ }
+ }
+
+ private TestSeveralScaleUpAndSeveralScaleDownDataObject testSeveralScaleUpAndSeveralScaleDownGeneral() throws Exception {
+ LOG.info("Topology with added nodes.");
+
+ CompletableFuture<Set<String>> dataNodesUpFut0 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut1 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut2 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 2);
+ CompletableFuture<Set<String>> dataNodesUpFut3 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 11);
+
+ assertTrue(waitForCondition(() -> distributionZoneManager.topVerFutures().size() == 3,
+ 3_000));
+
+ int topVer0 = 2;
+
+ int dataNodesRevision0 = 2;
+
+ CompletableFuture<Void> topVerUpFut0 = distributionZoneManager.topVerFutures().get(1L);
+ CompletableFuture<Void> topVerUpFut1 = distributionZoneManager.topVerFutures().get(2L);
+ CompletableFuture<Void> topVerUpFut2 = distributionZoneManager.topVerFutures().get(11L);
+
+ assertFalse(topVerUpFut0.isDone());
+ assertFalse(topVerUpFut1.isDone());
+ assertFalse(topVerUpFut2.isDone());
+
+ Set<String> threeNodes = Set.of("node0", "node1", "node2");
+
+ topologyWatchListenerOnUpdate(threeNodes, topVer0, dataNodesRevision0);
+
+ assertTrue(waitForCondition(() -> topVerUpFut0.isDone(), 3_000));
+ assertTrue(waitForCondition(() -> topVerUpFut1.isDone(), 3_000));
+ assertFalse(topVerUpFut2.isDone());
+
+ assertTrue(waitForCondition(() -> {
+ DistributionZoneManager.DataNodes dataNodesMeta0 = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ if (dataNodesMeta0 == null) {
+ return false;
+ }
+
+ return dataNodesMeta0.revisionScaleUpFutures().size() == 1;
+ },
+ 3_000));
+
+ DistributionZoneManager.DataNodes dataNodesMeta = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ CompletableFuture<Void> revision2Fut = dataNodesMeta.revisionScaleUpFutures().get((long) dataNodesRevision0);
+
+ assertFalse(revision2Fut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, threeNodes, true, dataNodesRevision0, dataNodesRevision0 + 1);
+
+ assertTrue(waitForCondition(() -> revision2Fut.isDone(),
+ 3_000));
+
+ assertEquals(threeNodes, dataNodesUpFut0.get());
+ assertEquals(threeNodes, dataNodesUpFut1.get());
+ assertEquals(threeNodes, dataNodesUpFut2.get());
+ assertFalse(dataNodesUpFut3.isDone());
+
+ assertTrue(distributionZoneManager.topVerFutures().size() == 1);
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+
+
+ LOG.info("Topology with removed nodes.");
+
+ CompletableFuture<Set<String>> dataNodesDownFut0 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut1 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut2 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 5);
+ CompletableFuture<Set<String>> dataNodesDownFut3 = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 6);
+
+ assertTrue(waitForCondition(() -> distributionZoneManager.topVerFutures().size() == 4,
+ 3_000));
+
+ CompletableFuture<Void> topVerDownFut0 = distributionZoneManager.topVerFutures().get(4L);
+ CompletableFuture<Void> topVerDownFut1 = distributionZoneManager.topVerFutures().get(5L);
+ CompletableFuture<Void> topVerDownFut2 = distributionZoneManager.topVerFutures().get(6L);
+
+ assertFalse(topVerDownFut0.isDone());
+ assertFalse(topVerDownFut1.isDone());
+ assertFalse(topVerDownFut2.isDone());
+
+ int topVer1 = 5;
+
+ int dataNodesRevision1 = dataNodesRevision0 + 2;
+
+ Set<String> twoNodes = Set.of("node0", "node1");
+
+ topologyWatchListenerOnUpdate(twoNodes, topVer1, dataNodesRevision1);
+
+ assertTrue(waitForCondition(() -> topVerDownFut0.isDone(), 3_000));
+ assertTrue(waitForCondition(() -> topVerDownFut1.isDone(), 3_000));
+ assertFalse(waitForCondition(() -> topVerDownFut2.isDone(), 3_000));
+
+ assertTrue(waitForCondition(() -> dataNodesMeta.revisionScaleDownFutures().size() == 1, 3_000));
+
+ CompletableFuture<Void> revision5Fut = dataNodesMeta.revisionScaleDownFutures().get((long) dataNodesRevision1);
+
+ assertFalse(revision5Fut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, twoNodes, false, dataNodesRevision1, dataNodesRevision1 + 1);
+
+ assertTrue(waitForCondition(() -> revision5Fut.isDone(), 3_000));
+
+ assertEquals(twoNodes, dataNodesDownFut0.get());
+ assertEquals(twoNodes, dataNodesDownFut1.get());
+ assertEquals(twoNodes, dataNodesDownFut2.get());
+ assertFalse(dataNodesDownFut3.isDone());
+
+ assertEquals(2, distributionZoneManager.topVerFutures().size());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+
+ int topVer2 = 20;
+
+ int dataNodesRevision2 = dataNodesRevision1 + 2;
+
+ return new TestSeveralScaleUpAndSeveralScaleDownDataObject(
+ topVer2,
+ dataNodesRevision2,
+ topVerUpFut2,
+ topVerDownFut2,
+ dataNodesMeta,
+ dataNodesUpFut3,
+ dataNodesDownFut3
+ );
+ }
+
+ @Test
+ void testScaleUpAndThenScaleDown() throws Exception {
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 5);
+
+ AtomicReference<CompletableFuture<Void>> topVerFut = new AtomicReference<>();
+
+ assertTrue(waitForCondition(() -> {
+ topVerFut.set(distributionZoneManager.topVerFutures().get(5L));
+
+ return topVerFut.get() != null;
+ },
+ 3_000));
+
+ assertFalse(topVerFut.get().isDone());
+
+ topologyWatchListenerOnUpdate(Set.of("node0", "node1"), 100, 10);
+
+ assertTrue(waitForCondition(() -> topVerFut.get().isDone(),
+ 3_000));
+
+ AtomicReference<CompletableFuture<Void>> revisionFut = new AtomicReference<>();
+
+ assertTrue(waitForCondition(() -> {
+ DistributionZoneManager.DataNodes dataNodes = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ if (dataNodes == null) {
+ return false;
+ }
+
+ revisionFut.set(dataNodes.revisionScaleUpFutures().get(10L));
+
+ return revisionFut.get() != null;
+ },
+ 3_000));
+
+ assertFalse(revisionFut.get().isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, Set.of("node0", "node1"), true, 10, 11);
+
+ assertTrue(waitForCondition(() -> revisionFut.get().isDone(),
+ 3_000));
+
+ assertEquals(Set.of("node0", "node1"), dataNodesFut.get());
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+
+ dataNodesFut = distributionZoneManager.getDataNodes(DEFAULT_ZONE_ID, 106);
+
+ topVerFut.set(null);
+
+ assertTrue(waitForCondition(() -> {
+ topVerFut.set(distributionZoneManager.topVerFutures().get(106L));
+
+ return topVerFut.get() != null;
+ },
+ 3_000));
+
+ assertFalse(topVerFut.get().isDone());
+
+ topologyWatchListenerOnUpdate(Set.of("node0"), 200, 12);
+
+ assertTrue(waitForCondition(() -> topVerFut.get().isDone(),
+ 3_000));
+
+ revisionFut.set(null);
+
+ assertTrue(waitForCondition(() -> {
+ DistributionZoneManager.DataNodes dataNodes = distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID);
+
+ if (dataNodes == null) {
+ return false;
+ }
+
+ revisionFut.set(dataNodes.revisionScaleDownFutures().get(12L));
+
+ return revisionFut.get() != null;
+ },
+ 3_000));
+
+ assertFalse(revisionFut.get().isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, Set.of("node0"), false, 12, 13);
+
+ assertTrue(waitForCondition(() -> revisionFut.get().isDone(),
+ 3_000));
+
+ assertEquals(Set.of("node0"), dataNodesFut.get());
+
+ assertTrue(distributionZoneManager.topVerFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleUpFutures().isEmpty());
+ assertTrue(distributionZoneManager.dataNodes().get(DEFAULT_ZONE_ID).revisionScaleDownFutures().isEmpty());
+ }
+
+ @Test
+ void testAwaitingScaleUpOnly() throws Exception {
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(Integer.MAX_VALUE).dataNodesAutoAdjustScaleDown(Integer.MAX_VALUE).build())
+ .get(3, TimeUnit.SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(0)
+ .dataNodesAutoAdjustScaleDown(Integer.MAX_VALUE)
+ .build()
+ )
+ .get(3, TimeUnit.SECONDS);
+
+ int zoneId = distributionZoneManager.getZoneId("zone1");
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.getDataNodes(zoneId, 1);
+
+ CompletableFuture<Void> topVerUpFut = distributionZoneManager.topVerFutures().get(1L);
+
+ assertFalse(topVerUpFut.isDone());
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ topologyWatchListenerOnUpdate(nodes0, 1, 1);
+
+ dataNodesWatchListenerOnUpdate(zoneId, nodes0, true, 1, 2);
+
+ dataNodesFut.get(3, TimeUnit.SECONDS);
+
+ dataNodesFut = distributionZoneManager.getDataNodes(zoneId, 2);
+
+ Set<String> nodes1 = Set.of("node0");
+
+ topologyWatchListenerOnUpdate(nodes1, 2, 2);
+
+ dataNodesFut.get(3, TimeUnit.SECONDS);
+ }
+
+ @Test
+ void testAwaitingScaleDownOnly() throws Exception {
Review Comment:
Taking into account the test name, we should check here that other zones do not wait for data nodes.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ boolean immediateScaleUp = immediateScaleUp0;
+ boolean immediateScaleDown = immediateScaleDown0;
+
+ if (!immediateScaleUp && !immediateScaleDown) {
+
+ DataNodes dataNodes = this.dataNodes.get(zoneId);
+
+ Set<String> nodes;
+
+ if (dataNodes == null) {
+ nodes = emptySet();
+ } else {
+ nodes = this.dataNodes.get(zoneId).nodes();
+ }
+
+ return completedFuture(nodes);
+ }
+
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.get(topVer);
+
+ if (topVerFut == null) {
+ topVerFut = new CompletableFuture<>();
+
+ topVerFutures.put(topVer, topVerFut);
Review Comment:
Could we use `putIfAbsent` here instead all of checks for null?
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
Review Comment:
Please, divide this method and extract part when we react on future completion, it will increase readability. Also please add comments right inside the code, currently it's auite hard to understand what's going on
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ boolean immediateScaleUp = immediateScaleUp0;
+ boolean immediateScaleDown = immediateScaleDown0;
+
+ if (!immediateScaleUp && !immediateScaleDown) {
+
+ DataNodes dataNodes = this.dataNodes.get(zoneId);
+
+ Set<String> nodes;
+
+ if (dataNodes == null) {
+ nodes = emptySet();
+ } else {
+ nodes = this.dataNodes.get(zoneId).nodes();
+ }
+
+ return completedFuture(nodes);
+ }
+
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.get(topVer);
+
+ if (topVerFut == null) {
+ topVerFut = new CompletableFuture<>();
+
+ topVerFutures.put(topVer, topVerFut);
+ }
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ topVerFut.thenAcceptAsync(ignored -> {
+ synchronized (dataNodesMutex) {
+ CompletableFuture<Void> topVerScaleUpFut = null;
+
+ if (immediateScaleUp) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null
+ && (dataNodes.get(zoneId) == null || dataNodes.get(zoneId).scaleUpRevision() < scaleUpRevision)) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ dataNodes.get(zoneId).revisionScaleUpFutures().ceilingEntry(scaleUpRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleUpFut = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleUpFut == null) {
+ topVerScaleUpFut = new CompletableFuture<>();
+
+ dataNodes.get(zoneId).revisionScaleUpFutures().put(scaleUpRevision, topVerScaleUpFut);
+ }
+ } else {
+ topVerScaleUpFut = completedFuture(null);
+ }
+ } else {
+ topVerScaleUpFut = completedFuture(null);
+ }
+
+ topVerScaleUpFut.thenAcceptAsync(ignored0 -> {
+ CompletableFuture<Void> topVerScaleDownFut = null;
+
+ synchronized (dataNodesMutex) {
+ if (immediateScaleDown) {
+ Map.Entry<Long, Long> scaleDownRevisionEntry = topVerAndScaleDownRevision.ceilingEntry(topVer);
+
+ Long scaleDownRevision = null;
+
+ if (scaleDownRevisionEntry != null) {
+ scaleDownRevision = scaleDownRevisionEntry.getValue();
+ }
+
+ if (scaleDownRevision != null
+ && (dataNodes.get(zoneId) == null || dataNodes.get(zoneId).scaleDownRevision() < scaleDownRevision)) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ dataNodes.get(zoneId).revisionScaleDownFutures().ceilingEntry(scaleDownRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleDownFut = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleDownFut == null) {
+ topVerScaleDownFut = new CompletableFuture<>();
+
+ dataNodes.get(zoneId).revisionScaleDownFutures.put(scaleDownRevision, topVerScaleDownFut);
+ }
+ } else {
+ topVerScaleDownFut = completedFuture(null);
+ }
+ } else {
+ topVerScaleDownFut = completedFuture(null);
+ }
+ }
+
+ topVerScaleDownFut.thenAcceptAsync(ignored1 -> {
Review Comment:
Why do we need this code? Why we do this only got scale down?
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when dataNodesAutoAdjustScaleUp
+ * or dataNodesAutoAdjustScaleDown equals to 0.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private DistributionZonesConfiguration zonesConfiguration;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ @BeforeEach
+ void setUp() throws Exception {
+ VaultManager vaultManager = mock(VaultManager.class);
+
+ when(vaultManager.get(any())).thenReturn(completedFuture(null));
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ metaStorageManager = mock(MetaStorageManager.class);
+
+ doAnswer(invocation -> {
+ ByteArray key = invocation.getArgument(0);
+
+ WatchListener watchListener = invocation.getArgument(1);
+
+ if (Arrays.equals(key.bytes(), zoneLogicalTopologyPrefix().bytes())) {
+ topologyWatchListener = watchListener;
+ } else if (Arrays.equals(key.bytes(), zonesDataNodesPrefix().bytes())) {
+ dataNodesWatchListener = watchListener;
+ }
+
+ return null;
+ }).when(metaStorageManager).registerPrefixWatch(any(), any());
+
+ when(metaStorageManager.invoke(any())).thenReturn(completedFuture(StatementResultImpl.builder().result(new byte[] {0}).build()));
+
+ vaultManager.start();
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ LogicalTopologyServiceImpl logicalTopologyService = new LogicalTopologyServiceImpl(logicalTopology, cmgManager);
+
+ TablesConfiguration tablesConfiguration = mock(TablesConfiguration.class);
+
+ clusterCfgMgr = new ConfigurationManager(
+ List.of(DistributionZonesConfiguration.KEY),
+ Set.of(),
+ new TestConfigurationStorage(DISTRIBUTED),
+ List.of(),
+ List.of()
+ );
+
+ clusterCfgMgr.start();
+
+ zonesConfiguration = clusterCfgMgr.configurationRegistry()
+ .getConfiguration(DistributionZonesConfiguration.KEY);
+
+ distributionZoneManager = new DistributionZoneManager(
+ zonesConfiguration,
+ tablesConfiguration,
+ metaStorageManager,
+ logicalTopologyService,
+ vaultManager,
+ "node"
+ );
+
+ mockCmgLocalNodes();
+
+ distributionZoneManager.start();
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(0).dataNodesAutoAdjustScaleDown(0).build())
+ .get(3, TimeUnit.SECONDS);
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ distributionZoneManager.stop();
+ clusterCfgMgr.stop();
+ clusterStateStorage.stop();
+ }
+
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUpAndScaleDown() throws Exception {
Review Comment:
This and several tests below test await logic only for default zone, I would test the same logic, but for several non-default zones.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZonesUtil.java:
##########
@@ -266,4 +305,32 @@ public static Map<String, Integer> toDataNodesMap(Set<String> dataNodes) {
return dataNodesMap;
}
+
+ /**
+ * Utility method to check if one byte array starts with a specified sequence
+ * of bytes.
+ *
+ * @param array The array to check.
+ * @param prefix The prefix bytes to test for.
+ * @return {@code true} if the array starts with the bytes from the prefix.
+ */
+ public static boolean startsWith(byte[] array, byte[] prefix) {
Review Comment:
You could extract `org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage#startsWith` to some common place and reuse it. I prefer variant with `Arrays.equals`
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when dataNodesAutoAdjustScaleUp
+ * or dataNodesAutoAdjustScaleDown equals to 0.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
Review Comment:
I would add test scenarios when we have immediate scale up/down, and and in the middle of awaiting logic we change scale up/down to a non-zero value.
Also I would expect the scenario when zone was deleted
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -159,6 +167,36 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** Data nodes modification mutex. */
+ private final Object dataNodesMutex = new Object();
+
+ /** The last topology version wich was observed by distribution zone manager. */
+ private long lastTopVer;
+
+ /**
+ * Contains data nodes and meta info for zones.
+ * Map (zone id -> data nodes).
+ */
+ private final Map<Integer, DataNodes> dataNodes = new ConcurrentHashMap<>();
Review Comment:
If this and maps below are accessed only in synchronised block, do we really need `ConcurrentHashMap` here?
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -183,8 +221,89 @@ public void onTopologyLeap(LogicalTopologySnapshot newTopology) {
*/
private volatile Set<String> logicalTopology;
- /** Watch listener. Needed to unregister it on {@link DistributionZoneManager#stop()}. */
- private final WatchListener watchListener;
+ /** Watch listener for logical topology keys. */
+ private final WatchListener topologyWatchListener;
+
+ /** Watch listener for data nodes keys. */
+ private final WatchListener dataNodesWatchListener;
+
+ /**
+ * Contains data nodes and meta info.
+ */
+ static class DataNodes {
+ /** Data nodes. */
+ private Set<String> nodes;
+
+ /** Scale up metastorage revision of current nodes value. */
+ private long scaleUpRevision;
+
+ /** Scale down metastorage revision of current nodes value. */
+ private long scaleDownRevision;
+
+ /**
+ * The map contains futures which are completed when zone manager observe data nodes bound to appropriate scale up revision.
+ * Map (revision -> future).
+ */
+ private final NavigableMap<Long, CompletableFuture<Void>> revisionScaleUpFutures = new ConcurrentSkipListMap();
+
+ /**
+ * The map contains futures which are completed when zone manager observe data nodes bound to appropriate scale down revision.
+ * Map (revision -> future).
+ */
+ private final NavigableMap<Long, CompletableFuture<Void>> revisionScaleDownFutures = new ConcurrentSkipListMap();
+
+ DataNodes() {
+ nodes = emptySet();
+ }
+
+ DataNodes(Set<String> nodes, long scaleUpRevision, long scaleDownRevision) {
+ this.nodes = nodes;
+ this.scaleUpRevision = scaleUpRevision;
+ this.scaleDownRevision = scaleDownRevision;
+ }
+
+ Set<String> nodes() {
+ return nodes;
+ }
+
+ void nodes(Set<String> nodes) {
+ this.nodes = nodes;
+ }
+
+ long scaleUpRevision() {
+ return scaleUpRevision;
+ }
+
+ void scaleUpRevision(long scaleUpRevision) {
+ this.scaleUpRevision = scaleUpRevision;
+ }
+
+ long scaleDownRevision() {
+ return scaleDownRevision;
+ }
+
+ void scaleDownRevision(long scaleDownRevision) {
+ this.scaleDownRevision = scaleDownRevision;
+ }
+
+ NavigableMap<Long, CompletableFuture<Void>> revisionScaleUpFutures() {
+ return revisionScaleUpFutures;
+ }
+
+ NavigableMap<Long, CompletableFuture<Void>> revisionScaleDownFutures() {
+ return revisionScaleDownFutures;
+ }
+ }
+
+ @TestOnly
+ Map<Integer, DataNodes> dataNodes() {
Review Comment:
This two `TestOnly` methods could be invoked concurrently
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
Review Comment:
We need links to `dataNodesAutoAdjustScaleUp`, `dataNodesAutoAdjustScaleDown`. User must be able to understand what these concepts mean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1159665006
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
+
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = getImmediateTimers(zoneId, topVerFut);
+
+ CompletableFuture<Void> topVerScaleUpFut = scaleUpAwaiting(zoneId, timerValuesFut);
+
+ CompletableFuture<Void> topVerScaleDownFut = scaleDownAwaiting(zoneId, timerValuesFut);
+
+ return getDataNodesFuture(zoneId, topVerScaleUpFut, topVerScaleDownFut);
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ synchronized (dataNodesMutex) {
Review Comment:
Fixed. I removed the dataNodesMutex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1160515027
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -168,6 +183,19 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** The tracker for last topology version which was observed by distribution zone manager. */
+ private final PendingComparableValuesTracker<Long> topVerTracker;
+
+ /**
+ * The last meta storage revision on which scale up timer was started.
+ */
+ private long lastScaleUpRevision;
Review Comment:
AFAIK we need volatile here(and also for lastScaleDownRevision), because otherwise you might miss lastScale<>Revision updates triggered from within onUpdateScaleUp()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1160635718
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -168,6 +183,19 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** The tracker for last topology version which was observed by distribution zone manager. */
+ private final PendingComparableValuesTracker<Long> topVerTracker;
+
+ /**
+ * The last meta storage revision on which scale up timer was started.
+ */
+ private long lastScaleUpRevision;
Review Comment:
Fixed
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +540,111 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ )
+ .thenCompose(ignored -> getDataNodesFuture(zoneId));
Review Comment:
Fixed
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -132,6 +143,10 @@ public class DistributionZoneManager implements IgniteComponent {
/** Default number of zone partitions. */
public static final int DEFAULT_PARTITION_COUNT = 25;
+ /** Value for the distribution zones' timers which means that data nodes changing for distribution zone will be
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1135744172
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +550,183 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown}
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ dataNodes.putIfAbsent(zoneId, new DataNodes());
+
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ boolean immediateScaleUp = immediateScaleUp0;
+ boolean immediateScaleDown = immediateScaleDown0;
+
+ if (!immediateScaleUp && !immediateScaleDown) {
+ synchronized (dataNodesMutex) {
+
+ DataNodes dataNodes = this.dataNodes.get(zoneId);
+
+ Set<String> nodes;
+
+ if (dataNodes == null) {
+ nodes = emptySet();
+ } else {
+ nodes = this.dataNodes.get(zoneId).nodes();
+ }
+
+
+ return completedFuture(nodes);
+ }
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null && dataNodes.get(zoneId).scaleUpRevision() < scaleUpRevision) {
Review Comment:
Thanks. Fixed it here and other places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156779272
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1157494481
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
+
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = getImmediateTimers(zoneId, topVerFut);
+
+ CompletableFuture<Void> topVerScaleUpFut = scaleUpAwaiting(zoneId, timerValuesFut);
+
+ CompletableFuture<Void> topVerScaleDownFut = scaleDownAwaiting(zoneId, timerValuesFut);
+
+ return getDataNodesFuture(zoneId, topVerScaleUpFut, topVerScaleDownFut);
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ return topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ return completedFuture(null);
+ }
+ }
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be scheduled immediate.
+ *
+ * @param zoneId Zone id.
+ * @param topVerFut Future for chaining.
+ * @return Future.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId, CompletableFuture<Void> topVerFut) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
+
+ timerValuesFut.complete(new IgniteBiTuple<>(
+ zoneCfg.dataNodesAutoAdjustScaleUp().value() == 0,
Review Comment:
0 to constant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163050578
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -945,24 +1147,57 @@ public CompletableFuture<Void> onUpdate(WatchEvent evt) {
}
try {
- assert evt.single() : "Expected an event with one entry but was an event with several entries with keys: "
+ assert evt.entryEvents().size() == 2 :
+ "Expected an event with logical topology and logical topology version entries but was events with keys: "
+ evt.entryEvents().stream().map(entry -> entry.newEntry() == null ? "null" : entry.newEntry().key())
.collect(toList());
- Entry newEntry = evt.entryEvent().newEntry();
+ long topVer = 0;
+
+ byte[] newLogicalTopologyBytes = null;
+
+ Set<String> newLogicalTopology = null;
+
+ long revision = 0;
+
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
- long revision = newEntry.revision();
+ if (Arrays.equals(e.key(), zonesLogicalTopologyVersionKey().bytes())) {
+ topVer = bytesToLong(e.value());
- byte[] newLogicalTopologyBytes = newEntry.value();
+ revision = e.revision();
+ } else if (Arrays.equals(e.key(), zonesLogicalTopologyKey().bytes())) {
+ newLogicalTopologyBytes = e.value();
- Set<String> newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ }
+ }
+
+ assert newLogicalTopology != null;
+ assert revision > 0;
+
+ Set<String> newLogicalTopology0 = newLogicalTopology;
Set<String> removedNodes =
- logicalTopology.stream().filter(node -> !newLogicalTopology.contains(node)).collect(toSet());
+ logicalTopology.stream().filter(node -> !newLogicalTopology0.contains(node)).collect(toSet());
Set<String> addedNodes =
newLogicalTopology.stream().filter(node -> !logicalTopology.contains(node)).collect(toSet());
+ if (!addedNodes.isEmpty()) {
+ lastScaleUpRevision = revision;
+ }
+
+ if (!removedNodes.isEmpty()) {
+ lastScaleDownRevision = revision;
+ }
+
+ //The topology version must be updated after the lastScaleUpRevision and lastScaleDownRevision are updated.
+ //This is necessary in order to when topology version waiters will be notified that topology version is updated
Review Comment:
I've rephrased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163727884
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -168,6 +186,19 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** The tracker for the last topology version which was observed by distribution zone manager. */
+ private final PendingComparableValuesTracker<Long> topVerTracker;
+
+ /**
+ * The last meta storage revision on which scale up timer was started.
Review Comment:
Fixed.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -168,6 +186,19 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** The tracker for the last topology version which was observed by distribution zone manager. */
+ private final PendingComparableValuesTracker<Long> topVerTracker;
+
+ /**
+ * The last meta storage revision on which scale up timer was started.
+ */
+ private volatile long lastScaleUpRevision;
+
+ /**
+ * The last meta storage revision on which scale down timer was started.
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1131486759
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZonesUtil.java:
##########
@@ -266,4 +305,32 @@ public static Map<String, Integer> toDataNodesMap(Set<String> dataNodes) {
return dataNodesMap;
}
+
+ /**
+ * Utility method to check if one byte array starts with a specified sequence
+ * of bytes.
+ *
+ * @param array The array to check.
+ * @param prefix The prefix bytes to test for.
+ * @return {@code true} if the array starts with the bytes from the prefix.
+ */
+ public static boolean startsWith(byte[] array, byte[] prefix) {
Review Comment:
Thanks, fixed.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -480,6 +625,152 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone. If the dataNodesAutoAdjustScaleUp or dataNodesAutoAdjustScaleDown
+ * equals to 0 then returned completable future will be completed when data nodes will be saved to the meta storage.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> getDataNodes(int zoneId, long topVer) {
Review Comment:
I rework this code and add comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1159663476
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -937,24 +1269,65 @@ public CompletableFuture<Void> onUpdate(WatchEvent evt) {
}
try {
- assert evt.single() : "Expected an event with one entry but was an event with several entries with keys: "
+ assert evt.entryEvents().size() == 2 :
+ "Expected an event with logical topology and logical topology version entries but was events with keys: "
+ evt.entryEvents().stream().map(entry -> entry.newEntry() == null ? "null" : entry.newEntry().key())
.collect(toList());
- Entry newEntry = evt.entryEvent().newEntry();
+ long topVer = 0;
+
+ byte[] newLogicalTopologyBytes = null;
- long revision = newEntry.revision();
+ Set<String> newLogicalTopology = null;
- byte[] newLogicalTopologyBytes = newEntry.value();
+ long revision = 0;
- Set<String> newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
+
+ if (Arrays.equals(e.key(), zonesLogicalTopologyVersionKey().bytes())) {
+ topVer = bytesToLong(e.value());
+
+ revision = e.revision();
+ } else if (Arrays.equals(e.key(), zonesLogicalTopologyKey().bytes())) {
+ newLogicalTopologyBytes = e.value();
+
+ newLogicalTopology = fromBytes(newLogicalTopologyBytes);
+ }
+ }
+
+ assert newLogicalTopology != null;
+ assert revision > 0;
+
+ Set<String> newLogicalTopology0 = newLogicalTopology;
Set<String> removedNodes =
- logicalTopology.stream().filter(node -> !newLogicalTopology.contains(node)).collect(toSet());
+ logicalTopology.stream().filter(node -> !newLogicalTopology0.contains(node)).collect(toSet());
Set<String> addedNodes =
newLogicalTopology.stream().filter(node -> !logicalTopology.contains(node)).collect(toSet());
+ synchronized (dataNodesMutex) {
+ lastTopVer = topVer;
+
+ //Associates topology version and scale up meta storage revision.
+ if (!addedNodes.isEmpty()) {
Review Comment:
Thanks. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163051005
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,910 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.DistributionZoneManager.ZoneState;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TableChange;
+import org.apache.ignite.internal.schema.configuration.TableConfiguration;
+import org.apache.ignite.internal.schema.configuration.TableView;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultEntry;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private DistributionZonesConfiguration zonesConfiguration;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ @BeforeEach
+ void setUp() {
+ vaultManager = mock(VaultManager.class);
+
+ when(vaultManager.get(zonesLogicalTopologyKey())).thenReturn(completedFuture(new VaultEntry(zonesLogicalTopologyKey(), null)));
+
+ when(vaultManager.get(zonesLogicalTopologyVersionKey()))
+ .thenReturn(completedFuture(new VaultEntry(zonesLogicalTopologyVersionKey(), longToBytes(0))));
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ metaStorageManager = mock(MetaStorageManager.class);
+
+ doAnswer(invocation -> {
+ ByteArray key = invocation.getArgument(0);
+
+ WatchListener watchListener = invocation.getArgument(1);
+
+ if (Arrays.equals(key.bytes(), zoneLogicalTopologyPrefix().bytes())) {
+ topologyWatchListener = watchListener;
+ } else if (Arrays.equals(key.bytes(), zonesDataNodesPrefix().bytes())) {
+ dataNodesWatchListener = watchListener;
+ }
+
+ return null;
+ }).when(metaStorageManager).registerPrefixWatch(any(), any());
+
+ when(metaStorageManager.invoke(any()))
+ .thenReturn(completedFuture(StatementResultImpl.builder().result(new byte[] {0}).build()));
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ LogicalTopologyServiceImpl logicalTopologyService = new LogicalTopologyServiceImpl(logicalTopology, cmgManager);
+
+ TablesConfiguration tablesConfiguration = mock(TablesConfiguration.class);
+
+ NamedConfigurationTree<TableConfiguration, TableView, TableChange> tables = mock(NamedConfigurationTree.class);
+
+ when(tablesConfiguration.tables()).thenReturn(tables);
+
+ NamedListView<TableView> tablesValue = mock(NamedListView.class);
+
+ when(tables.value()).thenReturn(tablesValue);
+
+ when(tablesValue.size()).thenReturn(0);
+
+ clusterCfgMgr = new ConfigurationManager(
+ List.of(DistributionZonesConfiguration.KEY),
+ Set.of(),
+ new TestConfigurationStorage(DISTRIBUTED),
+ List.of(),
+ List.of()
+ );
+
+ clusterCfgMgr.start();
+
+ zonesConfiguration = clusterCfgMgr.configurationRegistry()
+ .getConfiguration(DistributionZonesConfiguration.KEY);
+
+ distributionZoneManager = new DistributionZoneManager(
+ zonesConfiguration,
+ tablesConfiguration,
+ metaStorageManager,
+ logicalTopologyService,
+ vaultManager,
+ "node"
+ );
+
+ mockCmgLocalNodes();
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ distributionZoneManager.stop();
+ clusterCfgMgr.stop();
+ clusterStateStorage.stop();
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added and removed nodes.
+ */
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUpAndScaleDown() throws Exception {
+ startZoneManager(0);
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ LOG.info("Topology with added and removed nodes.");
+
+ Set<String> dataNodes = Set.of("node0", "node2");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ CompletableFuture<Void> revisionUpFut = distributionZoneManager.zonesState().get(DEFAULT_ZONE_ID)
+ .scaleUpRevisionTracker().waitFor(testData.dataNodesRevision2);
+
+ assertFalse(revisionUpFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, true, testData.dataNodesRevision2,
+ testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(revisionUpFut::isDone, 3000));
+
+ CompletableFuture<Void> revisionDownFut = distributionZoneManager.zonesState().get(DEFAULT_ZONE_ID).scaleDownRevisionTracker()
+ .waitFor(testData.dataNodesRevision2);
+
+ assertFalse(revisionDownFut.isDone());
+
+ assertFalse(testData.dataNodesUpFut3.isDone());
+ assertFalse(testData.dataNodesDownFut3.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, false, testData.dataNodesRevision2,
+ testData.dataNodesRevision2 + 2);
+
+ assertTrue(waitForCondition(revisionDownFut::isDone, 3000));
+
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added nodes.
+ */
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUp() throws Exception {
+ startZoneManager(0);
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ LOG.info("Topology with added nodes.");
+
+ Set<String> dataNodes = Set.of("node0", "node1", "node2");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ CompletableFuture<Void> revisionUpFut = distributionZoneManager.zonesState().get(DEFAULT_ZONE_ID)
+ .scaleUpRevisionTracker().waitFor(testData.dataNodesRevision2);
+
+ assertFalse(revisionUpFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, true, testData.dataNodesRevision2,
+ testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(revisionUpFut::isDone, 3000));
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with removed nodes.
+ */
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleDown() throws Exception {
+ startZoneManager(0);
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject testData = testSeveralScaleUpAndSeveralScaleDownGeneral();
+
+ LOG.info("Topology with removed nodes.");
+
+ Set<String> dataNodes = Set.of("node0");
+
+ topologyWatchListenerOnUpdate(dataNodes, testData.topVer2, testData.dataNodesRevision2);
+
+ assertTrue(testData.topVerUpFut2.isDone());
+ assertTrue(testData.topVerDownFut2.isDone());
+
+ CompletableFuture<Void> revisionDownFut = distributionZoneManager.zonesState().get(DEFAULT_ZONE_ID)
+ .scaleDownRevisionTracker().waitFor((long) testData.dataNodesRevision2);
+
+ assertFalse(revisionDownFut.isDone());
+
+ dataNodesWatchListenerOnUpdate(DEFAULT_ZONE_ID, dataNodes, false, testData.dataNodesRevision2,
+ testData.dataNodesRevision2 + 1);
+
+ assertTrue(waitForCondition(revisionDownFut::isDone, 3000));
+ assertEquals(dataNodes, testData.dataNodesUpFut3.get(3, SECONDS));
+ assertEquals(dataNodes, testData.dataNodesDownFut3.get(3, SECONDS));
+ }
+
+ private static class TestSeveralScaleUpAndSeveralScaleDownDataObject {
+ private final long topVer2;
+ private final long dataNodesRevision2;
+ private final CompletableFuture<Void> topVerUpFut2;
+ private final CompletableFuture<Void> topVerDownFut2;
+ private final CompletableFuture<Set<String>> dataNodesUpFut3;
+ private final CompletableFuture<Set<String>> dataNodesDownFut3;
+
+ TestSeveralScaleUpAndSeveralScaleDownDataObject(
+ long topVer2,
+ long dataNodesRevision2,
+ CompletableFuture<Void> topVerUpFut2,
+ CompletableFuture<Void> topVerDownFut2,
+ CompletableFuture<Set<String>> dataNodesUpFut3,
+ CompletableFuture<Set<String>> dataNodesDownFut3) {
+ this.topVer2 = topVer2;
+ this.dataNodesRevision2 = dataNodesRevision2;
+ this.topVerUpFut2 = topVerUpFut2;
+ this.topVerDownFut2 = topVerDownFut2;
+ this.dataNodesUpFut3 = dataNodesUpFut3;
+ this.dataNodesDownFut3 = dataNodesDownFut3;
+ }
+ }
+
+ /**
+ * This method invokes {@link DistributionZoneManager#topologyVersionedDataNodes(int, long)} with default and non-default zone id
+ * and different logical topology version. Collects data nodes futures.
+ * Simulates new logical topology with new nodes. Check that some of data nodes futures are completed.
+ * Simulates new logical topology with removed nodes. Check that some of data nodes futures are completed.
+ *
+ * @return Structure with data for continue testing.
+ */
+ private TestSeveralScaleUpAndSeveralScaleDownDataObject testSeveralScaleUpAndSeveralScaleDownGeneral()
Review Comment:
I've removed TestSeveralScaleUpAndSeveralScaleDownDataObject
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163048905
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,910 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.configuration.NamedConfigurationTree;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.DistributionZoneManager.ZoneState;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TableChange;
+import org.apache.ignite.internal.schema.configuration.TableConfiguration;
+import org.apache.ignite.internal.schema.configuration.TableView;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultEntry;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private DistributionZonesConfiguration zonesConfiguration;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ @BeforeEach
+ void setUp() {
+ vaultManager = mock(VaultManager.class);
+
+ when(vaultManager.get(zonesLogicalTopologyKey())).thenReturn(completedFuture(new VaultEntry(zonesLogicalTopologyKey(), null)));
+
+ when(vaultManager.get(zonesLogicalTopologyVersionKey()))
+ .thenReturn(completedFuture(new VaultEntry(zonesLogicalTopologyVersionKey(), longToBytes(0))));
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ metaStorageManager = mock(MetaStorageManager.class);
+
+ doAnswer(invocation -> {
+ ByteArray key = invocation.getArgument(0);
+
+ WatchListener watchListener = invocation.getArgument(1);
+
+ if (Arrays.equals(key.bytes(), zoneLogicalTopologyPrefix().bytes())) {
+ topologyWatchListener = watchListener;
+ } else if (Arrays.equals(key.bytes(), zonesDataNodesPrefix().bytes())) {
+ dataNodesWatchListener = watchListener;
+ }
+
+ return null;
+ }).when(metaStorageManager).registerPrefixWatch(any(), any());
+
+ when(metaStorageManager.invoke(any()))
+ .thenReturn(completedFuture(StatementResultImpl.builder().result(new byte[] {0}).build()));
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ LogicalTopologyServiceImpl logicalTopologyService = new LogicalTopologyServiceImpl(logicalTopology, cmgManager);
+
+ TablesConfiguration tablesConfiguration = mock(TablesConfiguration.class);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] alievmirza commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1128154601
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -954,6 +1288,92 @@ public void onError(Throwable e) {
};
}
+ private WatchListener createMetastorageDataNodesListener() {
Review Comment:
Please add javadoc with explanation what is going on in this listner
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -954,6 +1288,92 @@ public void onError(Throwable e) {
};
}
+ private WatchListener createMetastorageDataNodesListener() {
+ return new WatchListener() {
+ @Override
+ public void onUpdate(WatchEvent evt) {
+ if (!busyLock.enterBusy()) {
+ throw new IgniteInternalException(NODE_STOPPING_ERR, new NodeStoppingException());
+ }
+
+ try {
+ int zoneId = 0;
+
+ Set<String> newDataNodes = null;
+
+ long scaleUpRevision = 0;
+
+ long scaleDownRevision = 0;
+
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
+
+ if (startsWith(e.key(), zoneDataNodesKey().bytes())) {
+ zoneId = extractZoneId(e.key());
+
+ byte[] dataNodesBytes = e.value();
+
+ if (dataNodesBytes != null) {
+ newDataNodes = DistributionZonesUtil.dataNodes(fromBytes(dataNodesBytes));
+ } else {
+ newDataNodes = emptySet();
+ }
+ } else if (startsWith(e.key(), zoneScaleUpChangeTriggerKey().bytes())) {
+ if (e.value() != null) {
+ scaleUpRevision = bytesToLong(e.value());
+ }
+ } else if (startsWith(e.key(), zoneScaleDownChangeTriggerKey().bytes())) {
+ if (e.value() != null) {
+ scaleDownRevision = bytesToLong(e.value());
+ }
+ }
+ }
+
+ synchronized (dataNodesMutex) {
+ DataNodes dataNodesMeta = dataNodes.get(zoneId);
+
+ if (dataNodesMeta != null) {
+ dataNodesMeta.nodes(newDataNodes);
+
+ if (scaleUpRevision > 0) {
Review Comment:
this code could be refactored and optimised, we have code duplication `if (scale*Revision > 0) {`
Also completion of futures from map can be extracted to a util method
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when dataNodesAutoAdjustScaleUp
+ * or dataNodesAutoAdjustScaleDown equals to 0.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
Review Comment:
also we need test scenarios, when non-immediate timer was triggered, and we try to get data nodes, I would expect that we receive recent propagated data nodes
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -954,6 +1288,92 @@ public void onError(Throwable e) {
};
}
+ private WatchListener createMetastorageDataNodesListener() {
+ return new WatchListener() {
+ @Override
+ public void onUpdate(WatchEvent evt) {
+ if (!busyLock.enterBusy()) {
+ throw new IgniteInternalException(NODE_STOPPING_ERR, new NodeStoppingException());
+ }
+
+ try {
+ int zoneId = 0;
+
+ Set<String> newDataNodes = null;
+
+ long scaleUpRevision = 0;
+
+ long scaleDownRevision = 0;
+
+ for (EntryEvent event : evt.entryEvents()) {
+ Entry e = event.newEntry();
+
+ if (startsWith(e.key(), zoneDataNodesKey().bytes())) {
+ zoneId = extractZoneId(e.key());
+
+ byte[] dataNodesBytes = e.value();
+
+ if (dataNodesBytes != null) {
+ newDataNodes = DistributionZonesUtil.dataNodes(fromBytes(dataNodesBytes));
+ } else {
+ newDataNodes = emptySet();
+ }
+ } else if (startsWith(e.key(), zoneScaleUpChangeTriggerKey().bytes())) {
+ if (e.value() != null) {
+ scaleUpRevision = bytesToLong(e.value());
+ }
+ } else if (startsWith(e.key(), zoneScaleDownChangeTriggerKey().bytes())) {
+ if (e.value() != null) {
+ scaleDownRevision = bytesToLong(e.value());
+ }
+ }
+ }
+
+ synchronized (dataNodesMutex) {
+ DataNodes dataNodesMeta = dataNodes.get(zoneId);
+
+ if (dataNodesMeta != null) {
+ dataNodesMeta.nodes(newDataNodes);
+
+ if (scaleUpRevision > 0) {
+ dataNodesMeta.scaleUpRevision(scaleUpRevision);
+ }
+
+ if (scaleDownRevision > 0) {
+ dataNodesMeta.scaleDownRevision(scaleDownRevision);
+ }
+
+ if (scaleUpRevision > 0) {
+ SortedMap<Long, CompletableFuture<Void>> revisionScaleUpFuts =
+ dataNodes.get(zoneId).revisionScaleUpFutures().headMap(scaleUpRevision, true);
+
+ revisionScaleUpFuts.values().forEach(v -> v.complete(null));
+
+ revisionScaleUpFuts.clear();
+ }
+
+ if (scaleDownRevision > 0) {
+ SortedMap<Long, CompletableFuture<Void>> revisionScaleDownFuts =
+ dataNodes.get(zoneId).revisionScaleDownFutures().headMap(scaleDownRevision, true);
+
+ revisionScaleDownFuts.values().forEach(v -> v.complete(null));
+
+ revisionScaleDownFuts.clear();
+ }
+ }
+ }
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ @Override
+ public void onError(Throwable e) {
+ LOG.warn("Unable to process logical topology event", e);
Review Comment:
this is not a logical topology event
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156783772
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
Review Comment:
I updated java doc. But I don't think this is the last java doc fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156790436
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
Review Comment:
I think its is not possible. As we discussed we will use this method in the configuration closure so we don't need to throw this exception. And first in this closure we get zone id by zone name. If we there is no zone with given zone name than the exception will be thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1166978507
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,605 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willBe;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalNode;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologySnapshot;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.Conditions;
+import org.apache.ignite.internal.metastorage.dsl.Operations;
+import org.apache.ignite.internal.metastorage.impl.StandaloneMetaStorageManager;
+import org.apache.ignite.internal.metastorage.server.SimpleInMemoryKeyValueStorage;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.internal.vault.inmemory.InMemoryVaultService;
+import org.apache.ignite.lang.ByteArray;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.NetworkAddress;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ @InjectConfiguration
+ private TablesConfiguration tablesConfiguration;
+
+ @InjectConfiguration
+ protected DistributionZonesConfiguration zonesConfiguration;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ protected SimpleInMemoryKeyValueStorage keyValueStorage;
+
+ private final List<IgniteComponent> components = new ArrayList<>();
+
+ @BeforeEach
+ void setUp() throws Exception {
+ vaultManager = new VaultManager(new InMemoryVaultService());
+
+ assertThat(vaultManager.put(zonesLogicalTopologyKey(), null), willCompleteSuccessfully());
+ assertThat(vaultManager.put(zonesLogicalTopologyVersionKey(), longToBytes(0)), willCompleteSuccessfully());
+
+ components.add(vaultManager);
+
+ keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
+
+ metaStorageManager = StandaloneMetaStorageManager.create(vaultManager, keyValueStorage);
+
+ components.add(metaStorageManager);
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ components.add(clusterStateStorage);
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ distributionZoneManager = new DistributionZoneManager(
+ zonesConfiguration,
+ tablesConfiguration,
+ metaStorageManager,
+ new LogicalTopologyServiceImpl(logicalTopology, cmgManager),
+ vaultManager,
+ "test"
+ );
+
+ mockCmgLocalNodes();
+
+ // Not adding 'distributionZoneManager' on purpose, it's started manually.
+ components.forEach(IgniteComponent::start);
+
+ metaStorageManager.deployWatches();
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ components.add(distributionZoneManager);
+
+ Collections.reverse(components);
+
+ IgniteUtils.closeAll(components.stream().map(c -> c::beforeNodeStop));
+ IgniteUtils.closeAll(components.stream().map(c -> c::stop));
+ }
+
+ /**
+ * This test invokes {@link DistributionZoneManager#topologyVersionedDataNodes(int, long)} with default and non-default zone id
+ * and different logical topology versions.
+ * Simulates new logical topology with new nodes and with removed nodes. Check that data nodes futures are completed in right order.
+ */
+ @Disabled("https://issues.apache.org/jira/browse/IGNITE-19288")
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUpAndScaleDown() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone0")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId0 = distributionZoneManager.getZoneId("zone0");
+ int zoneId1 = distributionZoneManager.getZoneId("zone1");
+
+ LOG.info("Topology with added nodes.");
+
+ CompletableFuture<Set<String>> dataNodesUpFut0 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut1 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut2 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 2);
+ CompletableFuture<Set<String>> dataNodesUpFut3 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 11);
+ CompletableFuture<Set<String>> dataNodesUpFut4 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut5 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 2);
+ CompletableFuture<Set<String>> dataNodesUpFut6 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut7 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 2);
+
+ int topVer0 = 2;
+
+ Set<String> threeNodes = Set.of("node0", "node1", "node2");
+
+ setLogicalTopologyInMetaStorage(threeNodes, topVer0);
+
+ assertEquals(threeNodes, dataNodesUpFut0.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut1.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut2.get(3, SECONDS));
+
+ assertEquals(threeNodes, dataNodesUpFut4.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut5.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut6.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut7.get(3, SECONDS));
+ assertFalse(dataNodesUpFut3.isDone());
+
+
+ LOG.info("Topology with removed nodes.");
+
+ CompletableFuture<Set<String>> dataNodesDownFut0 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut1 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut2 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 5);
+ CompletableFuture<Set<String>> dataNodesDownFut3 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 6);
+ CompletableFuture<Set<String>> dataNodesDownFut4 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut5 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 5);
+ CompletableFuture<Set<String>> dataNodesDownFut6 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut7 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 5);
+
+ int topVer1 = 5;
+
+ Set<String> twoNodes = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(twoNodes, topVer1);
+
+ assertEquals(twoNodes, dataNodesDownFut0.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut1.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut2.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut4.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut5.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut6.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut7.get(3, SECONDS));
+ assertFalse(dataNodesDownFut3.isDone());
+
+ int topVer2 = 20;
+
+ LOG.info("Topology with added and removed nodes.");
+
+ Set<String> dataNodes = Set.of("node0", "node2");
+
+ setLogicalTopologyInMetaStorage(dataNodes, topVer2);
+
+ assertEquals(dataNodes, dataNodesUpFut3.get(3, SECONDS));
+ assertEquals(dataNodes, dataNodesDownFut3.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added nodes.
+ */
+ @Test
+ void testScaleUpAndThenScaleDown() throws Exception {
+ startZoneManager(0);
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 5);
+
+ assertFalse(dataNodesFut.isDone());
+
+ long topVer = 100;
+
+ Set<String> dataNodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(dataNodes0, topVer);
+
+ assertFalse(dataNodesFut.isDone());
+
+ assertEquals(dataNodes0, dataNodesFut.get(3, SECONDS));
+
+ dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 106);
+
+ Set<String> dataNodes1 = Set.of("node0");
+
+ setLogicalTopologyInMetaStorage(dataNodes1, topVer + 100);
+
+ assertFalse(dataNodesFut.isDone());
+
+ assertEquals(dataNodes1, dataNodesFut.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added and removed nodes for the zone with
+ * dataNodesAutoAdjustScaleUp is immediate and dataNodesAutoAdjustScaleDown is non-zero.
+ */
+ @Test
+ void testAwaitingScaleUpOnly() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE).dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId = distributionZoneManager.getZoneId("zone1");
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(zoneId, 1);
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(nodes0, 1);
+
+ assertEquals(nodes0, dataNodesFut.get(3, SECONDS));
+
+ dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(zoneId, 2);
+
+ assertFalse(dataNodesFut.isDone());
+
+ setLogicalTopologyInMetaStorage(Set.of("node0"), 2);
+
+ assertEquals(nodes0, dataNodesFut.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added and removed nodes for the zone with
+ * dataNodesAutoAdjustScaleUp is non-zero and dataNodesAutoAdjustScaleDown is immediate. And checks that other zones
+ * non-zero timers doesn't affect.
+ */
+ @Test
+ void testAwaitingScaleDownOnly() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE).dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone0")
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone2")
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId0 = distributionZoneManager.getZoneId("zone0");
+ int zoneId1 = distributionZoneManager.getZoneId("zone1");
+ int zoneId2 = distributionZoneManager.getZoneId("zone2");
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 1);
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(nodes0, 1);
+
+ dataNodesFut.get(3, SECONDS);
+
+ CompletableFuture<Set<String>> dataNodesFut1Zone0 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 2);
+ CompletableFuture<Set<String>> dataNodesFut1 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 2);
+ CompletableFuture<Set<String>> dataNodesFut1Zone2 = distributionZoneManager.topologyVersionedDataNodes(zoneId2, 2);
+
+ assertFalse(dataNodesFut1Zone0.isDone());
+ assertFalse(dataNodesFut1.isDone());
+ assertFalse(dataNodesFut1Zone2.isDone());
+
+ Set<String> nodes1 = Set.of("node0");
+
+ distributionZoneManager.alterZone("zone1", new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE).dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+
+ System.out.println("setLogicalTopologyInMetaStorage_2");
+ setLogicalTopologyInMetaStorage(nodes1, 2);
+
+ assertEquals(nodes1, dataNodesFut1.get(3, SECONDS));
+
+ CompletableFuture<Set<String>> dataNodesFut2 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 3);
+
+ Set<String> nodes2 = Set.of("node0", "node1");
+
+ assertFalse(dataNodesFut2.isDone());
+
+ setLogicalTopologyInMetaStorage(nodes2, 3);
+
+ assertEquals(nodes1, dataNodesFut2.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed immediately for the zone with
+ * dataNodesAutoAdjustScaleUp is non-zero and dataNodesAutoAdjustScaleDown is non-zero.
+ */
+ @Test
+ void testWithOutAwaiting() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE).dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId = distributionZoneManager.getZoneId("zone1");
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(zoneId, 1);
+
+ assertFalse(dataNodesFut.isDone());
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(nodes0, 1);
+
+ assertEquals(emptySet(), dataNodesFut.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed exceptionally if the zone was removed while
+ * data nodes awaiting.
+ */
+ @Disabled("https://issues.apache.org/jira/browse/IGNITE-19255")
+ @Test
+ void testRemoveZoneWhileAwaitingDataNodes() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone0")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId = distributionZoneManager.getZoneId("zone0");
+
+ CompletableFuture<Set<String>> dataNodesFut0 = distributionZoneManager.topologyVersionedDataNodes(zoneId, 5);
+
+ setLogicalTopologyInMetaStorage(Set.of("node0", "node1"), 100);
+
+ assertFalse(dataNodesFut0.isDone());
+
+ assertEquals(Set.of("node0", "node1"), dataNodesFut0.get(3, SECONDS));
+
+ CompletableFuture<Set<String>> dataNodesFut1 = distributionZoneManager.topologyVersionedDataNodes(zoneId, 106);
+
+ setLogicalTopologyInMetaStorage(Set.of("node0", "node2"), 200);
+
+ assertFalse(dataNodesFut1.isDone());
+
+ distributionZoneManager.dropZone("zone0").get();
+
+ assertThrowsWithCause(() -> dataNodesFut1.get(3, SECONDS), DistributionZoneWasRemovedException.class);
+ }
+
+ /**
+ * Test checks that data nodes futures are completed with old data nodes if dataNodesAutoAdjustScaleUp
+ * and dataNodesAutoAdjustScaleDown timer increased to non-zero value.
+ */
+ @Disabled("https://issues.apache.org/jira/browse/IGNITE-19255")
+ @Test
+ void testScaleUpScaleDownAreChangedWhileAwaitingDataNodes() throws Exception {
+ startZoneManager(0);
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(nodes0, 1);
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 1);
+
+ assertEquals(nodes0, dataNodesFut.get(3, SECONDS));
+
+ Set<String> nodes1 = Set.of("node0", "node2");
+
+ dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 2);
+
+ setLogicalTopologyInMetaStorage(nodes1, 2);
+
+ assertFalse(dataNodesFut.isDone());
+
+ //need to create new zone to fix assert invariant which is broken in this test environment.
+ distributionZoneManager.createZone(new DistributionZoneConfigurationParameters.Builder("zone0")
+ .dataNodesAutoAdjustScaleUp(1000).dataNodesAutoAdjustScaleDown(1000).build())
+ .get(3, SECONDS);
+
+ assertFalse(dataNodesFut.isDone());
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(1000).dataNodesAutoAdjustScaleDown(1000).build())
+ .get(3, SECONDS);
+
+ assertEquals(nodes0, dataNodesFut.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes are initialized on zone manager start.
+ */
+ @Test
+ void testInitializedDataNodesOnZoneManagerStart() throws Exception {
+ Set<String> dataNodes = Set.of("node0", "node1");
+
+ Map<ByteArray, byte[]> valEntries = new HashMap<>();
+
+ valEntries.put(zonesLogicalTopologyKey(), toBytes(dataNodes));
+ valEntries.put(zonesLogicalTopologyVersionKey(), longToBytes(3));
+
+ vaultManager.putAll(valEntries);
+
+ Collection<LogicalNode> nodes = new ArrayList<>();
+
+ nodes.add(new LogicalNode(new ClusterNode("node0", "node0", new NetworkAddress("local", 1))));
+ nodes.add(new LogicalNode(new ClusterNode("node1", "node1", new NetworkAddress("local", 1))));
+
+ when(cmgManager.logicalTopology()).thenReturn(completedFuture(new LogicalTopologySnapshot(3, nodes)));
+
+ startZoneManager(10);
+
+ assertEquals(dataNodes, distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 2)
+ .get(3, SECONDS));
+ }
+
+ private void startZoneManager(long revision) throws Exception {
+ vaultManager.put(new ByteArray("applied_revision"), longToBytes(revision)).get();
+
+ distributionZoneManager.start();
+
+ distributionZoneManager.alterZone(
+ DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+ }
+
+ private void setLogicalTopologyInMetaStorage(Set<String> nodes, long topVer) {
+ CompletableFuture<Boolean> invokeFuture = metaStorageManager.invoke(
+ Conditions.exists(zonesLogicalTopologyKey()),
+ List.of(
+ Operations.put(zonesLogicalTopologyKey(), toBytes(nodes)),
+ Operations.put(zonesLogicalTopologyVersionKey(), longToBytes(topVer))
+ ),
+ List.of(Operations.noop())
+ );
+
+ assertThat(invokeFuture, willBe(true));
+ }
+
+ private void setNodesWatchInMetaStorage(int zoneId, Set<String> nodes, boolean isScaleUp, long scaleRevision) {
Review Comment:
Now it is not needed. I removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1166978902
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,605 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willBe;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalNode;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologySnapshot;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.Conditions;
+import org.apache.ignite.internal.metastorage.dsl.Operations;
+import org.apache.ignite.internal.metastorage.impl.StandaloneMetaStorageManager;
+import org.apache.ignite.internal.metastorage.server.SimpleInMemoryKeyValueStorage;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.internal.vault.inmemory.InMemoryVaultService;
+import org.apache.ignite.lang.ByteArray;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.NetworkAddress;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ @InjectConfiguration
+ private TablesConfiguration tablesConfiguration;
+
+ @InjectConfiguration
+ protected DistributionZonesConfiguration zonesConfiguration;
Review Comment:
Fixed. Copy/paste.
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,605 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willBe;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalNode;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologySnapshot;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.Conditions;
+import org.apache.ignite.internal.metastorage.dsl.Operations;
+import org.apache.ignite.internal.metastorage.impl.StandaloneMetaStorageManager;
+import org.apache.ignite.internal.metastorage.server.SimpleInMemoryKeyValueStorage;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.internal.vault.inmemory.InMemoryVaultService;
+import org.apache.ignite.lang.ByteArray;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.NetworkAddress;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ @InjectConfiguration
+ private TablesConfiguration tablesConfiguration;
+
+ @InjectConfiguration
+ protected DistributionZonesConfiguration zonesConfiguration;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ protected SimpleInMemoryKeyValueStorage keyValueStorage;
Review Comment:
Fixed. Copy/paste.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1131486455
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.configuration.annotation.ConfigurationType.DISTRIBUTED;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneLogicalTopologyPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesDataNodesPrefix;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.storage.TestConfigurationStorage;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.EntryEvent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchEvent;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.StatementResultImpl;
+import org.apache.ignite.internal.metastorage.impl.EntryImpl;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.lang.ByteArray;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when dataNodesAutoAdjustScaleUp
+ * or dataNodesAutoAdjustScaleDown equals to 0.
+ */
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private DistributionZonesConfiguration zonesConfiguration;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ @BeforeEach
+ void setUp() throws Exception {
+ VaultManager vaultManager = mock(VaultManager.class);
+
+ when(vaultManager.get(any())).thenReturn(completedFuture(null));
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ metaStorageManager = mock(MetaStorageManager.class);
+
+ doAnswer(invocation -> {
+ ByteArray key = invocation.getArgument(0);
+
+ WatchListener watchListener = invocation.getArgument(1);
+
+ if (Arrays.equals(key.bytes(), zoneLogicalTopologyPrefix().bytes())) {
+ topologyWatchListener = watchListener;
+ } else if (Arrays.equals(key.bytes(), zonesDataNodesPrefix().bytes())) {
+ dataNodesWatchListener = watchListener;
+ }
+
+ return null;
+ }).when(metaStorageManager).registerPrefixWatch(any(), any());
+
+ when(metaStorageManager.invoke(any())).thenReturn(completedFuture(StatementResultImpl.builder().result(new byte[] {0}).build()));
+
+ vaultManager.start();
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ LogicalTopologyServiceImpl logicalTopologyService = new LogicalTopologyServiceImpl(logicalTopology, cmgManager);
+
+ TablesConfiguration tablesConfiguration = mock(TablesConfiguration.class);
+
+ clusterCfgMgr = new ConfigurationManager(
+ List.of(DistributionZonesConfiguration.KEY),
+ Set.of(),
+ new TestConfigurationStorage(DISTRIBUTED),
+ List.of(),
+ List.of()
+ );
+
+ clusterCfgMgr.start();
+
+ zonesConfiguration = clusterCfgMgr.configurationRegistry()
+ .getConfiguration(DistributionZonesConfiguration.KEY);
+
+ distributionZoneManager = new DistributionZoneManager(
+ zonesConfiguration,
+ tablesConfiguration,
+ metaStorageManager,
+ logicalTopologyService,
+ vaultManager,
+ "node"
+ );
+
+ mockCmgLocalNodes();
+
+ distributionZoneManager.start();
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(0).dataNodesAutoAdjustScaleDown(0).build())
+ .get(3, TimeUnit.SECONDS);
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ distributionZoneManager.stop();
+ clusterCfgMgr.stop();
+ clusterStateStorage.stop();
+ }
+
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUpAndScaleDown() throws Exception {
Review Comment:
I added non-default zones. Actually awaiting logic doesn't depend on default or non-default zones.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZonesUtil.java:
##########
@@ -136,6 +171,10 @@ static ByteArray zonesLogicalTopologyVersionKey() {
return DISTRIBUTION_ZONES_LOGICAL_TOPOLOGY_VERSION_KEY;
}
+ static ByteArray zonesDataNodesPrefix() {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1132038601
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -183,8 +221,89 @@ public void onTopologyLeap(LogicalTopologySnapshot newTopology) {
*/
private volatile Set<String> logicalTopology;
- /** Watch listener. Needed to unregister it on {@link DistributionZoneManager#stop()}. */
- private final WatchListener watchListener;
+ /** Watch listener for logical topology keys. */
+ private final WatchListener topologyWatchListener;
+
+ /** Watch listener for data nodes keys. */
+ private final WatchListener dataNodesWatchListener;
+
+ /**
+ * Contains data nodes and meta info.
+ */
+ static class DataNodes {
+ /** Data nodes. */
+ private Set<String> nodes;
+
+ /** Scale up metastorage revision of current nodes value. */
+ private long scaleUpRevision;
+
+ /** Scale down metastorage revision of current nodes value. */
+ private long scaleDownRevision;
+
+ /**
+ * The map contains futures which are completed when zone manager observe data nodes bound to appropriate scale up revision.
+ * Map (revision -> future).
+ */
+ private final NavigableMap<Long, CompletableFuture<Void>> revisionScaleUpFutures = new ConcurrentSkipListMap();
+
+ /**
+ * The map contains futures which are completed when zone manager observe data nodes bound to appropriate scale down revision.
+ * Map (revision -> future).
+ */
+ private final NavigableMap<Long, CompletableFuture<Void>> revisionScaleDownFutures = new ConcurrentSkipListMap();
+
+ DataNodes() {
+ nodes = emptySet();
+ }
+
+ DataNodes(Set<String> nodes, long scaleUpRevision, long scaleDownRevision) {
+ this.nodes = nodes;
+ this.scaleUpRevision = scaleUpRevision;
+ this.scaleDownRevision = scaleDownRevision;
+ }
+
+ Set<String> nodes() {
+ return nodes;
+ }
+
+ void nodes(Set<String> nodes) {
+ this.nodes = nodes;
+ }
+
+ long scaleUpRevision() {
+ return scaleUpRevision;
+ }
+
+ void scaleUpRevision(long scaleUpRevision) {
+ this.scaleUpRevision = scaleUpRevision;
+ }
+
+ long scaleDownRevision() {
+ return scaleDownRevision;
+ }
+
+ void scaleDownRevision(long scaleDownRevision) {
+ this.scaleDownRevision = scaleDownRevision;
+ }
+
+ NavigableMap<Long, CompletableFuture<Void>> revisionScaleUpFutures() {
+ return revisionScaleUpFutures;
+ }
+
+ NavigableMap<Long, CompletableFuture<Void>> revisionScaleDownFutures() {
+ return revisionScaleDownFutures;
+ }
+ }
+
+ @TestOnly
+ Map<Integer, DataNodes> dataNodes() {
Review Comment:
Now I use it in synchronized block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163049629
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,133 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it completes the future with data nodes of the specified zone.
+ * This method must be invoked in a change configuration closure to guarantee that the zone is exists and values of scaleUp/scaleDown
+ * timers are up to date.
+ * The returned future can be completed with {@link DistributionZoneNotFoundException} and {@link DistributionZoneWasRemovedException}
+ * in case when the distribution zone was removed during method execution.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = awaitTopologyVersion(topVer)
+ .thenCompose(ignored -> getImmediateTimers(zoneId));
+
+ return allOf(
+ timerValuesFut.thenCompose(timerValues -> scaleUpAwaiting(zoneId, timerValues.get1())),
+ timerValuesFut.thenCompose(timerValues -> scaleDownAwaiting(zoneId, timerValues.get2()))
+ ).thenCompose(ignored -> getDataNodesFuture(zoneId));
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ return inBusyLock(busyLock, () -> topVerTracker.waitFor(topVer));
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be started immediate.
+ *
+ * @param zoneId Zone id.
+ * @return Future with the result.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId) {
+ return inBusyLock(busyLock, () -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
+
+ return completedFuture(new IgniteBiTuple<>(
+ zoneCfg.dataNodesAutoAdjustScaleUp().value() == IMMEDIATE_TIMER_VALUE,
+ zoneCfg.dataNodesAutoAdjustScaleDown().value() == IMMEDIATE_TIMER_VALUE
+ ));
+ });
+ }
+
+ /**
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than waits for writing data nodes triggered
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163728597
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,148 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining the data nodes of the specified zone.
+ * If {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} and
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate then it waits that the data nodes
+ * are up-to-date for the passed topology version.
+ *
+ * <p>If the values of auto adjust scale up and auto adjust scale down are zero, then on the cluster topology changes
+ * the data nodes for the zone should be updated immediately. Therefore, this method must return the data nodes which is calculated
+ * based on the topology with passed or greater version. Since the date nodes value is updated asynchronously, this method waits for
+ * the date nodes to be updated with new nodes in the topology if the value of auto adjust scale up is 0. And also waits for
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1164134059
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -505,6 +543,148 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining the data nodes of the specified zone.
+ * If {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} and
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate then it waits that the data nodes
+ * are up-to-date for the passed topology version.
+ *
+ * <p>If the values of auto adjust scale up and auto adjust scale down are zero, then on the cluster topology changes
+ * the data nodes for the zone should be updated immediately. Therefore, this method must return the data nodes which is calculated
+ * based on the topology with passed or greater version. Since the date nodes value is updated asynchronously, this method waits for
+ * the date nodes to be updated with new nodes in the topology if the value of auto adjust scale up is 0. And also waits for
+ * the date nodes to be updated with nodes that have left the topology if the value of auto adjust scale down is 0.
+ * After the zone manager has observed the logical topology change and the data nodes value is updated according to cluster topology,
+ * then this method completes the returned future with the current value of data nodes.
+ *
+ * <p>If the value of auto adjust scale up is greater than zero, then it is not necessary to wait for the data nodes update triggered
+ * by new nodes in cluster topology. Similarly if the value of auto adjust scale down is greater than zero, then it is not necessary to
+ * wait for the data nodes update triggered by new nodes that have left the topology in cluster topology.
+ *
+ * <p>The returned future can be completed with {@link DistributionZoneNotFoundException} and
+ * {@link DistributionZoneWasRemovedException} in case when the distribution zone was removed during method execution.
Review Comment:
Discussed in private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1163766347
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -484,6 +537,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] alievmirza commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1166971143
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,605 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willBe;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalNode;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologySnapshot;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.Conditions;
+import org.apache.ignite.internal.metastorage.dsl.Operations;
+import org.apache.ignite.internal.metastorage.impl.StandaloneMetaStorageManager;
+import org.apache.ignite.internal.metastorage.server.SimpleInMemoryKeyValueStorage;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.internal.vault.inmemory.InMemoryVaultService;
+import org.apache.ignite.lang.ByteArray;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.NetworkAddress;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ @InjectConfiguration
+ private TablesConfiguration tablesConfiguration;
+
+ @InjectConfiguration
+ protected DistributionZonesConfiguration zonesConfiguration;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ protected SimpleInMemoryKeyValueStorage keyValueStorage;
Review Comment:
why protected?
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,605 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willBe;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalNode;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologySnapshot;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.Conditions;
+import org.apache.ignite.internal.metastorage.dsl.Operations;
+import org.apache.ignite.internal.metastorage.impl.StandaloneMetaStorageManager;
+import org.apache.ignite.internal.metastorage.server.SimpleInMemoryKeyValueStorage;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.internal.vault.inmemory.InMemoryVaultService;
+import org.apache.ignite.lang.ByteArray;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.NetworkAddress;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ @InjectConfiguration
+ private TablesConfiguration tablesConfiguration;
+
+ @InjectConfiguration
+ protected DistributionZonesConfiguration zonesConfiguration;
+
+ private WatchListener topologyWatchListener;
+
+ private WatchListener dataNodesWatchListener;
+
+ protected SimpleInMemoryKeyValueStorage keyValueStorage;
+
+ private final List<IgniteComponent> components = new ArrayList<>();
+
+ @BeforeEach
+ void setUp() throws Exception {
+ vaultManager = new VaultManager(new InMemoryVaultService());
+
+ assertThat(vaultManager.put(zonesLogicalTopologyKey(), null), willCompleteSuccessfully());
+ assertThat(vaultManager.put(zonesLogicalTopologyVersionKey(), longToBytes(0)), willCompleteSuccessfully());
+
+ components.add(vaultManager);
+
+ keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
+
+ metaStorageManager = StandaloneMetaStorageManager.create(vaultManager, keyValueStorage);
+
+ components.add(metaStorageManager);
+
+ cmgManager = mock(ClusterManagementGroupManager.class);
+
+ clusterStateStorage = new TestClusterStateStorage();
+
+ components.add(clusterStateStorage);
+
+ logicalTopology = new LogicalTopologyImpl(clusterStateStorage);
+
+ distributionZoneManager = new DistributionZoneManager(
+ zonesConfiguration,
+ tablesConfiguration,
+ metaStorageManager,
+ new LogicalTopologyServiceImpl(logicalTopology, cmgManager),
+ vaultManager,
+ "test"
+ );
+
+ mockCmgLocalNodes();
+
+ // Not adding 'distributionZoneManager' on purpose, it's started manually.
+ components.forEach(IgniteComponent::start);
+
+ metaStorageManager.deployWatches();
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ components.add(distributionZoneManager);
+
+ Collections.reverse(components);
+
+ IgniteUtils.closeAll(components.stream().map(c -> c::beforeNodeStop));
+ IgniteUtils.closeAll(components.stream().map(c -> c::stop));
+ }
+
+ /**
+ * This test invokes {@link DistributionZoneManager#topologyVersionedDataNodes(int, long)} with default and non-default zone id
+ * and different logical topology versions.
+ * Simulates new logical topology with new nodes and with removed nodes. Check that data nodes futures are completed in right order.
+ */
+ @Disabled("https://issues.apache.org/jira/browse/IGNITE-19288")
+ @Test
+ void testSeveralScaleUpAndSeveralScaleDownThenScaleUpAndScaleDown() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone0")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId0 = distributionZoneManager.getZoneId("zone0");
+ int zoneId1 = distributionZoneManager.getZoneId("zone1");
+
+ LOG.info("Topology with added nodes.");
+
+ CompletableFuture<Set<String>> dataNodesUpFut0 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut1 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut2 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 2);
+ CompletableFuture<Set<String>> dataNodesUpFut3 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 11);
+ CompletableFuture<Set<String>> dataNodesUpFut4 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut5 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 2);
+ CompletableFuture<Set<String>> dataNodesUpFut6 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 1);
+ CompletableFuture<Set<String>> dataNodesUpFut7 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 2);
+
+ int topVer0 = 2;
+
+ Set<String> threeNodes = Set.of("node0", "node1", "node2");
+
+ setLogicalTopologyInMetaStorage(threeNodes, topVer0);
+
+ assertEquals(threeNodes, dataNodesUpFut0.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut1.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut2.get(3, SECONDS));
+
+ assertEquals(threeNodes, dataNodesUpFut4.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut5.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut6.get(3, SECONDS));
+ assertEquals(threeNodes, dataNodesUpFut7.get(3, SECONDS));
+ assertFalse(dataNodesUpFut3.isDone());
+
+
+ LOG.info("Topology with removed nodes.");
+
+ CompletableFuture<Set<String>> dataNodesDownFut0 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut1 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut2 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 5);
+ CompletableFuture<Set<String>> dataNodesDownFut3 = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 6);
+ CompletableFuture<Set<String>> dataNodesDownFut4 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut5 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 5);
+ CompletableFuture<Set<String>> dataNodesDownFut6 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 4);
+ CompletableFuture<Set<String>> dataNodesDownFut7 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 5);
+
+ int topVer1 = 5;
+
+ Set<String> twoNodes = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(twoNodes, topVer1);
+
+ assertEquals(twoNodes, dataNodesDownFut0.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut1.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut2.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut4.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut5.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut6.get(3, SECONDS));
+ assertEquals(twoNodes, dataNodesDownFut7.get(3, SECONDS));
+ assertFalse(dataNodesDownFut3.isDone());
+
+ int topVer2 = 20;
+
+ LOG.info("Topology with added and removed nodes.");
+
+ Set<String> dataNodes = Set.of("node0", "node2");
+
+ setLogicalTopologyInMetaStorage(dataNodes, topVer2);
+
+ assertEquals(dataNodes, dataNodesUpFut3.get(3, SECONDS));
+ assertEquals(dataNodes, dataNodesDownFut3.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added nodes.
+ */
+ @Test
+ void testScaleUpAndThenScaleDown() throws Exception {
+ startZoneManager(0);
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 5);
+
+ assertFalse(dataNodesFut.isDone());
+
+ long topVer = 100;
+
+ Set<String> dataNodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(dataNodes0, topVer);
+
+ assertFalse(dataNodesFut.isDone());
+
+ assertEquals(dataNodes0, dataNodesFut.get(3, SECONDS));
+
+ dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 106);
+
+ Set<String> dataNodes1 = Set.of("node0");
+
+ setLogicalTopologyInMetaStorage(dataNodes1, topVer + 100);
+
+ assertFalse(dataNodesFut.isDone());
+
+ assertEquals(dataNodes1, dataNodesFut.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added and removed nodes for the zone with
+ * dataNodesAutoAdjustScaleUp is immediate and dataNodesAutoAdjustScaleDown is non-zero.
+ */
+ @Test
+ void testAwaitingScaleUpOnly() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE).dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId = distributionZoneManager.getZoneId("zone1");
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(zoneId, 1);
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(nodes0, 1);
+
+ assertEquals(nodes0, dataNodesFut.get(3, SECONDS));
+
+ dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(zoneId, 2);
+
+ assertFalse(dataNodesFut.isDone());
+
+ setLogicalTopologyInMetaStorage(Set.of("node0"), 2);
+
+ assertEquals(nodes0, dataNodesFut.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed on topology with added and removed nodes for the zone with
+ * dataNodesAutoAdjustScaleUp is non-zero and dataNodesAutoAdjustScaleDown is immediate. And checks that other zones
+ * non-zero timers doesn't affect.
+ */
+ @Test
+ void testAwaitingScaleDownOnly() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE).dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone0")
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone2")
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId0 = distributionZoneManager.getZoneId("zone0");
+ int zoneId1 = distributionZoneManager.getZoneId("zone1");
+ int zoneId2 = distributionZoneManager.getZoneId("zone2");
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 1);
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(nodes0, 1);
+
+ dataNodesFut.get(3, SECONDS);
+
+ CompletableFuture<Set<String>> dataNodesFut1Zone0 = distributionZoneManager.topologyVersionedDataNodes(zoneId0, 2);
+ CompletableFuture<Set<String>> dataNodesFut1 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 2);
+ CompletableFuture<Set<String>> dataNodesFut1Zone2 = distributionZoneManager.topologyVersionedDataNodes(zoneId2, 2);
+
+ assertFalse(dataNodesFut1Zone0.isDone());
+ assertFalse(dataNodesFut1.isDone());
+ assertFalse(dataNodesFut1Zone2.isDone());
+
+ Set<String> nodes1 = Set.of("node0");
+
+ distributionZoneManager.alterZone("zone1", new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE).dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+
+ System.out.println("setLogicalTopologyInMetaStorage_2");
+ setLogicalTopologyInMetaStorage(nodes1, 2);
+
+ assertEquals(nodes1, dataNodesFut1.get(3, SECONDS));
+
+ CompletableFuture<Set<String>> dataNodesFut2 = distributionZoneManager.topologyVersionedDataNodes(zoneId1, 3);
+
+ Set<String> nodes2 = Set.of("node0", "node1");
+
+ assertFalse(dataNodesFut2.isDone());
+
+ setLogicalTopologyInMetaStorage(nodes2, 3);
+
+ assertEquals(nodes1, dataNodesFut2.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed immediately for the zone with
+ * dataNodesAutoAdjustScaleUp is non-zero and dataNodesAutoAdjustScaleDown is non-zero.
+ */
+ @Test
+ void testWithOutAwaiting() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE).dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone1")
+ .dataNodesAutoAdjustScaleUp(INFINITE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(INFINITE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId = distributionZoneManager.getZoneId("zone1");
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(zoneId, 1);
+
+ assertFalse(dataNodesFut.isDone());
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(nodes0, 1);
+
+ assertEquals(emptySet(), dataNodesFut.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes futures are completed exceptionally if the zone was removed while
+ * data nodes awaiting.
+ */
+ @Disabled("https://issues.apache.org/jira/browse/IGNITE-19255")
+ @Test
+ void testRemoveZoneWhileAwaitingDataNodes() throws Exception {
+ startZoneManager(0);
+
+ distributionZoneManager.createZone(
+ new DistributionZoneConfigurationParameters.Builder("zone0")
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE)
+ .build()
+ )
+ .get(3, SECONDS);
+
+ int zoneId = distributionZoneManager.getZoneId("zone0");
+
+ CompletableFuture<Set<String>> dataNodesFut0 = distributionZoneManager.topologyVersionedDataNodes(zoneId, 5);
+
+ setLogicalTopologyInMetaStorage(Set.of("node0", "node1"), 100);
+
+ assertFalse(dataNodesFut0.isDone());
+
+ assertEquals(Set.of("node0", "node1"), dataNodesFut0.get(3, SECONDS));
+
+ CompletableFuture<Set<String>> dataNodesFut1 = distributionZoneManager.topologyVersionedDataNodes(zoneId, 106);
+
+ setLogicalTopologyInMetaStorage(Set.of("node0", "node2"), 200);
+
+ assertFalse(dataNodesFut1.isDone());
+
+ distributionZoneManager.dropZone("zone0").get();
+
+ assertThrowsWithCause(() -> dataNodesFut1.get(3, SECONDS), DistributionZoneWasRemovedException.class);
+ }
+
+ /**
+ * Test checks that data nodes futures are completed with old data nodes if dataNodesAutoAdjustScaleUp
+ * and dataNodesAutoAdjustScaleDown timer increased to non-zero value.
+ */
+ @Disabled("https://issues.apache.org/jira/browse/IGNITE-19255")
+ @Test
+ void testScaleUpScaleDownAreChangedWhileAwaitingDataNodes() throws Exception {
+ startZoneManager(0);
+
+ Set<String> nodes0 = Set.of("node0", "node1");
+
+ setLogicalTopologyInMetaStorage(nodes0, 1);
+
+ CompletableFuture<Set<String>> dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 1);
+
+ assertEquals(nodes0, dataNodesFut.get(3, SECONDS));
+
+ Set<String> nodes1 = Set.of("node0", "node2");
+
+ dataNodesFut = distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 2);
+
+ setLogicalTopologyInMetaStorage(nodes1, 2);
+
+ assertFalse(dataNodesFut.isDone());
+
+ //need to create new zone to fix assert invariant which is broken in this test environment.
+ distributionZoneManager.createZone(new DistributionZoneConfigurationParameters.Builder("zone0")
+ .dataNodesAutoAdjustScaleUp(1000).dataNodesAutoAdjustScaleDown(1000).build())
+ .get(3, SECONDS);
+
+ assertFalse(dataNodesFut.isDone());
+
+ distributionZoneManager.alterZone(DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(1000).dataNodesAutoAdjustScaleDown(1000).build())
+ .get(3, SECONDS);
+
+ assertEquals(nodes0, dataNodesFut.get(3, SECONDS));
+ }
+
+ /**
+ * Test checks that data nodes are initialized on zone manager start.
+ */
+ @Test
+ void testInitializedDataNodesOnZoneManagerStart() throws Exception {
+ Set<String> dataNodes = Set.of("node0", "node1");
+
+ Map<ByteArray, byte[]> valEntries = new HashMap<>();
+
+ valEntries.put(zonesLogicalTopologyKey(), toBytes(dataNodes));
+ valEntries.put(zonesLogicalTopologyVersionKey(), longToBytes(3));
+
+ vaultManager.putAll(valEntries);
+
+ Collection<LogicalNode> nodes = new ArrayList<>();
+
+ nodes.add(new LogicalNode(new ClusterNode("node0", "node0", new NetworkAddress("local", 1))));
+ nodes.add(new LogicalNode(new ClusterNode("node1", "node1", new NetworkAddress("local", 1))));
+
+ when(cmgManager.logicalTopology()).thenReturn(completedFuture(new LogicalTopologySnapshot(3, nodes)));
+
+ startZoneManager(10);
+
+ assertEquals(dataNodes, distributionZoneManager.topologyVersionedDataNodes(DEFAULT_ZONE_ID, 2)
+ .get(3, SECONDS));
+ }
+
+ private void startZoneManager(long revision) throws Exception {
+ vaultManager.put(new ByteArray("applied_revision"), longToBytes(revision)).get();
+
+ distributionZoneManager.start();
+
+ distributionZoneManager.alterZone(
+ DEFAULT_ZONE_NAME, new DistributionZoneConfigurationParameters.Builder(DEFAULT_ZONE_NAME)
+ .dataNodesAutoAdjustScaleUp(IMMEDIATE_TIMER_VALUE)
+ .dataNodesAutoAdjustScaleDown(IMMEDIATE_TIMER_VALUE).build())
+ .get(3, SECONDS);
+ }
+
+ private void setLogicalTopologyInMetaStorage(Set<String> nodes, long topVer) {
+ CompletableFuture<Boolean> invokeFuture = metaStorageManager.invoke(
+ Conditions.exists(zonesLogicalTopologyKey()),
+ List.of(
+ Operations.put(zonesLogicalTopologyKey(), toBytes(nodes)),
+ Operations.put(zonesLogicalTopologyVersionKey(), longToBytes(topVer))
+ ),
+ List.of(Operations.noop())
+ );
+
+ assertThat(invokeFuture, willBe(true));
+ }
+
+ private void setNodesWatchInMetaStorage(int zoneId, Set<String> nodes, boolean isScaleUp, long scaleRevision) {
Review Comment:
Unused method
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneAwaitDataNodesTest.java:
##########
@@ -0,0 +1,605 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.distributionzones;
+
+import static java.util.Collections.emptySet;
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_ID;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.DEFAULT_ZONE_NAME;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.IMMEDIATE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZoneManager.INFINITE_TIMER_VALUE;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.toDataNodesMap;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneDataNodesKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleDownChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zoneScaleUpChangeTriggerKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyKey;
+import static org.apache.ignite.internal.distributionzones.DistributionZonesUtil.zonesLogicalTopologyVersionKey;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willBe;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.ByteUtils.longToBytes;
+import static org.apache.ignite.internal.util.ByteUtils.toBytes;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.raft.ClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.raft.TestClusterStateStorage;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopology;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyImpl;
+import org.apache.ignite.internal.cluster.management.topology.LogicalTopologyServiceImpl;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalNode;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologySnapshot;
+import org.apache.ignite.internal.configuration.ConfigurationManager;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZoneConfigurationSchema;
+import org.apache.ignite.internal.distributionzones.configuration.DistributionZonesConfiguration;
+import org.apache.ignite.internal.distributionzones.exception.DistributionZoneWasRemovedException;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.MetaStorageManager;
+import org.apache.ignite.internal.metastorage.WatchListener;
+import org.apache.ignite.internal.metastorage.dsl.Conditions;
+import org.apache.ignite.internal.metastorage.dsl.Operations;
+import org.apache.ignite.internal.metastorage.impl.StandaloneMetaStorageManager;
+import org.apache.ignite.internal.metastorage.server.SimpleInMemoryKeyValueStorage;
+import org.apache.ignite.internal.schema.configuration.TablesConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.vault.VaultManager;
+import org.apache.ignite.internal.vault.inmemory.InMemoryVaultService;
+import org.apache.ignite.lang.ByteArray;
+import org.apache.ignite.network.ClusterNode;
+import org.apache.ignite.network.NetworkAddress;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests awaiting data nodes algorithm in distribution zone manager in case when
+ * {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * or {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} are immediate.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class DistributionZoneAwaitDataNodesTest extends IgniteAbstractTest {
+ private static final IgniteLogger LOG = Loggers.forClass(DistributionZoneAwaitDataNodesTest.class);
+
+ private MetaStorageManager metaStorageManager;
+
+ private DistributionZoneManager distributionZoneManager;
+
+ private LogicalTopology logicalTopology;
+
+ private ClusterStateStorage clusterStateStorage;
+
+ private ConfigurationManager clusterCfgMgr;
+
+ private ClusterManagementGroupManager cmgManager;
+
+ private VaultManager vaultManager;
+
+ @InjectConfiguration
+ private TablesConfiguration tablesConfiguration;
+
+ @InjectConfiguration
+ protected DistributionZonesConfiguration zonesConfiguration;
Review Comment:
why protected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1154136650
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -500,6 +545,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null && zoneState.scaleUpRevision() < scaleUpRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ zoneState.revisionScaleUpFutures().ceilingEntry(scaleUpRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleUpFut0 = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleUpFut0 == null) {
+ topVerScaleUpFut0 = new CompletableFuture<>();
+
+ zoneState.revisionScaleUpFutures().put(scaleUpRevision, topVerScaleUpFut0);
+ }
+
+ topVerScaleUpFut0.handle((ignored0, e) -> {
+ if (e == null) {
+ topVerScaleUpFut.complete(null);
+ } else {
+ topVerScaleUpFut.completeExceptionally(e);
+ }
+
+ return null;
+ });
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ }
+ });
+
+ CompletableFuture<Void> topVerScaleDownFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ CompletableFuture<Void> topVerScaleDownFut0 = null;
+
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ if (immediateScaleDown.get()) {
+ Map.Entry<Long, Long> scaleDownRevisionEntry = topVerAndScaleDownRevision.ceilingEntry(topVer);
+
+ Long scaleDownRevision = null;
+
+ if (scaleDownRevisionEntry != null) {
+ scaleDownRevision = scaleDownRevisionEntry.getValue();
+ }
+
+ if (scaleDownRevision != null && zoneState.scaleDownRevision() < scaleDownRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ zoneState.revisionScaleDownFutures().ceilingEntry(scaleDownRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleDownFut0 = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleDownFut0 == null) {
+ topVerScaleDownFut0 = new CompletableFuture<>();
+
+ zoneState.revisionScaleDownFutures().put(scaleDownRevision, topVerScaleDownFut0);
+ }
+
+ topVerScaleDownFut0.handle((ignored0, e) -> {
+ if (e == null) {
+ topVerScaleDownFut.complete(null);
+ } else {
+ topVerScaleDownFut.completeExceptionally(e);
+ }
+
+ return null;
+ });
+ } else {
+ topVerScaleDownFut.complete(null);
Review Comment:
What does this clause means? Please add the javadoc explaining why it's possible.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -500,6 +545,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null && zoneState.scaleUpRevision() < scaleUpRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ zoneState.revisionScaleUpFutures().ceilingEntry(scaleUpRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleUpFut0 = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleUpFut0 == null) {
+ topVerScaleUpFut0 = new CompletableFuture<>();
+
+ zoneState.revisionScaleUpFutures().put(scaleUpRevision, topVerScaleUpFut0);
+ }
+
+ topVerScaleUpFut0.handle((ignored0, e) -> {
+ if (e == null) {
+ topVerScaleUpFut.complete(null);
+ } else {
+ topVerScaleUpFut.completeExceptionally(e);
+ }
+
+ return null;
+ });
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ }
+ });
+
+ CompletableFuture<Void> topVerScaleDownFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ CompletableFuture<Void> topVerScaleDownFut0 = null;
+
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
Review Comment:
How it's possible to specify incorrect zoneId. It might be an assertion, but not DistributionZoneNotFoundException because we have zone existence validation on table create/altering etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156779075
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -163,6 +176,30 @@ public class DistributionZoneManager implements IgniteComponent {
*/
private final Map<Integer, ZoneState> zonesState;
+ /** Data nodes modification mutex. */
+ private final Object dataNodesMutex = new Object();
+
+ /** The last topology version which was observed by distribution zone manager. */
+ private long lastTopVer;
Review Comment:
Added initialization. But I have some doubts about the correct initialization of lastScaleUpRevision/lastScaleDownRevision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1156787964
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -500,6 +545,189 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing started nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing stopped nodes
+ * corresponding to the topology version to data nodes into the meta storage .
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Set<String>> dataNodesFut = new CompletableFuture<>();
+
+ CompletableFuture<Void> topVerFut;
+
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ topVerFut = topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ topVerFut = completedFuture(null);
+ }
+ }
+
+ var immediateScaleUp = new AtomicBoolean();
+ var immediateScaleDown = new AtomicBoolean();
+
+ topVerFut = topVerFut.thenAccept(ignored -> {
+ boolean immediateScaleUp0 = false;
+ boolean immediateScaleDown0 = false;
+
+ if (zoneId == DEFAULT_ZONE_ID) {
+ immediateScaleUp0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleUp().value() == 0;
+ immediateScaleDown0 = zonesConfiguration.defaultDistributionZone().dataNodesAutoAdjustScaleDown().value() == 0;
+ } else {
+ NamedConfigurationTree<DistributionZoneConfiguration, DistributionZoneView, DistributionZoneChange> zones =
+ zonesConfiguration.distributionZones();
+
+ for (int i = 0; i < zones.value().size(); i++) {
+ DistributionZoneView zone = zones.value().get(i);
+
+ if (zone.zoneId() == zoneId) {
+ immediateScaleUp0 = zone.dataNodesAutoAdjustScaleUp() == 0;
+ immediateScaleDown0 = zone.dataNodesAutoAdjustScaleDown() == 0;
+
+ break;
+ }
+ }
+ }
+
+ immediateScaleUp.set(immediateScaleUp0);
+ immediateScaleDown.set(immediateScaleDown0);
+ });
+
+ CompletableFuture<Void> topVerScaleUpFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ CompletableFuture<Void> topVerScaleUpFut0 = null;
+
+ if (immediateScaleUp.get()) {
+ Map.Entry<Long, Long> scaleUpRevisionEntry = topVerAndScaleUpRevision.ceilingEntry(topVer);
+
+ Long scaleUpRevision = null;
+
+ if (scaleUpRevisionEntry != null) {
+ scaleUpRevision = scaleUpRevisionEntry.getValue();
+ }
+
+ if (scaleUpRevision != null && zoneState.scaleUpRevision() < scaleUpRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ zoneState.revisionScaleUpFutures().ceilingEntry(scaleUpRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleUpFut0 = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleUpFut0 == null) {
+ topVerScaleUpFut0 = new CompletableFuture<>();
+
+ zoneState.revisionScaleUpFutures().put(scaleUpRevision, topVerScaleUpFut0);
+ }
+
+ topVerScaleUpFut0.handle((ignored0, e) -> {
+ if (e == null) {
+ topVerScaleUpFut.complete(null);
+ } else {
+ topVerScaleUpFut.completeExceptionally(e);
+ }
+
+ return null;
+ });
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ } else {
+ topVerScaleUpFut.complete(null);
+ }
+ }
+ });
+
+ CompletableFuture<Void> topVerScaleDownFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ CompletableFuture<Void> topVerScaleDownFut0 = null;
+
+ synchronized (dataNodesMutex) {
+ ZoneState zoneState = zonesState.get(zoneId);
+
+ if (zoneState == null) {
+ throw new DistributionZoneNotFoundException(zoneId);
+ }
+
+ if (immediateScaleDown.get()) {
+ Map.Entry<Long, Long> scaleDownRevisionEntry = topVerAndScaleDownRevision.ceilingEntry(topVer);
+
+ Long scaleDownRevision = null;
+
+ if (scaleDownRevisionEntry != null) {
+ scaleDownRevision = scaleDownRevisionEntry.getValue();
+ }
+
+ if (scaleDownRevision != null && zoneState.scaleDownRevision() < scaleDownRevision) {
+ Map.Entry<Long, CompletableFuture<Void>> ceilingEntry =
+ zoneState.revisionScaleDownFutures().ceilingEntry(scaleDownRevision);
+
+ if (ceilingEntry != null) {
+ topVerScaleDownFut0 = ceilingEntry.getValue();
+ }
+
+ if (topVerScaleDownFut0 == null) {
+ topVerScaleDownFut0 = new CompletableFuture<>();
+
+ zoneState.revisionScaleDownFutures().put(scaleDownRevision, topVerScaleDownFut0);
+ }
+
+ topVerScaleDownFut0.handle((ignored0, e) -> {
+ if (e == null) {
+ topVerScaleDownFut.complete(null);
+ } else {
+ topVerScaleDownFut.completeExceptionally(e);
+ }
+
+ return null;
+ });
+ } else {
+ topVerScaleDownFut.complete(null);
Review Comment:
I updated java doc. But I don't think this is the last java doc fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sanpwc commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1157209807
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
+
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = getImmediateTimers(zoneId, topVerFut);
+
+ CompletableFuture<Void> topVerScaleUpFut = scaleUpAwaiting(zoneId, timerValuesFut);
+
+ CompletableFuture<Void> topVerScaleDownFut = scaleDownAwaiting(zoneId, timerValuesFut);
+
+ return getDataNodesFuture(zoneId, topVerScaleUpFut, topVerScaleDownFut);
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ return topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ return completedFuture(null);
+ }
+ }
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be scheduled immediate.
+ *
+ * @param zoneId Zone id.
+ * @param topVerFut Future for chaining.
+ * @return Future.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId, CompletableFuture<Void> topVerFut) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
Review Comment:
Busy locks are missing here and there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] sergeyuttsel commented on a diff in pull request #1729: IGNITE-18756 Awaiting for data nodes updating in distribution zone manager.
Posted by "sergeyuttsel (via GitHub)" <gi...@apache.org>.
sergeyuttsel commented on code in PR #1729:
URL: https://github.com/apache/ignite-3/pull/1729#discussion_r1160508020
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
+
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = getImmediateTimers(zoneId, topVerFut);
+
+ CompletableFuture<Void> topVerScaleUpFut = scaleUpAwaiting(zoneId, timerValuesFut);
+
+ CompletableFuture<Void> topVerScaleDownFut = scaleDownAwaiting(zoneId, timerValuesFut);
+
+ return getDataNodesFuture(zoneId, topVerScaleUpFut, topVerScaleDownFut);
+ }
+
+ /**
+ * Waits for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ *
+ * @param topVer Topology version.
+ * @return Future for chaining.
+ */
+ private CompletableFuture<Void> awaitTopologyVersion(long topVer) {
+ synchronized (dataNodesMutex) {
+ if (topVer > lastTopVer) {
+ return topVerFutures.computeIfAbsent(topVer, key -> new CompletableFuture<>());
+ } else {
+ return completedFuture(null);
+ }
+ }
+ }
+
+ /**
+ * Transforms {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp}
+ * and {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} values to boolean values.
+ * True if it equals to zero and false if it greater than zero. Zero means that data nodes changing must be scheduled immediate.
+ *
+ * @param zoneId Zone id.
+ * @param topVerFut Future for chaining.
+ * @return Future.
+ */
+ private CompletableFuture<IgniteBiTuple<Boolean, Boolean>> getImmediateTimers(int zoneId, CompletableFuture<Void> topVerFut) {
+ CompletableFuture<IgniteBiTuple<Boolean, Boolean>> timerValuesFut = new CompletableFuture<>();
+
+ topVerFut.thenAccept(ignored -> {
+ DistributionZoneConfiguration zoneCfg = getZoneById(zonesConfiguration, zoneId);
Review Comment:
fixed.
##########
modules/distribution-zones/src/main/java/org/apache/ignite/internal/distributionzones/DistributionZoneManager.java:
##########
@@ -501,6 +543,222 @@ public int getZoneId(String name) {
}
}
+ /**
+ * The method for obtaining data nodes of the specified zone.
+ * The flow for the future completion:
+ * Waiting for DistributionZoneManager observe passed topology version or greater version in topologyWatchListener.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleUp} equals to 0 than wait for writing data nodes triggered
+ * by started nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * If the {@link DistributionZoneConfigurationSchema#dataNodesAutoAdjustScaleDown} equals to 0 than wait for writing data nodes
+ * triggered by stopped nodes and corresponding to the passed topology version or greater topology version
+ * to the data nodes into the meta storage.
+ * After waiting it returns the future with data nodes of the specified zone.
+ *
+ * @param zoneId Zone id.
+ * @param topVer Topology version.
+ * @return The data nodes future.
+ */
+ public CompletableFuture<Set<String>> topologyVersionedDataNodes(int zoneId, long topVer) {
+ CompletableFuture<Void> topVerFut = awaitTopologyVersion(topVer);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org