You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2020/10/13 01:34:05 UTC
[GitHub] [cassandra] dcapwell commented on a change in pull request #759: Improve machinery for testing bootstrap and range movements
dcapwell commented on a change in pull request #759:
URL: https://github.com/apache/cassandra/pull/759#discussion_r503601390
##########
File path: test/distributed/org/apache/cassandra/distributed/impl/Instance.java
##########
@@ -438,24 +457,27 @@ public void startup(ICluster cluster)
// TODO: this is more than just gossip
if (config.has(GOSSIP))
{
- StorageService.instance.initServer();
+ StorageService.instance.registerDaemon(CassandraDaemon.getInstanceForTesting());
+ StorageService.instance.initServer(StorageService.RING_DELAY,
+ !startupOptions.contains(StartupOption.SKIP_RING_JOIN));
Review comment:
this is an addition to the API, why not leverage the config api we have already? `joinRing` is already defined as
```
private static final boolean joinRing = Boolean.parseBoolean(System.getProperty("cassandra.join_ring", "true"));
```
so it mostly highlights that we want the ability to override system properties in isolation (a test I just added called this out as well); so rather than adding a new interface API into jvm-dtest, what about leveraging the config API?
This also addresses ring delay, which is defined as
```
private static int getRingDelay()
{
String newdelay = System.getProperty("cassandra.ring_delay_ms");
if (newdelay != null)
{
logger.info("Overriding RING_DELAY to {}ms", newdelay);
return Integer.parseInt(newdelay);
}
else
return 30 * 1000;
}
```
##########
File path: test/distributed/org/apache/cassandra/distributed/impl/Instance.java
##########
@@ -438,24 +457,27 @@ public void startup(ICluster cluster)
// TODO: this is more than just gossip
if (config.has(GOSSIP))
{
- StorageService.instance.initServer();
+ StorageService.instance.registerDaemon(CassandraDaemon.getInstanceForTesting());
Review comment:
FYI if you rebase this already exists in each branch.
##########
File path: test/distributed/org/apache/cassandra/distributed/impl/Instance.java
##########
@@ -438,24 +457,27 @@ public void startup(ICluster cluster)
// TODO: this is more than just gossip
if (config.has(GOSSIP))
{
- StorageService.instance.initServer();
+ StorageService.instance.registerDaemon(CassandraDaemon.getInstanceForTesting());
+ StorageService.instance.initServer(StorageService.RING_DELAY,
+ !startupOptions.contains(StartupOption.SKIP_RING_JOIN));
StorageService.instance.removeShutdownHook();
Gossiper.waitToSettle();
}
- else
+ else if (!startupOptions.contains(StartupOption.SKIP_RING_JOIN))
{
- initializeRing(cluster);
- }
+ cluster.stream().forEach(peer -> {
Review comment:
this doesn't look to match the `cassandra.join_ring` property, why do we mock out the ring if we ask to skip joining the ring?
What I see is now, if you ask to not join the ring we only setup gossip to know about this instance
```
Collection<Token> tokens = SystemKeyspace.getSavedTokens();
if (!tokens.isEmpty())
{
tokenMetadata.updateNormalTokens(tokens, FBUtilities.getBroadcastAddressAndPort());
// order is important here, the gossiper can fire in between adding these two states. It's ok to send TOKENS without STATUS, but *not* vice versa.
List<Pair<ApplicationState, VersionedValue>> states = new ArrayList<Pair<ApplicationState, VersionedValue>>();
states.add(Pair.create(ApplicationState.TOKENS, valueFactory.tokens(tokens)));
states.add(Pair.create(ApplicationState.STATUS_WITH_PORT, valueFactory.hibernate(true)));
states.add(Pair.create(ApplicationState.STATUS, valueFactory.hibernate(true)));
Gossiper.instance.addLocalApplicationStates(states);
}
doAuthSetup(true);
```
so feel that we should keep this behavior here as well, so should only do our mocking of the ring when gossip isn't enabled.
##########
File path: test/distributed/org/apache/cassandra/distributed/impl/IsolatedExecutor.java
##########
@@ -139,6 +139,10 @@
private static <E extends Serializable> E transferOneObject(E object, ClassLoader classLoader, Method deserializeOnInstance)
{
+ // already the right object
+ if (object.getClass().getClassLoader() == classLoader)
Review comment:
if the lambda is in the class loader, then do we know all objects it touches are also in the class loader?
##########
File path: test/distributed/org/apache/cassandra/distributed/action/GossipHelper.java
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.action;
+
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+import net.openhft.chronicle.core.util.SerializableBiFunction;
+import net.openhft.chronicle.core.util.SerializableConsumer;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.distributed.api.ICluster;
+import org.apache.cassandra.distributed.api.IInstance;
+import org.apache.cassandra.distributed.api.IInvokableInstance;
+import org.apache.cassandra.distributed.api.IIsolatedExecutor;
+import org.apache.cassandra.distributed.impl.Instance;
+import org.apache.cassandra.distributed.shared.VersionedApplicationState;
+import org.apache.cassandra.gms.ApplicationState;
+import org.apache.cassandra.gms.EndpointState;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.gms.VersionedValue;
+import org.apache.cassandra.io.util.DataInputBuffer;
+import org.apache.cassandra.io.util.DataOutputBuffer;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.TokenMetadata;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.schema.MigrationManager;
+import org.apache.cassandra.service.PendingRangeCalculatorService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.FBUtilities;
+
+import static org.apache.cassandra.distributed.impl.DistributedTestSnitch.toCassandraInetAddressAndPort;
+
+
+public class GossipHelper
+{
+
+ public static ClusterAction clusterAction(InstanceAction action)
+ {
+ return cluster -> cluster.stream().forEach(instance -> {
+ action.apply(cluster, instance);
+ });
+ }
+
+ public static ClusterAction clusterAction(InstanceAction action, Predicate<IInvokableInstance> filter)
+ {
+ return cluster -> cluster.stream().forEach(instance -> {
+ if (filter.test(instance))
+ action.apply(cluster, instance);
+ });
+ }
+
+ public static ClusterAction clusterAction(InstanceAction action, int... instanceIds)
+ {
+ return cluster -> {
+ for (int idx : instanceIds)
+ action.apply(cluster, cluster.get(idx));
+ };
+
+ }
+
+ public static InstanceAction statusToBootstrap(IInvokableInstance newNode)
+ {
+ return new StatusToBootstrap(newNode);
+ }
+
+ public static ClusterAction statusToNormal()
+ {
+ return new ClusterAction()
+ {
+ public void apply(ICluster<IInvokableInstance> cluster)
+ {
+ cluster.stream().forEach((peer) -> {
+ clusterAction(statusToNormal(peer)).apply(cluster);
+ });
+ }
+ };
+ }
+
+ public static InstanceAction statusToNormal(IInvokableInstance newNode)
+ {
+ return new StatusToBootstrap(newNode);
+ }
+
+ public static InstanceAction statusToLeaving(IInvokableInstance newNode)
+ {
+ return new StatusToLeaving(newNode);
+ }
+
+ public static InstanceAction bootstrap()
+ {
+ return new BootstrapAction();
+ }
+
+ public static InstanceAction bootstrap(boolean joinRing, Duration waitForBootstrap, Duration waitForSchema)
+ {
+ return new BootstrapAction(joinRing, waitForBootstrap, waitForSchema);
+ }
+
+ public static InstanceAction disseminateGossipState(IInvokableInstance newNode)
+ {
+ return new DisseminateGossipState(newNode);
+ }
+
+ public static InstanceAction pullSchemaFrom(IInvokableInstance pullFrom)
+ {
+ return new PullSchemaFrom(pullFrom);
+ }
+
+ private static class StatusToBootstrap implements InstanceAction
+ {
+ final IInvokableInstance newNode;
+
+ public StatusToBootstrap(IInvokableInstance newNode)
+ {
+ this.newNode = newNode;
+ }
+
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance instance)
+ {
+ changeGossipState(instance,
+ newNode,
+ Arrays.asList(tokens(newNode),
+ statusBootstrapping(newNode),
+ statusWithPortBootstrapping(newNode)));
+ }
+ }
+
+ private static class StatusToLeaving implements InstanceAction
+ {
+ final IInvokableInstance newNode;
+
+ public StatusToLeaving(IInvokableInstance newNode)
+ {
+ this.newNode = newNode;
+ }
+
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance instance)
+ {
+ changeGossipState(instance,
+ newNode,
+ Arrays.asList(tokens(newNode),
+ statusLeaving(newNode),
+ statusWithPortLeaving(newNode)));
+ }
+ }
+
+ private static InstanceAction disableBinary = (cluster, instance) -> {
+ instance.runOnInstance(() -> {
+ StorageService.instance.stopNativeTransport();
+ });
+ };
+
+ public static InstanceAction disableBinary()
+ {
+ return disableBinary;
+ }
+
+ private static class DisseminateGossipState implements InstanceAction
+ {
+ final IInvokableInstance[] from;
+
+ public DisseminateGossipState(IInvokableInstance... from)
+ {
+ this.from = from;
+ }
+
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance instance)
+ {
+ Map<InetSocketAddress, byte[]> m = new HashMap<>();
+ for (IInvokableInstance node : from)
+ {
+ byte[] epBytes = node.callsOnInstance(() -> {
+ EndpointState epState = Gossiper.instance.getEndpointStateForEndpoint(FBUtilities.getBroadcastAddressAndPort());
+ return toBytes(epState);
+ }).call();
+ m.put(node.broadcastAddress(), epBytes);
+ }
+
+ instance.appliesOnInstance((IIsolatedExecutor.SerializableFunction<Map<InetSocketAddress, byte[]>, Void>)
+ (map) -> {
+ Map<InetAddressAndPort, EndpointState> newState = new HashMap<>();
+ for (Map.Entry<InetSocketAddress, byte[]> e : map.entrySet())
+ newState.put(toCassandraInetAddressAndPort(e.getKey()), fromBytes(e.getValue()));
+
+ Gossiper.runInGossipStageBlocking(() -> {
+ Gossiper.instance.applyStateLocally(newState);
+ });
+ return null;
+ }).apply(m);
+ }
+ }
+
+ public static byte[] toBytes(EndpointState epState)
+ {
+ try (DataOutputBuffer out = new DataOutputBuffer(1024))
+ {
+ EndpointState.serializer.serialize(epState, out, MessagingService.current_version);
+ return out.toByteArray();
+ }
+ catch (IOException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static EndpointState fromBytes(byte[] bytes)
+ {
+ try (DataInputBuffer in = new DataInputBuffer(bytes))
+ {
+ return EndpointState.serializer.deserialize(in, MessagingService.current_version);
+ }
+ catch (Throwable t)
+ {
+ throw new RuntimeException(t);
+ }
+ }
+
+ private static class PullSchemaFrom implements InstanceAction
+ {
+ final IInvokableInstance pullFrom;
+
+ public PullSchemaFrom(IInvokableInstance pullFrom)
+ {
+ this.pullFrom = pullFrom;
+ }
+
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance pullTo)
+ {
+ InetSocketAddress addr = pullFrom.broadcastAddress();
+
+ pullTo.runOnInstance(() -> {
+ InetAddressAndPort endpoint = toCassandraInetAddressAndPort(addr);
+ EndpointState state = Gossiper.instance.getEndpointStateForEndpoint(endpoint);
+ MigrationManager.scheduleSchemaPull(endpoint, state);
+ MigrationManager.waitUntilReadyForBootstrap();
+ });
+ }
+ }
+
+ public static class BootstrapAction implements InstanceAction
Review comment:
same as above, why not just have a function? The interface doesn't offer anything and forces the cluster param; which is ignored
##########
File path: test/distributed/org/apache/cassandra/distributed/action/GossipHelper.java
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.action;
+
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+import net.openhft.chronicle.core.util.SerializableBiFunction;
+import net.openhft.chronicle.core.util.SerializableConsumer;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.distributed.api.ICluster;
+import org.apache.cassandra.distributed.api.IInstance;
+import org.apache.cassandra.distributed.api.IInvokableInstance;
+import org.apache.cassandra.distributed.api.IIsolatedExecutor;
+import org.apache.cassandra.distributed.impl.Instance;
+import org.apache.cassandra.distributed.shared.VersionedApplicationState;
+import org.apache.cassandra.gms.ApplicationState;
+import org.apache.cassandra.gms.EndpointState;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.gms.VersionedValue;
+import org.apache.cassandra.io.util.DataInputBuffer;
+import org.apache.cassandra.io.util.DataOutputBuffer;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.TokenMetadata;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.schema.MigrationManager;
+import org.apache.cassandra.service.PendingRangeCalculatorService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.FBUtilities;
+
+import static org.apache.cassandra.distributed.impl.DistributedTestSnitch.toCassandraInetAddressAndPort;
+
+
+public class GossipHelper
+{
+
+ public static ClusterAction clusterAction(InstanceAction action)
+ {
+ return cluster -> cluster.stream().forEach(instance -> {
+ action.apply(cluster, instance);
+ });
+ }
+
+ public static ClusterAction clusterAction(InstanceAction action, Predicate<IInvokableInstance> filter)
+ {
+ return cluster -> cluster.stream().forEach(instance -> {
+ if (filter.test(instance))
+ action.apply(cluster, instance);
+ });
+ }
+
+ public static ClusterAction clusterAction(InstanceAction action, int... instanceIds)
+ {
+ return cluster -> {
+ for (int idx : instanceIds)
+ action.apply(cluster, cluster.get(idx));
+ };
+
+ }
+
+ public static InstanceAction statusToBootstrap(IInvokableInstance newNode)
+ {
+ return new StatusToBootstrap(newNode);
+ }
+
+ public static ClusterAction statusToNormal()
+ {
+ return new ClusterAction()
+ {
+ public void apply(ICluster<IInvokableInstance> cluster)
+ {
+ cluster.stream().forEach((peer) -> {
+ clusterAction(statusToNormal(peer)).apply(cluster);
+ });
+ }
+ };
+ }
+
+ public static InstanceAction statusToNormal(IInvokableInstance newNode)
+ {
+ return new StatusToBootstrap(newNode);
+ }
+
+ public static InstanceAction statusToLeaving(IInvokableInstance newNode)
+ {
+ return new StatusToLeaving(newNode);
+ }
+
+ public static InstanceAction bootstrap()
+ {
+ return new BootstrapAction();
+ }
+
+ public static InstanceAction bootstrap(boolean joinRing, Duration waitForBootstrap, Duration waitForSchema)
+ {
+ return new BootstrapAction(joinRing, waitForBootstrap, waitForSchema);
+ }
+
+ public static InstanceAction disseminateGossipState(IInvokableInstance newNode)
+ {
+ return new DisseminateGossipState(newNode);
+ }
+
+ public static InstanceAction pullSchemaFrom(IInvokableInstance pullFrom)
+ {
+ return new PullSchemaFrom(pullFrom);
+ }
+
+ private static class StatusToBootstrap implements InstanceAction
+ {
+ final IInvokableInstance newNode;
+
+ public StatusToBootstrap(IInvokableInstance newNode)
+ {
+ this.newNode = newNode;
+ }
+
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance instance)
+ {
+ changeGossipState(instance,
+ newNode,
+ Arrays.asList(tokens(newNode),
+ statusBootstrapping(newNode),
+ statusWithPortBootstrapping(newNode)));
+ }
+ }
+
+ private static class StatusToLeaving implements InstanceAction
+ {
+ final IInvokableInstance newNode;
+
+ public StatusToLeaving(IInvokableInstance newNode)
+ {
+ this.newNode = newNode;
+ }
+
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance instance)
+ {
+ changeGossipState(instance,
+ newNode,
+ Arrays.asList(tokens(newNode),
+ statusLeaving(newNode),
+ statusWithPortLeaving(newNode)));
+ }
+ }
+
+ private static InstanceAction disableBinary = (cluster, instance) -> {
+ instance.runOnInstance(() -> {
+ StorageService.instance.stopNativeTransport();
+ });
+ };
+
+ public static InstanceAction disableBinary()
+ {
+ return disableBinary;
+ }
+
+ private static class DisseminateGossipState implements InstanceAction
+ {
+ final IInvokableInstance[] from;
+
+ public DisseminateGossipState(IInvokableInstance... from)
+ {
+ this.from = from;
+ }
+
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance instance)
+ {
+ Map<InetSocketAddress, byte[]> m = new HashMap<>();
+ for (IInvokableInstance node : from)
+ {
+ byte[] epBytes = node.callsOnInstance(() -> {
+ EndpointState epState = Gossiper.instance.getEndpointStateForEndpoint(FBUtilities.getBroadcastAddressAndPort());
+ return toBytes(epState);
+ }).call();
+ m.put(node.broadcastAddress(), epBytes);
+ }
+
+ instance.appliesOnInstance((IIsolatedExecutor.SerializableFunction<Map<InetSocketAddress, byte[]>, Void>)
+ (map) -> {
+ Map<InetAddressAndPort, EndpointState> newState = new HashMap<>();
+ for (Map.Entry<InetSocketAddress, byte[]> e : map.entrySet())
+ newState.put(toCassandraInetAddressAndPort(e.getKey()), fromBytes(e.getValue()));
+
+ Gossiper.runInGossipStageBlocking(() -> {
+ Gossiper.instance.applyStateLocally(newState);
+ });
+ return null;
+ }).apply(m);
+ }
+ }
+
+ public static byte[] toBytes(EndpointState epState)
+ {
+ try (DataOutputBuffer out = new DataOutputBuffer(1024))
+ {
+ EndpointState.serializer.serialize(epState, out, MessagingService.current_version);
+ return out.toByteArray();
+ }
+ catch (IOException e)
+ {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static EndpointState fromBytes(byte[] bytes)
+ {
+ try (DataInputBuffer in = new DataInputBuffer(bytes))
+ {
+ return EndpointState.serializer.deserialize(in, MessagingService.current_version);
+ }
+ catch (Throwable t)
+ {
+ throw new RuntimeException(t);
+ }
+ }
+
+ private static class PullSchemaFrom implements InstanceAction
+ {
+ final IInvokableInstance pullFrom;
+
+ public PullSchemaFrom(IInvokableInstance pullFrom)
+ {
+ this.pullFrom = pullFrom;
+ }
+
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance pullTo)
Review comment:
why not just have a function? I don't see what the `InstanceAction` interface is providing.
```
GossipHelper.pullSchemaFrom(cluster.get(1), newInstance);
// of if you static import
pullSchemaFrom(cluster.get(1), newInstance);
```
vs
```
new PullSchemaFrom(cluster.get(1)).apply(cluster, newInstance);
```
##########
File path: test/distributed/org/apache/cassandra/distributed/test/BootstrapTest.java
##########
@@ -18,81 +18,254 @@
package org.apache.cassandra.distributed.test;
+import java.net.InetSocketAddress;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.LockSupport;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import org.junit.Assert;
import org.junit.Test;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.SimpleStatement;
+import com.datastax.driver.core.Statement;
+import com.datastax.driver.core.policies.RoundRobinPolicy;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
import org.apache.cassandra.distributed.Cluster;
+import org.apache.cassandra.distributed.action.GossipHelper;
+import org.apache.cassandra.distributed.action.InstanceAction;
import org.apache.cassandra.distributed.api.ConsistencyLevel;
import org.apache.cassandra.distributed.api.ICluster;
+import org.apache.cassandra.distributed.api.IInstance;
import org.apache.cassandra.distributed.api.IInstanceConfig;
+import org.apache.cassandra.distributed.api.IInvokableInstance;
+import org.apache.cassandra.distributed.api.IMessage;
+import org.apache.cassandra.distributed.api.IMessageFilters;
import org.apache.cassandra.distributed.api.TokenSupplier;
+import org.apache.cassandra.distributed.impl.DistributedTestSnitch;
+import org.apache.cassandra.distributed.impl.RowUtil;
import org.apache.cassandra.distributed.shared.NetworkTopology;
+import org.apache.cassandra.locator.EndpointsForRange;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.AssertUtil;
+import org.apache.cassandra.utils.FBUtilities;
+import static org.apache.cassandra.distributed.action.GossipHelper.*;
import static org.apache.cassandra.distributed.api.Feature.GOSSIP;
+import static org.apache.cassandra.distributed.api.Feature.NATIVE_PROTOCOL;
import static org.apache.cassandra.distributed.api.Feature.NETWORK;
+import static org.apache.cassandra.distributed.shared.AssertUtils.assertRows;
+import static org.apache.cassandra.distributed.shared.AssertUtils.row;
-// TODO: this test should be removed after running in-jvm dtests is set up via the shared API repository
public class BootstrapTest extends TestBaseImpl
{
-
@Test
public void bootstrapTest() throws Throwable
{
int originalNodeCount = 2;
int expandedNodeCount = originalNodeCount + 1;
- Cluster.Builder builder = builder().withNodes(originalNodeCount)
- .withTokenSupplier(TokenSupplier.evenlyDistributedTokens(expandedNodeCount))
- .withNodeIdTopology(NetworkTopology.singleDcNetworkTopology(expandedNodeCount, "dc0", "rack0"))
- .withConfig(config -> config.with(NETWORK, GOSSIP));
-
- Map<Integer, Long> withBootstrap = null;
- Map<Integer, Long> naturally = null;
- try (Cluster cluster = builder.withNodes(originalNodeCount).start())
+
+ try (Cluster cluster = builder().withNodes(originalNodeCount)
+ .withTokenSupplier(TokenSupplier.evenlyDistributedTokens(expandedNodeCount))
+ .withNodeIdTopology(NetworkTopology.singleDcNetworkTopology(expandedNodeCount, "dc0", "rack0"))
+ .withConfig(config -> config.with(NETWORK, GOSSIP))
+ .start())
{
- populate(cluster);
+ populate(cluster,0, 100);
IInstanceConfig config = cluster.newInstanceConfig();
- config.set("auto_bootstrap", true);
+ IInvokableInstance newInstance = cluster.initialize(config);
+ newInstance.startup(cluster, IInstance.StartupOption.SKIP_RING_JOIN);
+ clusterAction(statusToBootstrap(newInstance)).apply(cluster);
+ pullSchemaFrom(cluster.get(1)).apply(cluster, newInstance);
+ GossipHelper.bootstrap().apply(cluster, newInstance);
- cluster.bootstrap(config).startup();
+ for (Map.Entry<Integer, Long> e : count(cluster).entrySet())
+ Assert.assertEquals(e.getValue().longValue(), 100L);
+ }
+ }
- cluster.stream().forEach(instance -> {
- instance.nodetool("cleanup", KEYSPACE, "tbl");
- });
+ @Test
+ public void testPendingWrites() throws Throwable
+ {
+ int originalNodeCount = 2;
+ int expandedNodeCount = originalNodeCount + 1;
- withBootstrap = count(cluster);
+ try (Cluster cluster = builder().withNodes(originalNodeCount)
+ .withTokenSupplier(TokenSupplier.evenlyDistributedTokens(expandedNodeCount))
+ .withNodeIdTopology(NetworkTopology.singleDcNetworkTopology(expandedNodeCount, "dc0", "rack0"))
+ .withConfig(config -> config.with(NETWORK, GOSSIP))
+ .start())
+ {
+ populate(cluster, 0, 100);
+
+ IInstanceConfig config = cluster.newInstanceConfig();
+ IInvokableInstance newInstance = cluster.initialize(config);
+ newInstance.startup(cluster, IInstance.StartupOption.SKIP_RING_JOIN);
+ clusterAction(statusToBootstrap(newInstance)).apply(cluster);
+
+ bootstrap(false, Duration.ofSeconds(60), Duration.ofSeconds(10)).apply(cluster, newInstance);
+
+ cluster.get(1).acceptsOnInstance((InetSocketAddress ip) -> {
+ Set<InetAddressAndPort> set = new HashSet<>();
+ for (Map.Entry<Range<Token>, EndpointsForRange.Builder> e : StorageService.instance.getTokenMetadata().getPendingRanges(KEYSPACE))
+ {
+ set.addAll(e.getValue().build().endpoints());
+ }
+ Assert.assertEquals(set.size(), 1);
+ Assert.assertTrue(String.format("%s should contain %s", set, ip),
+ set.contains(DistributedTestSnitch.toCassandraInetAddressAndPort(ip)));
+ }).accept(cluster.get(3).broadcastAddress());
+
+ populate(cluster, 100, 150);
+
+ joinRing().apply(cluster, newInstance);
+ clusterAction(disseminateGossipState(newInstance),1, 2).apply(cluster);
+
+ cluster.get(1).acceptsOnInstance((InetSocketAddress ip) -> {
+ Set<InetAddressAndPort> set = new HashSet<>();
+ for (Map.Entry<Range<Token>, EndpointsForRange.Builder> e : StorageService.instance.getTokenMetadata().getPendingRanges(KEYSPACE))
+ set.addAll(e.getValue().build().endpoints());
+ assert set.size() == 0 : set;
+ }).accept(cluster.get(3).broadcastAddress());
+
+ for (Map.Entry<Integer, Long> e : count(cluster).entrySet())
+ Assert.assertEquals(e.getValue().longValue(), 150L);
}
+ }
- builder = builder.withNodes(expandedNodeCount)
- .withTokenSupplier(TokenSupplier.evenlyDistributedTokens(expandedNodeCount))
- .withConfig(config -> config.with(NETWORK, GOSSIP));
+ @Test
+ public void internodeConnectionsDuringDecom() throws Throwable
+ {
+ ExecutorService executor = Executors.newFixedThreadPool(1);
+ try (Cluster cluster = builder().withNodes(4)
+ .withConfig(config -> config.with(NETWORK, GOSSIP, NATIVE_PROTOCOL))
+ .start())
+ {
+ populate(cluster, 0, 100);
- try (ICluster cluster = builder.start())
+ new DecomissionAction().apply(cluster, cluster.get(1));
+
+ cluster.filters().allVerbs().from(1).messagesMatching((i, i1, iMessage) -> {
+ throw new AssertionError("Decomissioned node should not send any messages");
+ }).drop();
+
+
+ Map<Integer, Long> connectionAttempts = new HashMap<>();
+ long deadline = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(10);
+
+ // Wait 10 seconds and check if there are any new connection attempts to the decomissioned node
+ while (System.currentTimeMillis() <= deadline)
+ {
+ for (int i = 2; i <= cluster.size(); i++)
+ {
+ Object[][] res = cluster.get(i).executeInternal("SELECT active_connections, connection_attempts FROM system_views.internode_outbound WHERE address = '127.0.0.1' AND port = 7012");
+ Assert.assertEquals(1, res.length);
+ Assert.assertEquals(0L, ((Long) res[0][0]).longValue());
+ long attempts = ((Long) res[0][1]).longValue();
+ if (connectionAttempts.get(i) == null)
+ connectionAttempts.put(i, attempts);
+ else
+ Assert.assertEquals(connectionAttempts.get(i), (Long) attempts);
+ }
+ LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(100));
+ }
+ }
+ }
+
+ @Test
+ public void clientConnectionsDuringDecom() throws Throwable
+ {
+ ExecutorService executor = Executors.newFixedThreadPool(1);
+ try (Cluster cluster = builder().withNodes(4)
+ .withConfig(config -> config.with(NETWORK, GOSSIP, NATIVE_PROTOCOL))
+ .start())
{
- populate(cluster);
- naturally = count(cluster);
+ populate(cluster, 0, 100);
+
+ try (com.datastax.driver.core.Cluster client = com.datastax.driver.core.Cluster.builder().addContactPoint("127.0.0.1")
+ .withLoadBalancingPolicy(new RoundRobinPolicy())
+ .build();
+ Session session = client.connect())
+ {
+ CompletableFuture.runAsync(() -> {
+ while (!Thread.currentThread().isInterrupted())
+ {
+ if (client.getMetadata().getAllHosts().size() == cluster.size())
+ return;
+ LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(100));
+ }
+ }, executor).get(1, TimeUnit.MINUTES);
+
+ new DecomissionAction().apply(cluster, cluster.get(1));
+
+ Assert.assertEquals(cluster.size() - 1, client.getMetadata().getAllHosts().size());
+ for (int i = 0; i < 100; i++)
+ {
+ Statement select = new SimpleStatement("select * from " + KEYSPACE + ".tbl WHERE pk = " + i).setConsistencyLevel(com.datastax.driver.core.ConsistencyLevel.ALL);
+ final ResultSet resultSet = session.execute(select);
+ assertRows(RowUtil.toObjects(resultSet), row(i, i, i));
+ }
+ Assert.assertEquals(cluster.size() - 1, client.getMetadata().getAllHosts().size());
+ }
}
+ }
- for (Map.Entry<Integer, Long> e : withBootstrap.entrySet())
- Assert.assertTrue(e.getValue() >= naturally.get(e.getKey()));
+ public static InstanceAction joinRing()
+ {
+ return new JoinRing();
}
- public void populate(ICluster cluster)
+ private static class JoinRing implements InstanceAction
{
- cluster.schemaChange("CREATE KEYSPACE " + KEYSPACE + " WITH replication = {'class': 'SimpleStrategy', 'replication_factor': " + 3 + "};");
- cluster.schemaChange("CREATE TABLE " + KEYSPACE + ".tbl (pk int, ck int, v int, PRIMARY KEY (pk, ck))");
+ public void apply(ICluster<IInvokableInstance> cluster, IInvokableInstance instance)
Review comment:
this looks like the unsafe version of `nodetool join`, is there a reason to avoid bootstrap?
Also, not sure what the difference is to the below, it looks mostly the same
```
System.setProperty("cassandra.auto_bootstrap", "false");
instance.nodetoolResults("join").asserts().success();
```
##########
File path: test/distributed/org/apache/cassandra/distributed/action/GossipHelper.java
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.action;
+
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+import net.openhft.chronicle.core.util.SerializableBiFunction;
+import net.openhft.chronicle.core.util.SerializableConsumer;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.distributed.api.ICluster;
+import org.apache.cassandra.distributed.api.IInstance;
+import org.apache.cassandra.distributed.api.IInvokableInstance;
+import org.apache.cassandra.distributed.api.IIsolatedExecutor;
+import org.apache.cassandra.distributed.impl.Instance;
+import org.apache.cassandra.distributed.shared.VersionedApplicationState;
+import org.apache.cassandra.gms.ApplicationState;
+import org.apache.cassandra.gms.EndpointState;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.gms.VersionedValue;
+import org.apache.cassandra.io.util.DataInputBuffer;
+import org.apache.cassandra.io.util.DataOutputBuffer;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.TokenMetadata;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.schema.MigrationManager;
+import org.apache.cassandra.service.PendingRangeCalculatorService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.FBUtilities;
+
+import static org.apache.cassandra.distributed.impl.DistributedTestSnitch.toCassandraInetAddressAndPort;
+
+
+public class GossipHelper
+{
+
+ public static ClusterAction clusterAction(InstanceAction action)
Review comment:
ClusterAction seems to be a abstraction over looping, so feel the code is more clear by just looping
examples:
```
clusterAction(statusToBootstrap(newInstance)).apply(cluster);
// vs
cluster.forEach(i -> statusToBootstrap(i, newInstance));
```
or use a specific set of nodes
```
clusterAction(disseminateGossipState(newInstance), 1, 2).apply(cluster);
// vs
IntStream.of(1, 2).forEach(i -> disseminateGossipState(i, newInstance));
```
In these examples, it is more clear to see the loop than the ClusterAction abstraction.
##########
File path: test/distributed/org/apache/cassandra/distributed/action/GossipHelper.java
##########
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.action;
+
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+import net.openhft.chronicle.core.util.SerializableBiFunction;
+import net.openhft.chronicle.core.util.SerializableConsumer;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.distributed.api.ICluster;
+import org.apache.cassandra.distributed.api.IInstance;
+import org.apache.cassandra.distributed.api.IInvokableInstance;
+import org.apache.cassandra.distributed.api.IIsolatedExecutor;
+import org.apache.cassandra.distributed.impl.Instance;
+import org.apache.cassandra.distributed.shared.VersionedApplicationState;
+import org.apache.cassandra.gms.ApplicationState;
+import org.apache.cassandra.gms.EndpointState;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.gms.VersionedValue;
+import org.apache.cassandra.io.util.DataInputBuffer;
+import org.apache.cassandra.io.util.DataOutputBuffer;
+import org.apache.cassandra.locator.InetAddressAndPort;
+import org.apache.cassandra.locator.TokenMetadata;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.schema.MigrationManager;
+import org.apache.cassandra.service.PendingRangeCalculatorService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.FBUtilities;
+
+import static org.apache.cassandra.distributed.impl.DistributedTestSnitch.toCassandraInetAddressAndPort;
+
+
+public class GossipHelper
Review comment:
my comments feedback for this class is the `InstanceAction` interface, it would be simpler to just have a function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org